Microsoft has introduced a new native execution engine within Microsoft Fabric aimed at significantly improving the performance and efficiency of large-scale data processing workloads. The engine, which was designed and built with data engineering workloads in mind and running on Apache Spark, allows for faster query processing by executing the queries closer to the lakehouse infrastructure, thus reducing the overhead that was traditionally associated with the JVM-based Spark processing model. The native execution engine is based on a vectorized architecture that processes data in columnar batches, thus allowing queries to be executed more efficiently and with less data movement and serialization overhead. The engine is built on open-source technologies such as Velox and Apache Gluten and allows Spark workloads to be executed in optimized native code while still being compatible with existing Spark APIs and workloads. According to Microsoft, the new engine architecture allows Spark queries to be executed directly on the data layer without the need for code changes, thus ensuring that organizations can take advantage of the performance improvements without having to re-engineer their pipelines.
Also Read: Databricks Launches Brickbuilder Partner Network to Fuel Growth in the Agentic AI Era
The engine is optimized for execution on popular formats such as Parquet and Delta Lake, thus allowing for faster aggregation, join, and complex transformations that are commonly used in analytics and data engineering pipelines. Moreover, it also comes with a fallback mechanism that will automatically go back to the traditional Spark engine in case any unsupported features or functions are encountered. According to Microsoft, the native execution engine is integrated into the Fabric Runtime 1.3, which is based on Spark 3.5, and can be enabled at the environment, notebook, or Spark job level. Microsoft also mentioned that the technology represents a major architectural shift for the analytics environment in Fabric, allowing companies to analyze large datasets with improved speed, scalability, and efficiency. With the integration of high-performance native execution and the familiar Spark development experience, Microsoft aims to make complex data engineering tasks easier and more affordable for companies using Fabric as their unified analytics platform. The addition of the native execution engine also represents Microsoft’s ongoing commitment to improving the performance of its analytics stack, which will help companies accelerate data pipelines, optimize compute resources, and unlock faster insights into large-scale enterprise data environments.
