Articles

What the new ODBC Driver for Fabric Data Engineering means for your Spark team

The most consequential changes in enterprise data engineering sometimes arrive as a connection string.

On February 19, 2026, Microsoft released the ODBC Driver for Microsoft Fabric Data Engineering in public preview. It’s easy to skim past — connector announcements don’t usually change much. But this one quietly solves a problem that has been frustrating production Spark teams since Fabric launched: how do you run Spark SQL from a normal application, without notebooks, without Spark Job Definitions, without ever opening a browser?

ODBC is how. And the fact that Microsoft reached back to a 34-year-old standard to do it tells you something interesting about where Fabric is heading.

What actually shipped

Let me get specific. The driver is version 1.0.0, ODBC 3.x compliant, and runs on Windows 10/11 and Windows Server 2016+. Under the hood, it talks to Fabric’s Livy APIs. Every query you send through the ODBC interface spins up (or reuses) a Spark session on Fabric’s compute.

That distinction matters. The driver doesn’t bypass Spark. It wraps it. Your SQL statement travels through the ODBC layer, hits the Livy API, and executes as Spark SQL against your Lakehouse. This is not the same as connecting to the SQL Analytics Endpoint, which routes through a different engine entirely.

The session reuse feature deserves attention. If you’ve ever waited 30 to 45 seconds for a Fabric notebook to initialize, you’re familiar with Spark cold-start delays. The driver can hold onto an existing Spark session between queries rather than paying that startup tax every time. Set ReuseSession=true in your connection string, and consecutive queries from the same connection skip the initialization penalty.

Authentication covers five Entra ID flows: Azure CLI for local development, interactive browser for ad-hoc work, client credentials and certificates for service principals, and raw access token support. If your production pipelines already authenticate to other Fabric resources with a service principal, the same credentials work here.

What it doesn’t do: it’s Windows-only in this preview. No Linux, no macOS. It speaks Spark SQL only, not PySpark or the DataFrame API. And it’s a preview — Microsoft can change connection string parameters, error codes, and behavior before GA.

Three groups that should pay close attention

The driver’s value depends entirely on who you are and what you’re trying to connect.

Group one: .NET teams. Before this driver, getting a C# application to run Spark SQL against a Fabric Lakehouse meant either calling the Livy REST API directly (manual session management, custom error handling, lots of boilerplate) or routing through the SQL Analytics Endpoint (different engine, different performance profile, different limitations). Now it’s a connection string and System.Data.Odbc. That’s the kind of simplification that actually changes what people build.

Group two: BI tool users. Excel, legacy reporting platforms, anything that speaks ODBC — they can now connect directly to Spark compute on Fabric. This matters because Spark handles complex types like arrays, maps, and structs natively, plus it processes large analytical workloads differently than the SQL endpoint. If your Lakehouse tables use nested schemas, this driver exposes them directly rather than flattening them.

Group three: platform engineers. If you run Azure DevOps pipelines, GitHub Actions, or custom orchestrators that need to validate data or execute Spark SQL as part of a deployment, the ODBC driver with service principal auth gives you a programmatic, credential-managed path with no UI interaction required. This is what “infrastructure as code” looks like for Spark connectivity.

Trade-offs to plan for

Every feature comes with trade-offs, and it’s worth understanding these before you roll out broadly.

Every ODBC connection that creates a new Spark session consumes Fabric capacity. Imagine ten analysts each open an ODBC connection from their BI tool. That’s ten concurrent Spark sessions, all burning CU seconds. The session reuse feature helps within a single connection, but it doesn’t pool sessions across users. On a shared capacity, CU consumption can add up faster than you’d expect.

Then there’s the timeout problem. Fabric’s Livy sessions have a default idle timeout. If an analyst runs a query, spends eight minutes reading the results, and runs another, the session may have timed out. The next query pays the full cold-start penalty again. For interactive workflows, it’s worth planning for this — users will see variable response times, and understanding why helps set the right expectations.

The Windows-only constraint creates a real deployment asymmetry. Many data engineering teams develop on macOS or Linux. They can use the JDBC driver locally (which is cross-platform) but can’t use the ODBC driver until they deploy to a Windows CI/CD agent or server. That means some behaviors will only surface in the deployment environment, so factor in extra validation time for Windows-hosted stages.

A rollout checklist for Spark team leads

If you’re evaluating this driver for production, here’s a concrete sequence:

  1. Map your current connectivity. Catalog every application and tool querying your Lakehouse today. Note which ones use the SQL Analytics Endpoint, which call Livy directly, and which use the JDBC driver. The ODBC driver fills gaps — it doesn’t need to replace things that already work.

  2. Benchmark session reuse under your actual patterns. Set ReuseSession=true and run your typical query workload. Measure the difference between first-query latency (cold start) and subsequent-query latency (warm session). If your workload involves long idle gaps between queries, session reuse won’t save you much, and you’ll need to decide whether to accept the latency or build a keep-alive mechanism.

  3. Model the capacity cost before rolling out broadly. For each application or tool that would use the driver, estimate concurrent Spark sessions. Multiply by CU cost per session-hour. Compare this against routing the same queries through the SQL Analytics Endpoint. For simple aggregations on well-structured tables, the SQL endpoint is often cheaper. Reserve the ODBC-to-Spark path for workloads that genuinely need Spark’s capabilities.

  4. Use service principal auth from day one. Azure CLI auth is fine for a proof of concept. In production, configure a dedicated service principal with minimum permissions on your workspace. Store credentials in Azure Key Vault. Personal tokens in pipelines are something you’ll want to migrate away from early.

  5. Abstract the connection layer. Because this is a preview, put the ODBC connection behind an interface in your application code. If you need to fall back to direct Livy API calls or swap in the JDBC driver, you should be able to do that without touching business logic.

  6. Set up session monitoring and alerts. Use the Fabric capacity metrics app or monitoring APIs to track active Spark sessions. Alert if the session count crosses a threshold tied to your CU budget. This catches runaway connections before they become a capacity incident.

  7. Pin the driver version. Download 1.0.0, deploy it to your target machines, and only upgrade after testing the new version against your workloads. Auto-updating preview drivers in production is a risk worth avoiding.

Where this fits in Fabric’s arc

There’s a pattern worth noticing. First Microsoft shipped notebooks. Then Spark Job Definitions. Then the JDBC driver for Java. Now the ODBC driver for everything else. Each release pushes Spark compute further from the Fabric browser UI and closer to the tools and workflows teams already use.

The direction is unmistakable: Microsoft wants Fabric’s Lakehouse queryable from anywhere, through whatever protocol your application already speaks. Two years ago, Spark in Fabric meant opening a browser and writing notebook cells. Today it means passing a connection string to pyodbc or System.Data.Odbc and running SQL from whatever runtime you prefer.

For Spark teams already running in Fabric, the ODBC driver is a pragmatic addition that fills a real connectivity gap. For teams evaluating Fabric, it lowers the integration barrier with existing .NET, Python, and BI toolchains. And for the platform engineers who spend their days wiring systems together, it replaces custom Livy API wrappers with a standard interface that every operating system and language already knows how to talk to.

Sometimes the most interesting changes arrive in the most unremarkable packaging.

This post was written with help from anthropic/claude-opus-4-6

fabric-cicd Is Now Officially Supported — Here’s Your Production Deployment Checklist

Three days ago, Microsoft promoted fabric-cicd from community project to officially supported tool. That Python library your team has been running in a “we’re still figuring out our deployment process” sort of way now carries Microsoft’s name and their support commitment.

That shift matters in three concrete places. First, your compliance team can stop asking “is this thing even supported?” Second, you can open Microsoft support tickets when it breaks. Third, the roadmap is no longer a volunteer effort. Features will land faster. Bugs will get fixed on a schedule.

But here’s where most teams stall. They read the announcement, nod approvingly, and then do absolutely nothing different. The notebook still gets deployed by clicking sync in the browser. The lakehouse GUID is still hardcoded. The “production” workspace is still one bad merge away from serving yesterday’s dev code to the entire analytics team.

An announcement without an execution plan is just news. Let’s build the plan.

What Fabric-CICD Does (and Where It Stops)

Understand the boundaries before you reorganize your deployment story. fabric-cicd is a Python library. You give it a Git repository, a target workspace ID, and a list of item types. It reads the item definitions from the repo, resolves dependencies between them, applies parameter substitutions, and pushes everything to the workspace. It can also remove orphan items that exist in the workspace but no longer appear in your repo.

It supports 25 item types: Notebooks, SparkJobDefinitions, Environments, Lakehouses, DataPipelines, SemanticModels, Warehouses, and 18 others. Every deployment is a full deployment. No commit diffs, no incremental updates. The entire in-scope state gets pushed every time.

Where it stops: it won’t manage your Spark compute sizing, it won’t migrate lakehouse data between environments, and it won’t coordinate multi-workspace transactions atomically. Those gaps are yours to fill. That’s not a weakness. A tool that owns its scope and does it well beats one that covers everything and nails nothing.

Prerequisite Zero: Get Your Git House in Order

This is the part that takes longer than anyone budgets for.

fabric-cicd reads from a Git repository. If your Fabric workspace isn’t connected to one, the tool has nothing to deploy. And plenty of Spark teams are still running workspaces where notebooks were born in the browser, edited in the browser, and will die in the browser without ever touching version control.

Connect your workspace to Azure DevOps or GitHub through Fabric’s Git Integration. Every notebook, every Spark job definition, every environment configuration goes into source control. All of it.

If your repo currently contains items named notebook_v2_final_FINAL_USE_THIS_ONE — and honestly, most of us have been there — now’s the time to clean that up before automating. Automating a disorganized repo just moves the disorganization faster. Getting the foundation right first saves real time down the road.

Your target state when this prerequisite is done: a main branch that mirrors production, feature branches for development work, and a merge strategy the whole team agrees on. fabric-cicd reads from a directory on disk. What it reads needs to be coherent.

The Parameter File: The Single Most Important Artifact

The parameter.yml file is where fabric-cicd learns the difference between your dev environment and production. Without it, you’re deploying identical configurations everywhere, which means your production notebooks will happily point at your dev lakehouse.

For Spark teams, four categories of parameter entries matter:

Default Lakehouse IDs. Every notebook binds to a lakehouse by GUID. In dev, that GUID points to your sandbox with test data. In production, it points to the lakehouse with three months of curated, retention-managed data. The parameter file swaps those GUIDs at deploy time. Miss one, and your production job reads from a lakehouse that got wiped last Tuesday.

Default Lakehouse Workspace IDs. If your production lakehouse lives in a separate workspace from dev (and it should), this mapping covers that scope. Lakehouse GUIDs alone aren’t enough when workspaces differ between environments.

Connection strings. Any notebook that pulls from an external data source needs environment-specific connection details. Hardcoded connection strings are how you end up running your production Spark cluster against a dev SQL database. That kind of mismatch can get expensive quickly — and it’s entirely preventable with proper parameterization.

Notebook parameter cells. Fabric lets you define parameter cells in notebooks. Every value that changes between environments belongs there, referenced by parameter.yml. Not in a comment. Not in a variable halfway down the notebook. In the parameter cell, where the tooling can find it.

The mechanism is find-and-replace. fabric-cicd scans your repository files for specific strings and swaps in the values for the target environment. This means the GUIDs in your repo must be consistent. If someone manually edited a lakehouse ID through the browser after a sync, the parameter file won’t catch the mismatch. Deployments will succeed. The notebook will fail. Those are the worst kind of bugs: silent ones.

Build Your Pipeline in Four Stages

Here’s a pipeline structure built for Spark teams, in the order things should execute:

Stage 1: Validate. Run your tests before anything deploys. If you have PySpark unit tests (even five of them), execute them against a local SparkSession or a lightweight Fabric environment. This catches broken imports, renamed functions, and bad type signatures. The goal isn’t 100% test coverage. The goal is catching the obvious failures before they reach a workspace anyone else depends on.

Stage 2: Build. Initialize the FabricWorkspace object with your target workspace ID, environment name, repository path, and scoped item types. For Spark teams, start with ["Notebook", "SparkJobDefinition", "Environment", "Lakehouse"]. Do not scope every item type on day one. Start with the items you deploy weekly. Expand scope after the first month, when you’ve seen how it behaves.

Stage 3: Deploy. Call publish_all_items(). The tool resolves dependency ordering, so if a notebook depends on a lakehouse that depends on an environment configuration, the sequence is handled. After publishing, call unpublish_all_orphan_items() to clean up workspace items that no longer appear in the repo. Skipping orphan cleanup means your workspace accumulates dead items that confuse the team and waste capacity.

Stage 4: Verify. This is the stage teams skip, and the one that saves them. After deployment, run a smoke test against the target workspace. Can the notebook open? Does it bind to the correct lakehouse? Can a lightweight execution complete without errors? A deployment that returns exit code zero but leaves notebooks pointing at a deleted lakehouse is not a successful deployment. Your pipeline shouldn’t treat it as one.

Guardrails Worth the Setup Cost

Guardrails turn a pipeline from a deployment mechanism into a safety net. These four are worth the setup time:

Approval gates. Require explicit human approval before any deployment to Production. fabric-cicd won’t enforce this for you. Wire it into your pipeline platform: Azure DevOps release gates, GitHub Actions environments with required reviewers. The first time a broken merge auto-deploys to production, you’ll wish you had spent the twenty minutes setting this up.

Service principal authentication. Run your pipeline under a service principal, not a user account. Give the principal workspace contributor access on the target workspace. Nothing more. When someone leaves the team or changes roles, deployments keep working because they never depended on that person’s credentials.

Tested rollback. Since fabric-cicd does full deployments from the repo, rollback means redeploying the last known-good commit. Conceptually clean. But “conceptually clean” doesn’t help you during an incident when stakeholders need answers fast. Test the rollback. Revert a deployment on a Tuesday afternoon when nothing is on fire. Confirm the workspace returns to its previous state. If you haven’t tested it, your rollback plan is still untested — and untested plans have a way of surprising you at the worst possible moment.

Deployment artifacts. Every pipeline run should log which items deployed, which parameters were substituted, and which orphans were removed. When production breaks and someone asks “what changed since yesterday?”, the answer should take thirty seconds, not three hours of comparing workspace states by hand.

Spark-Specific Problems Nobody Warns You About

General CI/CD guidance covers the broad strokes. Spark teams hit problems that live in the details:

Lakehouse bindings are buried in notebook content. The notebook-content.py file contains lakehouse and workspace GUIDs. If your parameter.yml misses even one of these, the production notebook opens to a “lakehouse not found” error. Audit every notebook, including the utility notebooks that other notebooks call with %run. Those hidden dependencies are where the bindings go wrong.

Environment items gate notebook execution. When your Spark notebooks depend on a custom Environment with specific Python libraries or Spark configuration properties, that Environment must exist in the target workspace before the notebooks arrive. The fabric-cicd dependency resolver handles this automatically, but only if Environment is in your item_type_in_scope. Scope just Notebook without Environment, and you’ll get clean deployments followed by runtime failures when the expected libraries don’t exist.

SparkJobDefinitions are not notebooks. SJDs carry executor counts, driver memory settings, reference files, and command-line arguments. All environment-specific values in these properties need coverage in your parameter file. Teams that parameterize their notebooks thoroughly and forget about their SJDs discover the gap when a production batch job runs with dev-sized executors and takes four times longer than expected.

Full deployment at scale needs scoping. Fifty notebooks deploy in minutes. Three hundred notebooks take longer and increase your blast radius. If your workspace has grown large, segment your repository by domain or narrow item_type_in_scope per pipeline to keep deployment times predictable and failures contained to a known set of items.

A Four-Week Migration Path

Starting from zero, here’s a timeline that’s aggressive but achievable:

Week 1: Git integration. Connect your workspace to source control. Rename items that need renaming. Agree on a branching strategy with the team. Write it down. Nothing deploys this week. This is foundation work, and skipping it makes everything after it harder.

Week 2: First deployment. Install fabric-cicd, write your initial parameter.yml, and run a deployment to a test workspace from the command line. Intentionally break the lakehouse binding in the parameter file. See what the error looks like. Fix it. Run it again. You want the team to recognize deployment failures before they encounter one under pressure.

Week 3: Pipeline construction. Build the CI/CD pipeline for Dev-to-Test promotion. Add approval gates, service principal auth, logging, and the verify stage. Run the pipeline ten times. Deliberately introduce a bad merge and watch the pipeline catch it. If it doesn’t catch it, fix the pipeline.

Week 4: Production extension. Extend the pipeline to include Production as a target. Add smoke tests. Test your rollback procedure. Write the runbook. Walk the team through it. Make sure at least two people can operate the pipeline without you in the room.

Four weeks. Not a quarter. Not a planning exercise that stalls in sprint three. A month of focused, methodical work that moves your Spark team from manual deployment to a process that runs the same way every time, whether it’s Tuesday at noon or Saturday at midnight.

The Real Takeaway

Microsoft giving fabric-cicd the official stamp means enterprise teams can stop hesitating. The library will get more attention, faster bug fixes, and broader item type support going forward.

But the tool is only half the story. A perfectly automated pipeline that deploys unparameterized notebooks to the wrong lakehouse is worse than manual deployment, because at least manual deployment forces someone to look at what they’re pushing. Automation works best when it’s built on a disciplined foundation — the checklist, the parameter file, the tested rollback, the verify stage.

Build the checklist. Work the checklist. Invest in the hard parts now, and they’ll pay you back in every deployment after.

This post was written with help from anthropic/claude-opus-4-6

The Spark-to-Warehouse Connector in Fabric: What It Does, How It Breaks, and When to Use It

The Spark-to-Warehouse Connector in Fabric: What It Does, How It Breaks, and When to Use It

There’s a connector that ships with every Fabric Spark runtime. It’s pre-installed. It requires no setup. And it lets your Spark notebooks read from—and write to—Fabric Data Warehouse tables as naturally as they read Delta tables from a Lakehouse.

Most Fabric Spark users don’t know it exists. The ones who do often run into the same three or four surprises. Let’s fix both problems.

What the connector actually is

The Spark connector for Fabric Data Warehouse (synapsesql) is a built-in extension to the Spark DataFrame API. It uses the TDS protocol to talk directly to the SQL engine behind your Warehouse (or the SQL analytics endpoint of a Lakehouse). You get read and write access to Warehouse tables from PySpark, Scala Spark, or Spark SQL.

One line of code to read:

from com.microsoft.spark.fabric.Constants import Constants

df = spark.read.synapsesql("my_warehouse.dbo.sales_fact")


One line to write:

df.write.mode("append").synapsesql("my_warehouse.dbo.sales_fact")


No connection strings. No passwords. No JDBC driver management. Authentication flows through Microsoft Entra—same identity you’re logged into your Fabric workspace with. The connector resolves the SQL endpoint automatically based on workspace context.

That’s the happy path. Now let’s talk about what actually happens when you use it.

Reading: the part that mostly just works

Reading from a Warehouse table into a Spark DataFrame is the connector’s strength. The synapsesql() call supports the full three-part naming convention: warehouse_name.schema_name.table_or_view_name. It works for tables and views, including views with joins across schemas.

A few things that are genuinely useful:

Predicate pushdown works. When you chain .filter() or .limit() onto your DataFrame, the connector pushes those constraints to the SQL engine. You’re not pulling the full table into Spark memory and then filtering—the SQL engine handles the filter and sends back the subset. This matters when your Warehouse tables have hundreds of millions of rows and you only need a time-sliced sample.

df = spark.read.synapsesql("my_warehouse.dbo.sales_fact") \
    .filter("order_date >= '2026-01-01'") \
    .select("order_id", "customer_id", "amount")


Cross-workspace reads work. If your Warehouse lives in a different workspace than your notebook’s attached Lakehouse, you pass the workspace ID:

df = spark.read \
    .option(Constants.WorkspaceId, "<target-workspace-id>") \
    .option(Constants.DatawarehouseId, "<warehouse-item-id>") \
    .synapsesql("my_warehouse.dbo.sales_fact")


This is genuinely powerful for hub-and-spoke architectures where your curated Warehouse sits in a production workspace and your data science notebooks live in a sandbox workspace.

Parallel reads are available. For large tables, you can partition the read across multiple Spark tasks, similar to spark.read.jdbc:

df = spark.read \
    .option("partitionColumn", "order_id") \
    .option("lowerBound", 1) \
    .option("upperBound", 10000000) \
    .option("numPartitions", 8) \
    .synapsesql("my_warehouse.dbo.sales_fact")


This splits the query into eight parallel reads, each fetching a range of order_id. Without this, you get a single-threaded read that will bottleneck on large tables.

Security models pass through. If your Warehouse has object-level security (OLS), row-level security (RLS), or column-level security (CLS), those policies are enforced when Spark reads the data. Your notebook sees exactly what your identity is authorized to see. This is a meaningful difference from reading Delta files directly via OneLake, where security operates at the workspace or folder level.

Custom T-SQL queries work too. You’re not limited to reading tables—you can pass arbitrary T-SQL:

df = spark.read \
    .option(Constants.DatabaseName, "my_warehouse") \
    .synapsesql("SELECT TOP 1000 * FROM dbo.sales_fact WHERE region = 'WEST'")


This is handy for complex aggregations or when you want the SQL engine to do the heavy lifting before data enters Spark.

Writing: the part with surprises

Write support for the Spark-to-Warehouse connector became generally available with Runtime 1.3. It works, and it solves a real architectural problem—but it has mechanics you need to understand.

How writes actually work under the hood

The connector uses a two-phase process:

  1. Stage: Spark writes your DataFrame to intermediate Parquet files in a staging location.
  2. Load: The connector issues a COPY INTO command, telling the Warehouse SQL engine to ingest from the staged files.

This is the same COPY INTO pattern that powers bulk ingestion into Fabric Data Warehouse generally. It’s optimized for throughput. It is not optimized for latency on small writes.

If you’re writing a DataFrame with 50 rows, the overhead of staging files and issuing COPY INTO means the write takes materially longer than you’d expect. For small, frequent writes, this connector is not the right tool. Use T-SQL INSERT statements through a SQL connection instead.

For batch writes of thousands to millions of rows, the connector performs well. The COPY INTO path is what the Warehouse was designed for.

Save modes

The connector supports four save modes:

  • errorifexists (default): Fails if the table already exists.
  • ignore: Silently skips the write if the table exists.
  • overwrite: Drops and recreates the table with new data.
  • append: Adds rows to the existing table.
df.write.mode("overwrite").synapsesql("my_warehouse.dbo.daily_aggregates")


A common pattern: Spark computes daily aggregations from Lakehouse Delta tables, then writes the results to a Warehouse table that Power BI reports connect to. The Warehouse’s result set caching (now generally available as of January 2026) means subsequent Power BI refreshes hit cache instead of recomputing.

The timestamp_ntz gotcha

This is the single most common error people hit when writing to a Warehouse from Spark.

If your DataFrame contains timestamp_ntz (timestamp without time zone) columns, the write will fail. Fabric Data Warehouse expects time-zone-aware timestamps. The fix is a cast before you write:

from pyspark.sql.functions import col

for c in df.columns:
    if dict(df.dtypes)[c] == "timestamp_ntz":
        df = df.withColumn(c, col(c).cast("timestamp"))

df.write.mode("append").synapsesql("my_warehouse.dbo.target_table")


This is not documented prominently enough. If you see a Py4JJavaError during write that mentions type conversion, timestamps are the first thing to check.

What you can’t write to

The connector writes to Warehouse tables only. You cannot write to the SQL analytics endpoint of a Lakehouse—it’s read-only. If you try, you’ll get an error. This seems obvious but trips people up because the same synapsesql() method handles both reads from Warehouses and Lakehouse SQL endpoints.

Private Link limitations

If Private Link is enabled at the workspace level, both read and write operations through the connector are unsupported. If Private Link is enabled at the tenant level only, writes are unsupported but reads still work. This is a significant limitation for security-conscious deployments. Check your network configuration before building pipelines that depend on this connector.

Time Travel is not supported

Fabric Data Warehouse now supports Time Travel queries. However, the Spark connector does not pass through Time Travel syntax. If you need to query a table as of a specific point in time, you’ll need to use a T-SQL connection directly rather than the synapsesql() method.

When to use Warehouse vs. Lakehouse as your serving layer

This is the architectural question that the connector’s existence forces you to answer. You’ve got data in your Lakehouse. Spark has transformed it. Now where does it go?

Use Lakehouse Delta tables when:

  • Your consumers are other Spark notebooks or Spark-based ML pipelines.
  • You need schema evolution flexibility (Delta’s schema merge).
  • You want to use OPTIMIZE, VACUUM, and Z-ORDER for table maintenance.
  • Your data scientists need direct file access through OneLake APIs.

Use Warehouse tables when:

  • Your primary consumers are Power BI reports or T-SQL analysts.
  • You need the Warehouse’s result set caching for repeated query patterns.
  • You need fine-grained security (RLS, CLS, OLS) that passes through to all consumers.
  • You want to use T-SQL stored procedures, views, and MERGE statements for downstream transformations.
  • You need cross-database queries that join Warehouse tables with Lakehouse tables or other Warehouse tables.

Use both when:

  • Spark processes and stores data in the Lakehouse (bronze → silver → gold medallion layers), then the connector writes final aggregations or serving tables to the Warehouse.
  • The Warehouse serves as the “last mile” between your data engineering work and your business intelligence layer.

The January 2026 GA of MERGE in Fabric Data Warehouse makes the “write to Warehouse” pattern significantly more useful. You can now do incremental upserts: Spark writes a staging table, then a T-SQL MERGE reconciles it with the target. This is a common pattern in data warehousing that was previously awkward in Fabric.

A concrete pattern: Spark ETL → Warehouse serving layer

Here’s the pattern I see working well in production:

# 1. Read from Lakehouse Delta tables (Spark native)
bronze = spark.read.format("delta").load("Tables/raw_orders")

# 2. Transform in Spark
silver = bronze.filter(col("status") != "cancelled") \
    .withColumn("order_date", col("order_ts").cast("date")) \
    .withColumn("amount_usd", col("amount") * col("fx_rate"))

gold = silver.groupBy("region", "order_date") \
    .agg(
        count("order_id").alias("order_count"),
        sum("amount_usd").alias("total_revenue")
    )

# 3. Write to Warehouse for Power BI consumption
gold.write.mode("overwrite").synapsesql("analytics_warehouse.dbo.daily_revenue")


The Lakehouse owns the raw and transformed data. Spark does the heavy compute. The Warehouse serves the final tables to downstream consumers with T-SQL access, caching, and fine-grained security.

The alternative—writing gold tables to the Lakehouse and having Power BI connect via the SQL analytics endpoint—also works. But the SQL analytics endpoint has a metadata sync delay after Spark writes new data. The Warehouse table is immediately consistent after the COPY INTO completes. If your reporting needs to reflect the latest pipeline run without a sync lag, the Warehouse path is more reliable.

Cross-database queries: the glue between them

Once you have data in both a Lakehouse and a Warehouse in the same workspace, you can query across them using T-SQL cross-database queries from the Warehouse:

SELECT w.customer_id, w.total_revenue, l.customer_segment
FROM analytics_warehouse.dbo.daily_revenue AS w
JOIN my_lakehouse.dbo.customer_dim AS l
    ON w.customer_id = l.customer_id


This means your Warehouse doesn’t need to contain all the data. It can hold the curated aggregations while joining against dimension tables that live in the Lakehouse. No data movement. No duplication. The SQL engine resolves both sources through OneLake.

Performance notes from the field

A few observations from real workloads:

Reads are faster than you expect. The TDS protocol connection to the Warehouse SQL engine is efficient. For typical analytical queries returning thousands to low millions of rows, the synapsesql() read is competitive with reading Delta files directly, especially when the Warehouse has statistics and result set caching enabled.

Writes are slower than Lakehouse writes. The two-phase staging + COPY INTO process adds overhead versus a direct df.write.format("delta").save() to Lakehouse tables. For a DataFrame with 10 million rows, expect the Warehouse write to take 2-5x longer than an equivalent Lakehouse Delta write. This is the tradeoff for getting immediate T-SQL access with full Warehouse capabilities.

Use parallel reads for large tables. The default single-partition read will bottleneck. Set numPartitions to match your Spark cluster’s available cores for large reads. The performance improvement is often 4-8x.

Proactive and incremental statistics refresh. As of January 2026, Fabric Data Warehouse supports proactive statistics refresh and incremental statistics. This means the query optimizer keeps statistics up to date automatically. Your synapsesql() reads benefit from better query plans without manual UPDATE STATISTICS calls.

The honest summary

The Spark connector for Fabric Data Warehouse is a well-designed bridge between two systems that many teams use side by side. It makes the read path simple and the write path possible without leaving your Spark notebook.

It is not a replacement for writing to Lakehouse Delta tables. It is an additional output path for when your downstream consumers need T-SQL, fine-grained security, result set caching, or immediate consistency. Use it when the Warehouse is the right serving layer. Don’t use it when Lakehouse is sufficient.

The biggest wins come from combining both: Spark for compute, Lakehouse for storage, Warehouse for serving. The connector is the plumbing that makes that architecture work without data pipelines in between.

If you’re heading to FabCon Atlanta (March 16-20, 2026), both the Data Warehouse and Data Engineering teams will be there. It’s a good place to pressure-test your architecture and see what’s coming next.

This post was written with help from anthropic/claude-opus-4-6

Fabric Spark billing just got clearer. Here’s how to make the most of it.

Somewhere in a shared Teams channel, a Fabric capacity admin is looking at the Capacity Metrics app and noticing Spark consumption is down 15% overnight. Same notebooks. Same schedules. Same engineers shipping code with the same amount of caffeine.

A quick thread later, the answer is clear: nothing is wrong. Microsoft introduced new billing operations, and AI usage is now visible in its own category.

That’s not a cost increase. That’s better instrumentation.

What actually changed

On February 13, 2026, Microsoft announced two new billing operations for Fabric: AI Functions and AI Services.

Previously, AI-related usage in notebooks was grouped under Spark operations. Calls made through fabric.functions, Azure OpenAI REST API, the Python SDK, and SynapseML were all reported in Spark. Text Analytics and Azure AI Translator calls from notebooks were also reflected there.

Now those costs are separated:

  • AI Functions covers Fabric AI function calls and Azure OpenAI Service usage in notebooks and Dataflows Gen2.
  • AI Services covers Text Analytics and Azure AI Translator usage from notebooks.

Both are billed under the Copilot and AI Capacity Usage CU meter.

Important: consumption rates did not change. You pay the same for the same work. What changed is visibility.

Why this reporting update is a win for operators

If you’ve ever tried to explain Spark trends that include hidden AI consumption, this update helps immediately.

Picture an F64 capacity. You historically allocated 70% of CU budget to Spark because that’s what Capacity Metrics showed. But Spark previously included AI consumption, so the category was doing two jobs at once.

Now Spark and AI can each tell their own story. That’s useful for:

  • more accurate workload attribution
  • cleaner alerting by operation type
  • better planning conversations with finance and platform teams

In other words: same total spend, sharper signal.

The migration checklist

There’s nothing to deploy and no code changes required. The opportunity is operational: update your monitoring and planning so you can benefit from the new detail right away.

1. Audit your AI function usage

Before the new operations appear in your Metrics app, find AI calls in your codebase. Search notebooks for:

  • fabric.functions calls
  • Azure OpenAI REST API calls (look for /openai/deployments/)
  • openai Python SDK usage within Fabric notebooks
  • SynapseML OpenAI transformers
  • Text Analytics API calls
  • Azure AI Translator calls

If there are no hits, this billing split likely won’t affect your current workloads. If there are many hits (common in mature notebook estates), estimate volume now so your post-change analysis is faster.

2. Baseline your current Spark consumption

Export the last 30 days of Capacity Metrics data for Spark operations and save it.

This is your before-state. After rollout, validate that total consumption (Spark + new AI operations) aligns with historical Spark totals. If it aligns, you’ve confirmed a reporting change. If not, you have a clear starting point for investigation.

3. Adjust your alerting thresholds

If you monitor Spark CU consumption via Capacity Metrics, Azure Monitor, or custom API polling, update thresholds after the split.

Recommended approach:

  • take your current Spark threshold
  • subtract estimated AI consumption from step 1
  • set that as the revised Spark threshold
  • add a separate alert for the Copilot and AI meter

If AI estimates are still rough, start with a conservative threshold and tune after a few weeks of separated data.

4. Update your capacity planning models

Add a dedicated row for AI consumption in any spreadsheet, Power BI report, or planning document that allocates CU budget by operation type.

The Copilot and AI Capacity Usage CU meter already existed for Copilot scenarios, but this may be the first time many Spark-first teams see meaningful workload usage there. Adding it now makes future reviews easier.

5. Set up a validation window

Choose a date after March 17 (when the new operations start appearing) and compare pre/post totals:

  • pre-change: Spark total
  • post-change: Spark + AI Functions + AI Services

Expect close alignment (allowing for normal workload variation and rounding). If variance is more than a few percent, open a support ticket. Microsoft described this as a reporting-only change with no rate modifications.

6. Share a quick team note before questions start

One short update prevents a lot of confusion:

“Microsoft is separating AI consumption from Spark billing into dedicated operations. Total cost is unchanged. Spark will appear lower, and Copilot and AI will appear higher. This improves visibility and tracking.”

That gives engineers context and helps finance teams interpret new categories correctly on day one.

Post-rollout checks that keep things clean

Consumption variance check. If post-change totals (Spark + AI Functions + AI Services) differ significantly from pre-change Spark trends, compare equivalent workload windows and rule out schedule, code, or capacity changes.

Expected operation visibility. If you confirmed AI usage in step 1 but AI Functions shows zero, check regional rollout timing from the Fabric blog before escalating.

Why separated AI spend is valuable

This platform-side categorization update gives teams a better lens on where capacity is being used.

Once AI usage is measurable independently, you can answer higher-quality questions:

  • Which AI workflows are creating the most value per CU?
  • Which calls are production-critical versus experimental leftovers?
  • Where should you optimize first for performance and cost?

That is exactly the kind of visibility mature platform teams want.

What this signals about Fabric billing

As Fabric workloads evolve, billing categories will continue to become more descriptive. That’s a good thing. Better category design means better operational decisions.

The admin in that Teams thread got clarity quickly: Spark wasn’t shrinking, observability was improving. Once the team updated dashboards and alerts, they had a more useful capacity model than they had the week before.

That’s the real upgrade here.

This post was written with help from anthropic/claude-opus-4-6

From Demo to Production: ML-Enriched Power BI in Microsoft Fabric

Microsoft published a new end-to-end pattern last week. Train a model inside Fabric. Score it against a governed semantic model. Push predictions straight into Power BI. No data exports. No credential juggling.

The blog post walks through a churn-prediction scenario. Semantic Link pulls data from a governed Power BI semantic model. MLflow tracks experiments and registers models. The PREDICT function runs batch inference in Spark. Real-time endpoints serve predictions through Dataflow Gen2. Everything lives in one workspace, one security context, one OneLake.

It reads well. It demos well.

But demo code is not production code. The gap between “it runs in my notebook” and “it runs every Tuesday at 4 AM without paging anyone” is exactly where Fabric Spark teams bleed time.

This is the checklist for crossing that gap.

Prerequisites that actually matter

The official blog assumes a Fabric-enabled workspace and a published semantic model. That is the starting line. Production is a different race.

Capacity planning comes first. Fabric Spark clusters consume capacity units. A batch scoring job running on an F64 during peak BI refresh hours competes for the same CUs your report viewers need. Run scoring in off-peak windows, or provision a separate capacity for data science workloads. Either way, know your CU ceiling before your first experiment. Discovering your scoring job throttles the CFO’s dashboard refresh is not a conversation you want to have.

Workspace isolation is not optional. Dev, test, prod. Semantic models promoted through deployment pipelines. ML experiments pinned to dev. Registered models promoted to prod only after validation passes. If your team trains models in the same workspace where finance runs their quarterly close dashboard, you are one accidental publish away from explaining why the revenue numbers just changed.

MLflow model signatures must be populated from day one. The PREDICT function requires them. No signature, no batch scoring. This constraint is easy to forget during prototyping and expensive to fix later. Make it a rule: every mlflow.sklearn.log_model call includes an infer_signature output. No exceptions. Write a pre-commit hook if you have to.

Semantic Link: the part most teams underestimate

Semantic Link connects your Power BI semantic model to your Spark notebooks. Call fabric.read_table() and you get governed data. Same measures and definitions your business users see in their reports. The data in your model’s training set matches what shows up in Power BI.

This matters more than it sounds.

Every analytics team that has been around long enough has a story about metric inconsistency. “Active customer” means one thing in the DAX model, another thing in the SQL pipeline, and a third thing in the data scientist’s Python notebook. The numbers diverge. Somebody notices. A week of forensic reconciliation follows.

Semantic Link kills that problem at the root. But only if you use it deliberately.

Start with fabric.list_measures(). Audit what DAX measures exist. Understand which ones your model depends on. Then pull data with fabric.read_table() rather than querying lakehouse tables directly. When you need to engineer features beyond what the semantic model provides, document every derivation in a version-controlled notebook. Written down and committed. Not living in someone’s memory or buried in a thread.

Training guardrails worth building

The Fabric blog shows a clean LightGBM training flow with MLflow autologging. That is the happy path. Production needs the unhappy path covered too.

Validate data before training. Check row counts against expected baselines. Check for null spikes in key columns. Check that the class distribution has not shifted beyond your predefined threshold. A model trained on corrupted or stale data produces confident garbage. Confident garbage is worse than no model at all, because people act on it.

Tag every experiment run. MLflow in Fabric supports custom tags. Use them aggressively. Tag each run with the semantic model version it pulled from, the notebook commit hash, and the data snapshot date. Three months from now, when a stakeholder asks why the model flagged 200 customers as high churn risk and zero of them actually left, you need to reconstruct exactly what happened. Without tags, you are guessing.

Build a champion-challenger gate. Before any new model version reaches production, it must beat the current model on a holdout set from the most recent data. Not any holdout set. The most recent one. Automate this comparison in a validation notebook that runs as a pipeline step before model registration. If the challenger fails to clear the margin you defined upfront, the pipeline halts. No override button. No “let’s just push it and see.” The gate exists to prevent optimism from substituting for evidence.

Batch scoring: the PREDICT function in production

Fabric’s PREDICT function is straightforward. Pass a registered MLflow model and a Spark DataFrame. Get predictions back. It supports scikit-learn, LightGBM, XGBoost, CatBoost, ONNX, PyTorch, TensorFlow, Keras, Spark, Statsmodels, and Prophet.

The production requirements are few but absolute.

Write predictions to a delta table in OneLake. Not to a temporary DataFrame that dies with the session. Partition that table by scoring date. Add a column for the model version that generated each row. This is your audit trail. When someone asks “why did customer 4471 show as high risk last Tuesday?”, you pull the partition, check the model version, and have an answer in minutes. Without that structure, the same question costs you a day.

Chain your scoring job to run after your semantic model refresh. Sequence matters. If the model scores data from the prior refresh cycle, your predictions are one step behind reality. Use Fabric pipelines to enforce the dependency explicitly. Refresh completes, scoring starts.

Real-time endpoints: know exactly what you are signing up for

Fabric now offers ML model endpoints in preview. Activate one from the model registry. Fabric spins up managed containers and gives you a REST API. Dataflow Gen2 can call the endpoint during data ingestion, enriching rows with predictions in flight.

The capability is real. The constraints are also real.

Real-time endpoints support a limited set of model flavors: Keras, LightGBM, scikit-learn, XGBoost, and (since January 2026) AutoML-trained models. PyTorch, TensorFlow, and ONNX are not supported for real-time serving. If your production model uses one of those frameworks, batch scoring is your only path.

The auto-sleep feature deserves attention. Endpoints scale capacity to zero after five minutes without traffic. The first request after sleep incurs a cold-start delay while containers spin back up. For use cases that need consistent sub-second latency, you have two options: disable auto-sleep and accept the continuous capacity cost, or send periodic synthetic requests to keep the endpoint warm.

The word “preview” is load-bearing here. Preview means the API can change between updates. Preview means SLAs are limited. Preview means you need a batch-scoring fallback in place before you route any production workflow through a real-time endpoint. Build the fallback first. Test it. Then add the real-time path as an optimization on top.

The rollback plan you need to write before you ship

Most teams build forward. They write the training pipeline, the scoring job, the endpoint, the Power BI report that consumes predictions. Then they ship.

Nobody writes the backward path. Until something goes wrong.

Your rollback plan has three parts.

First, keep at least two prior model versions in the registry. If the current version starts producing bad predictions, you roll back by updating the model alias. One API call. The scoring pipeline picks up the previous version on its next run.

Second, partition prediction tables by date and model version. Rolling back a model means nothing if downstream reports still display the bad predictions. With partitioned tables, you can filter or drop the scoring run from the misbehaving version and revert to the prior run’s output.

Third, a kill switch for real-time endpoints. One API call to deactivate the endpoint. Traffic falls back to the latest batch-scored delta table. Your Power BI report keeps working, just without real-time enrichment, while you figure out what went wrong.

Test this plan. Not on paper. Run the rollback end to end in your dev environment. Time it. If reverting to a stable state takes longer than fifteen minutes, your plan is too complicated. Simplify it until the timer clears.

Ship it

The architecture Microsoft described is sound. Semantic Link for governed data access. MLflow for experiment tracking and model registration. PREDICT for batch scoring to OneLake. Real-time endpoints for low-latency enrichment. Power BI consuming prediction tables through DirectLake or import.

But architecture alone does not keep a system running at 4 AM. The capacity plan does. The workspace isolation does. The data validation gate, the champion-challenger check, the scoring sequence, the endpoint fallback, the rollback drill. Those are what separate a demo from a service.

Do the checklist. Test the failure modes. Then ship.

This post was written with help from anthropic/claude-opus-4-6

Microsoft Fabric Warehouse + Spark: Interoperability Patterns That Actually Work

If you’ve spent any time in a Fabric workspace with both Data Engineering (Spark) and Data Warehouse, you’ve probably had this moment:

  • Spark is great for big transformations, complex parsing, and “just let me code it.”
  • The Warehouse is great for a curated SQL model, concurrency, and giving the BI world a stable contract.
  • And yet… teams still end up copying data around like they’re paid by the duplicate.

The good news: Fabric’s architectural bet is that OneLake + Delta is the contract surface across engines. That means you can design a pipeline where Spark and Warehouse cooperate instead of competing.

This post is a practical field guide to the integration patterns that work well in real projects:

  1. 3-part naming over the SQL endpoint (zero-copy default) – query Lakehouse Delta tables directly from Warehouse SQL without moving data.
  2. Spark → Warehouse (file-based ingest) using COPY INTO and OPENROWSET over OneLake paths – when workload evidence calls for materialization.
  3. Spark → Warehouse (table-to-table ingest) using cross-database queries / CTAS / INSERT…SELECT – same trigger.
  4. Warehouse → Spark (read-only consumption) by reading the Warehouse table’s published Delta logs from Spark.

Along the way, I’ll call out the trade-offs, the gotchas, and the operational guardrails that keep teams out of trouble.

Mental model: OneLake is the handshake

In Fabric, multiple experiences can produce and consume Delta Lake tables. Microsoft Learn describes Delta Lake as the standard analytics table format in Fabric, and notes that Delta tables produced by one engine (including Fabric Data Warehouse and Spark) can be consumed by other engines.

So instead of thinking “Spark output” and “Warehouse tables” as two unrelated worlds, treat them as:

  • A shared storage plane (OneLake)
  • An open table format (Delta + Parquet)
  • Two compute engines with different strengths

The rest is just choosing where to materialize — or whether to materialize at all.

Start here: 3-Part Naming over the SQL Endpoint

Before you copy anything, ask: do I actually need a separate materialized table?

Fabric’s SQL analytics endpoint automatically exposes every Lakehouse Delta table as a queryable SQL object. From the Warehouse, you can reference those tables directly using 3-part naming:

SELECT *
FROM MyLakehouse.dbo.clean_sales
WHERE OrderDate >= '2026-01-01';

No COPY INTO. No CTAS. No duplicate storage. The query runs against the Lakehouse’s Delta files through the SQL endpoint — zero-copy interoperability out of the box.

When this is enough (and it often is)

  • Ad-hoc analytics and exploration across Spark-produced datasets.
  • Lightweight joins between Warehouse dimensions and Lakehouse facts.
  • BI semantic models that don’t need sub-second concurrency at scale.
  • Early-stage projects where the workload profile isn’t settled yet.

When to materialize instead

Materialize into dedicated Warehouse tables (COPY INTO, CTAS, INSERT…SELECT) when workload evidence justifies it:

  • High concurrency: many concurrent queries hitting the same dataset consistently.
  • Recurring heavy joins/aggregations: repeated complex queries where pre-materialized tables measurably reduce compute.
  • Stricter SLA / CU predictability: when you need tighter control over query performance and capacity consumption.
  • Governance boundaries: when the Warehouse should own and version the serving-layer schema independently from the Lakehouse.

If none of those conditions apply, 3-part naming is the right default. You can always materialize later when the numbers say you should.

The CU tradeoff

Virtualization (3-part naming) shifts cost to query-time: every read traverses the SQL endpoint and pays CU at execution. Materialization (COPY INTO / CTAS) pays an ingestion and storage cost once, so repeated reads are faster and more predictable in CU terms. Neither is universally better — the right call depends on query frequency, data volume, and your capacity budget.

Pattern 1 — Spark → Warehouse via OneLake files (COPY INTO + OPENROWSET)

When to use it

Start with 3-part naming. Reach for COPY INTO / OPENROWSET file-based ingest only when workload evidence (sustained concurrency pressure, SLA requirements, or CU unpredictability) tells you virtualization isn’t enough. This pattern fits when:

  • Your Spark pipeline already produces files (Parquet/CSV/JSONL) under a Lakehouse Files path.
  • You need faster or more predictable query performance than the SQL endpoint provides for this dataset.
  • You want a clean separation: Spark writes files; Warehouse owns the serving tables.

Step 1: Write a “handoff” dataset from Spark

In Spark, write a handoff dataset into the Lakehouse Files area (not Tables). Conceptually:

(
  df
  .write
  .mode("overwrite")
  .format("parquet")
  .save("Files/handoff/sales_daily/")
)


Why Files? Because the Warehouse can point COPY INTO / OPENROWSET at file paths, and the Files area is designed to hold arbitrary file layouts.

Step 2: Inspect the file shape from the Warehouse (OPENROWSET)

Before you ingest, use OPENROWSET to browse a file (or a set of files) and confirm the schema is what you think it is.

Microsoft Learn documents that Fabric Warehouse OPENROWSET can read Parquet/CSV files, and that the files can be stored in Azure Blob Storage, ADLS, or Fabric OneLake (with OneLake reads called out as preview).

SELECT TOP 10 *
FROM OPENROWSET(
  BULK 'https://onelake.dfs.fabric.microsoft.com/<workspaceId>/<lakehouseId>/Files/handoff/sales_daily/*.parquet'
) AS rows;


Step 3: Ingest into a Warehouse table (COPY INTO)

The Fabric blog announcement for OneLake as a source for COPY INTO and OPENROWSET highlights the point of this feature: load and query Lakehouse file folders without external staging storage or SAS tokens.

COPY INTO dbo.SalesDaily
FROM 'https://onelake.dfs.fabric.microsoft.com/<workspaceId>/<lakehouseId>/Files/handoff/sales_daily/'
WITH (
  FILE_TYPE = 'PARQUET'
);


Operational guardrails

  • Treat the Files path as a handoff contract: version it, keep it predictable, and don’t “just drop random stuff in there.”
  • If you’ll query the same external data repeatedly, ingest it into a dedicated Warehouse table (Microsoft Learn notes repeated OPENROWSET access can be slower than querying a table).

Pattern 2 – Spark → Warehouse via in-workspace tables (CTAS / INSERT…SELECT)

When to use it

As with Pattern 1, start with 3-part naming and materialize via CTAS / INSERT…SELECT only when workload metrics confirm you need it. This pattern fits when:

  • Your Spark output is naturally a Delta table (Lakehouse Tables area) and 3-part naming queries against it hit concurrency or performance limits.
  • You want the Warehouse to own a curated serving-layer model (joins, dimensional modeling, computed columns) with predictable CU spend.
  • You prefer SQL-native table-to-table pipelines over file-level ingestion.

Step 1: Produce a curated Delta table with Spark

(
  df_clean
  .write
  .mode("overwrite")
  .format("delta")
  .save("Tables/clean_sales")
)


Step 2: Materialize a Warehouse table from the Lakehouse table

Microsoft Learn notes that for T-SQL ingestion, you can use patterns like INSERT…SELECT, SELECT INTO, or CREATE TABLE AS SELECT (CTAS) to create or update tables from other items in the same workspace (including lakehouses).

CREATE TABLE dbo.FactSales
AS
SELECT
  OrderDate,
  StoreId,
  ProductId,
  Quantity,
  NetAmount
FROM MyLakehouse.dbo.clean_sales;


For incremental loads you’ll often end up with a staging + merge strategy, but the key idea stays the same: Spark produces the curated dataset; the Warehouse owns the serving tables.

Pattern 3 – Warehouse → Spark via published Delta logs (read-only)

This is the pattern that surprises people (in a good way): the Warehouse isn’t a closed box.

Microsoft Learn documents that Warehouse user tables are stored in Parquet, and that Delta Lake logs are published for all user tables. The key consequence is that any engine that can read Delta tables can get direct access to Warehouse tables – read-only.

Step 1: Get the OneLake path for a Warehouse table

In the Warehouse UI, table Properties exposes the table’s URL / ABFS URI (Learn walks through the steps).

Step 2: Read the Warehouse table from Spark (read-only)

warehouse_table_path = "abfss://<workspace>@onelake.dfs.fabric.microsoft.com/<warehouseId>/Tables/dbo/FactSales"

fact_sales_df = spark.read.format("delta").load(warehouse_table_path)

  • This access is read-only from Spark. Writes must go through the Warehouse to maintain ACID compliance.
  • Delta log publishing is a background process after commits, so treat cross-engine visibility as “near real-time,” not “every millisecond.”

Bonus control: pause Delta log publishing

The same Learn doc describes an operational lever you can use when you need stability during a large set of changes:

ALTER DATABASE CURRENT SET DATA_LAKE_LOG_PUBLISHING = PAUSED;
-- ... bulk updates ...
ALTER DATABASE CURRENT SET DATA_LAKE_LOG_PUBLISHING = AUTO;


When publishing is paused, other engines see the pre-pause snapshot; Warehouse queries still see the latest.

Choosing an ownership model (so you don’t end up with two sources of truth)

The integration is easy. The contract is the hard part.

A simple rule that prevents a lot of pain:

  • If Spark is writing it: Warehouse can ingest it, but Spark owns the dataset.
  • If Warehouse is writing it: Spark can read it, but Warehouse owns the dataset.

In other words: pick one writer.

For most analytics teams, a good default is:

  • Spark owns bronze/silver (raw + cleaned Delta in the Lakehouse)
  • Warehouse owns gold (facts/dimensions, KPI-ready serving tables) — but “owns” doesn’t always mean “physically copies.” A cross-database query via 3-part naming can serve gold-layer reads without materialization.

Start with 3-part naming for cross-engine reads. Materialize across the boundary only when workload metrics — not assumptions — tell you to. Remember: virtualization shifts CU cost to query-time; materialization front-loads ingestion and storage so repeated reads are cheaper and more predictable. Let your actual usage patterns decide.

Quick checklist: production-hardening the Spark ↔ Warehouse boundary

  • Make the handoff explicit (a specific Files path or a specific Lakehouse table).
  • Version your schema (breaking changes should be intentional and tested).
  • Avoid singleton inserts into Warehouse; prefer bulk patterns (CTAS, INSERT…SELECT).
  • Validate row counts and freshness after each load (and alert on drift).
  • Treat Delta log publishing as eventual across engines; design your BI/ML expectations accordingly.

Summary

Fabric is at its best when you let each engine do what it’s good at:

  • Spark for transformation, enrichment, and complex data engineering logic.
  • Warehouse for the curated serving model and SQL-first consumers.

OneLake + Delta is the glue. Start with 3-part naming for zero-copy interoperability across engines, and materialize only when workload evidence justifies the extra storage and ingestion cost. That way you get the simplicity of one logical data layer without paying for copies you don’t need.

This post was written with help from Opus 4.6

References