Christopher Finlan

Most Spark notebook trouble is not caused by the notebook. It is caused by the little files loitering around it.

A notebook runs fine in dev. In test, it quietly depends on a config file somebody copied by hand. In prod, a helper script lives in a workspace folder nobody remembers creating. Then the team spends an afternoon acting baffled. This happens so often it barely even feels embarrassing anymore.

That is why Resources folder support in Git for Fabric notebooks matters. The official notebook source control and deployment docs now describe notebook resources living with the notebook in source control. Pair that with Fabric Git integration, which syncs workspace items with a Git repository for version control and collaboration, and the point lands pretty hard: Fabric is getting better at treating notebooks like real engineering artifacts instead of magical documents that somehow run businesses by optimism alone.

What changed

In plain English, supporting notebook files can now sit with the notebook in Git-backed workflows instead of floating around just outside the process.

That sounds small. It is not.

Once those files live in the repo, they show up in normal history and pull-request diffs. That is one of the verified, useful bits here. A reviewer can see the notebook change and the supporting-file change in the same place, at the same time, during the same discussion. For teams that have ever been burned by a side file nobody reviewed, that is not cosmetic. It is the whole game.

Why Spark teams should care

Spark notebook work grows side files the way garages grow mystery boxes. At first it is one harmless config. Then a helper module. Then a mapping file. Then a test fixture that was supposed to be temporary around the same time fax machines were supposed to be temporary.

The notebook stays visible. The dependencies do not.

That gap is where trouble starts. If the notebook looks versioned but the files it depends on are living somewhere else, you have a comforting illusion, not a reliable workflow.

Resources in Git makes the workflow more honest. The notebook is still the star of the show, but the supporting cast is finally on stage where people can see it.

What gets better in practice

I would not pretend this feature solves deployment. It does not. Git has never once saved a team from bad habits.

What it does do is remove one very stupid source of drift.

A few practical wins follow from that:

• Review gets better because the supporting files are visible in the same repo flow.
• Collaboration gets better because Fabric Git integration already uses the repository as the shared system of record.
• Change history gets better because notebook-supporting files now live in normal repository history instead of in folklore, Slack messages, or somebody’s Downloads folder.

None of that is glamorous. Good. Glamour is overrated. Inspectable systems are better.

What to do with the feature this week

Use it to tighten the contract around a production notebook. Do not use it as an excuse to create a beautifully organized junk drawer.

Be picky about what belongs in Resources

If a file materially affects how the notebook runs, how the team validates it, or how someone maintains it at 2 a.m., it is probably a candidate.

That usually includes things like config templates, helper code tied to the notebook, schema or mapping files, small test fixtures, and operating notes.

It does not include secrets, giant raw datasets, or random leftovers that nobody can defend with a straight face.

Review the file changes, not just the notebook

This is the part teams will skip if nobody insists on it.

If a notebook change also touches supporting files, review those files with the same seriousness. The feature is only useful if the newly visible files actually get looked at.

Standardize the layout before entropy arrives in a nice shirt

Pick a predictable folder pattern and keep it boring. Boring wins. Boring lets an on-call engineer guess where the relevant files live without a spiritual journey.

Test one real notebook, not a toy example

Take a notebook that already depends on side files. Move the right files into the repo-backed structure. Review the diff. Run it through the Git workflow your team already uses. Then turn that into the standard.

Anything else is just a product demo performed for yourselves.

The traps

This feature will make some teams better. It will also reveal which teams have been winging it.

The first trap is duplicate truth. If the same file or logic exists in three places, Git is not going to sort out the argument for you. Pick the real source of truth and cut over cleanly.

The second trap is false confidence. A file being in Git does not magically make your release process good. It just means the file is now versioned, which is a solid start and not a miracle.

The third trap is resource sprawl. Give engineers a new folder and, sooner or later, they will try to store civilization in it. Set limits early.

The real value

This is not a flashy announcement. It is a practical one. Those are usually the announcements worth paying attention to.

Fabric notebooks now have Resources folder support in Git. Fabric Git integration already syncs workspace items with repositories for version control and collaboration. And once notebook-supporting files live there too, their changes show up in repository history and pull-request diffs.

That is enough to matter.

For production Spark teams, the payoff is simple: fewer hidden dependencies, less hand-carried nonsense, and a notebook workflow that looks a little more like software engineering and a little less like amateur archaeology.

Official docs worth keeping open

• Notebook source control and deployment
• Overview of Fabric Git integration
• Get started with Git integration
• Git integration process
• Git source code format

This post was written with help from anthropic/claude-opus-4-6

Most Fabric deployment pain is not dramatic. It is slow, dumb, and expensive in the worst way. Somebody asks you to move a workspace full of notebooks, pipelines, reports, and models. Then the afternoon disappears into portal clicking, second-guessing, and the private terror that you forgot one dependency that will blow up later.

That is why the new bulk item-definition APIs matter.

Not because they are flashy. They are not. Not because they are finished. The official docs call both APIs beta and say they are for evaluation and development purposes, not recommended for production use. Good. Honesty is refreshing.

They matter because Fabric finally has official APIs for moving multiple item definitions in and out of a workspace in one operation. And the broader item definition overview says the quiet part out loud: definition-based APIs matter for fully automated deployment and bulk migrations. That is the operational opening teams have been waiting for.

First, what an “item definition” actually is

Fabric’s docs define an item definition as the structured set of files and metadata that describe how a Fabric item is built. Different item types have different formats and required parts.

That sounds abstract until you look at the wire format. In the Get Item Definition docs, a definition comes back as parts. Each part has a path, a payload, and a payload type. The sample uses InlineBase64. The platform file lives in that world too.

So no, this is not one magic blob. It is closer to a folder tree poured into JSON. Files, paths, and encoded content. The kind of thing automation can actually move without a human babysitter.

The supported-definition list is not trivial either. The overview includes notebooks, lakehouses, reports, semantic models, data pipelines, KQL dashboards, eventstreams, environments, and Spark job definitions, among others. If you live in Spark-heavy workspaces, that last piece matters.

What the bulk APIs actually do

Bulk Export Item Definitions (beta) lets you export item definitions from a workspace in a single operation. You can export all supported items or pass a specific list.

Bulk Import Item Definitions (beta) does the inverse. It imports multiple item definitions into a workspace, and the docs say the system will create new items or update existing ones based on whether the item already exists.

That is the boring sentence with teeth.

The export shape is practical. The sample response includes an itemDefinitionsIndex with item IDs and root paths, plus a definitionParts collection with file paths and InlineBase64 payloads. In other words, this is not portal smoke. It is structured material you can inspect, store, and move.

Why this changes the day job

Microsoft’s own overview says definition-based APIs matter for automated deployment and bulk migrations. That is not fluff. That is the foundation.

A sane workflow now looks like this:

1. Export item definitions from a dev workspace.
2. Store those definitions somewhere you can inspect and review.
3. Validate what changed.
4. Import the definitions into the next workspace.

Notice what I did not say. I did not say these APIs solve release management for you. They do not. They give you raw material. You still need naming discipline, environment strategy, and release gates. The API gives you lumber. It does not build the house.

But before this, a lot of Fabric promotion work still felt like moving furniture through a keyhole. Now it looks a lot more like files and operations, which is exactly what mature platform teams want.

The caveats are not optional

This is where a lot of blog posts start lying. Let’s not do that.

It is beta

The docs are explicit. Both APIs are beta. Both require beta=true in the query string. Both are described as evaluation and development features and not recommended for production use.

So if your first move is wiring this straight into your most fragile production deployment, that is not bold. That is sloppy.

Pilot it first. Use a low-risk workspace. Learn the payloads. Prove your rollback story. Then decide how far you want to push it.

Permissions will make or break your export

For both APIs, the caller needs a Contributor or higher role on the workspace. For delegated auth, the required scope is Items.ReadWrite.All.

The subtle trap is export completeness. When you export all items in a workspace, the docs say only items the caller has both read and write permissions for are exported. If you export a hand-picked list, the caller needs read and write permissions for every item on that list.

That means you can get a successful response and still end up with an incomplete export.

That is the kind of bug that ruins an evening.

If your item count looks light, do not start with conspiracy theories. Start with permissions.

App-only automation has a catch

Yes, the bulk APIs support user identities. They also support service principals and managed identities, but only when all items involved support service principals.

That caveat matters. It means the dream of fully headless CI/CD is real, but it is not universal. One unsupported item type in the batch can turn a clean automation story into a mess.

Check item support early, not the night before a demo.

These are long-running operations

Both bulk APIs use Fabric’s long-running operation pattern. Sometimes you get 200 OK. Sometimes you get 202 Accepted, plus a Location header, an x-ms-operation-id, and a Retry-After header.

That tells you exactly how to build the client:

• submit the request
• poll the operation status using the provided location or operation ID
• wait the number of seconds in Retry-After
• fetch the result when the operation succeeds

This is not the place for impatient, hard-coded polling loops. The service already told you how to behave. Listen to it.

Imports can fail in very ordinary ways

The bulk import docs list a few common errors worth taping to the wall:

• DuplicateDisplayNameAndType
• DependenciesCouldNotBeResolved
• InvalidFilesPath

The bulk export docs call out failures like ItemsHaveProtectedLabels too.

None of these are exotic. They are exactly the problems teams create when naming gets loose, paths drift, or governance details get ignored.

Why Spark teams should care

If your Fabric world revolves around Spark, this is the part worth circling in red.

The item definition overview includes notebooks and Spark job definitions in the supported definition-based universe. That means core Spark artifacts are moving closer to a model that automation can export, inspect, and promote in bulk.

That does not replace Git. It does not replace testing. It does not replace competent release discipline.

What it does replace is some of the dumb manual glue work. And frankly, that glue work has been stealing time from real engineering for too long.

When a platform cannot expose important artifacts as structured, movable definitions, every promotion feels a little haunted. You can do it. You just never fully trust it. Bulk import and export do not make that anxiety disappear, but they finally give Fabric teams firmer ground.

A better first move than “let’s automate everything”

If I were rolling this out today, I would keep it simple:

1. Export all supported item definitions from a test workspace.
2. Inspect the returned root paths and definition parts so you understand the structure.
3. Re-import them into another test workspace.
4. Validate what was created and what was updated.
5. Only then test service-principal execution and larger promotion flows.

Small, boring rehearsals beat heroic rollout plans every time.

The bottom line

These APIs are not glamorous. They are not finished. Microsoft is being quite clear about that.

But they are operationally important.

Fabric now has official bulk APIs for item definitions. The docs explicitly tie definition-based APIs to automated deployment and bulk migrations. For teams managing notebooks, pipelines, reports, semantic models, and Spark assets across workspaces, that is a real shift.

Not a promise. A shift.

It means Fabric is getting better at the thing serious teams need most: turning workspace assets into something you can move, review, and automate without a human performing portal surgery at midnight.

Official docs worth keeping open

• Overview of item definitions
• Get Item Definition
• Bulk Export Item Definitions (beta)
• Bulk Import Item Definitions (beta)
• Long running operations in Fabric REST APIs

This post was written with help from anthropic/claude-opus-4-6

Microsoft’s monthly feature summaries have a familiar problem. They flatten every change into the same cheerful pitch. A new cell editor mode gets about the same oxygen as a moving security boundary. If you run Spark seriously on Fabric, that is useless. You need to know which items change architecture, which clean up the daily notebook grind, and which quietly add a new failure mode.

February’s release has all three. The headline is not “more features.” The headline is that Fabric keeps removing excuses for portal-driven, manually operated Spark environments. More of the platform can now be secured, composed, and managed through code. That is good news. It also means the easier Microsoft makes this, the more discipline you need on your side.

The change that actually alters architecture

CMK support for notebook code

This is the big one.

Fabric notebooks can now run inside CMK-enabled workspaces, with notebook content and associated notebook metadata encrypted at rest using customer-owned keys in Azure Key Vault. Microsoft is not vague about the coverage. The post calls out cell source, cell output, and cell attachments.

If your team has been splitting its development pattern because notebooks were the odd object out in a tighter security model, that split is no longer structurally required. Plenty of enterprises ended up with an awkward arrangement: secure workspaces for governed assets, then a side channel for notebook authoring and iteration. February closes that gap.

The payoff is boring in the best way. Fewer workarounds. Fewer places where permissions drift. Fewer security reviews where someone has to explain why the code path lives outside the workspace standard applied to everything else.

It also changes the migration conversation. Teams that avoided notebooks in regulated environments can revisit that decision. Teams already on notebooks can ask whether a separate architecture still buys them anything except paperwork.

The catch is operational, not conceptual. Keys rotate. Policies get tightened. When notebook content and metadata sit under the same CMK envelope, key management stops being an abstract security exercise and starts touching the authoring surface your engineers use every day. If you do not test rotation and recovery in a non-production workspace first, you are volunteering to learn in public.

The workflow fix Spark teams needed months ago

Python notebooks finally get %run

This was overdue.

PySpark notebooks had a workable modularity story. Python notebooks did not. If you wanted shared setup logic, common helper functions, or a standardized preamble, you either copied code between notebooks or invented a packaging scheme to compensate for a missing primitive.

Now Python notebooks support %run. You can reference and execute other notebooks in the same execution context, then directly use the functions and variables defined there. That is the difference between notebook code as a pile of local accidents and notebook code as something you can organize on purpose.

There is one limitation, and it matters: today %run in Python notebooks supports notebook items only. It does not yet run .py modules from the notebook resources folder. Microsoft says that support is coming soon. Fine. “Coming soon” is not an architecture. Build around notebook references now, and treat resource-folder module execution as a future upgrade if it arrives on time.

The immediate move for most teams is simple. Pull duplicated utility code into shared notebooks. Keep them small. Keep ownership clear. Do not turn %run into a dependency swamp where every notebook imports half the workspace and nobody can explain execution order without drawing a crime-scene diagram.

Version history now tells you where a change came from

This sounds like a minor quality-of-life improvement until you have to debug a bad deployment before the second cup of coffee.

Fabric notebook version history now labels the source of each saved version. Direct edits in the notebook, Git synchronizations, deployment pipeline updates, and publishing via VS Code all show up as distinct origins. That one label removes a stupid amount of ambiguity.

Before this, the question “what changed?” was followed by the more annoying question “through which path?” In a serious CI/CD setup, that distinction is the whole investigation. A manual portal edit points you to one human. A Git sync points you to a repo change. A deployment pipeline update points you to release plumbing. VS Code publishing points you somewhere else again. Same broken notebook, different root cause.

If your team uses more than one of these paths, update the runbook. The first step in notebook incident triage should now be checking the version source before anyone starts diffing content like a raccoon digging through a dumpster.

Full-size mode is small, but not trivial

Full-size mode lets a single notebook cell fill the workspace for editing. That is not glamorous. It is just useful.

Large SQL blocks, ugly transformation cells, and screenshared code reviews all get easier when the interface stops fighting you. Features like this do not make press-release people happy, but they do shave friction off work that happens every day. I would not redesign an architecture around it. I would absolutely use it.

The broader pattern hiding inside the release

Fabric is making Spark more reachable from both directions

Two February items matter together.

The new Microsoft ODBC Driver for Fabric Data Engineering gives external applications and ODBC-compatible tools a supported path into Spark SQL on Fabric. Microsoft describes it as ODBC 3.x compliant, backed by Livy APIs, and built for OneLake and Lakehouse data with Entra ID authentication, proxy support, session reuse, and Spark SQL coverage that looks designed for real workloads instead of demos.

Then there is Semantic Link 0.13.0. That release expands management coverage across lakehouses, reports, semantic models, SQL endpoints, and Spark. Microsoft is explicit about the direction: creating and managing lakehouses and tables, cloning and rebinding reports, refreshing and monitoring semantic models, and administering SQL and Spark settings from code.

Put those together and the platform’s direction is obvious. Fabric wants Spark environments that can be queried from outside and administered from inside code, without the portal as the center of the universe. That is the right direction. The portal is useful. The portal is not a control plane.

This is also where teams get themselves into trouble. The moment workspace operations become scriptable, governance stops being a policy deck and becomes a permissions design problem. If every engineer can programmatically create lakehouses, modify Spark settings, and rebind reports, then congratulations: you have built an accidental infrastructure platform. Maybe that is fine. Maybe it is a terrible idea. Decide before the scripts proliferate.

My bias is blunt. Treat Semantic Link as production infrastructure tooling, not as a convenience library. Set conventions early. Define who can do what. Log changes. Review the scripts that touch shared assets. Otherwise you will end up with beautiful automation and feral workspaces.

The quiet footgun in the admin section

Fabric identity limits now scale higher, but Fabric will not save you from bad math

Fabric now raises the default tenant limit for Fabric identities from 1,000 to 10,000. That is a real scale change, and for some organizations it removes an artificial ceiling that was starting to pinch.

It also lets admins set custom limits and manage them through the Update Tenant Setting REST API. Good. That is how this should work.

The problem is the warning Microsoft slips into the text: Fabric does not validate whether your custom limit fits within your Entra ID resource quota.

That means the setting feels authoritative while depending on an external quota boundary it does not enforce. In other words, the UI and API will happily let you declare ambition. Entra ID is the system that decides whether ambition has a permit.

So before anyone bumps the limit because “10,000 sounds better,” check the Entra side first. If you automate the setting, add that quota check to the automation. This is not exotic engineering. It is basic adult supervision.

What I would do this week

If you own Spark on Fabric, February’s release suggests a short, unromantic punch list.

• Review whether CMK support lets you collapse any split workspace pattern built around notebook restrictions.
• Start using %run in Python notebooks for shared helpers, but keep the dependency graph understandable.
• Update notebook incident runbooks so version-source labels are part of first response.
• Decide whether the ODBC driver and Semantic Link belong in your standard platform toolkit, then put guardrails around both before usage spreads.
• Check Entra ID quotas before changing Fabric identity limits, especially if a script is going to do it for you.

That is the real shape of the month. A nicer notebook editor is fine. A new driver is nice. The deeper story is that Fabric keeps shifting Spark toward a model where security, reuse, and administration happen in code instead of in tribal knowledge and portal muscle memory. That is progress. It also means the teams that win will be the ones that pair new capability with restraint, because the platform is getting powerful enough to automate your mistakes at scale.

This post was written with help from anthropic/claude-opus-4-6