Real agents, real edge cases: tuning, evaluating, and breaking my golf lakehouse

Part 2 of a hands-on data + AI series built on a small, real dataset. In which I add new data, break almost everything, learn the production-shaped lessons the hard way, and end up with an agent that's measurably smarter — and an honest account of what broke along the way.

Lakehouse series: Part 1 · From spreadsheets to a real Lakehouse · Part 1.5 · Genie learns · Part 2 · Real agents, real edge cases (you are here)

---

A quick note on why

A few weeks before I started this post, I accepted a Field Engineering leadership role at Databricks. After nearly nine years at Microsoft, where most of my time was spent helping customers run AI workloads on Azure, the move felt like a natural alignment of two things I've cared about for a long time: agentic AI that genuinely earns its keep and structured data done with discipline.

I wrote Part 1 of this series in the seam between those two chapters — partly to refresh my platform knowledge end to end before showing up to my new team, partly because the best way I know to learn something is to build something real with it. The first post took a weekly golf league SaaS, extracted its data, and stood up a personal lakehouse on Azure Databricks with Unity Catalog, Lakeflow, a Genie space, and a Mosaic AI agent on top.

This post is what happens after that. I added two new data sources to make the agent smarter, broke about six things in the process, learned the lessons I needed to learn before I get my Databricks badge, and ended up with a working agent and a real eval baseline I can actually defend in a code review.

If you're new here — Part 1 sets up the lakehouse from scratch. This post assumes that foundation exists and goes from "the agent works on the easy questions" to "the agent stands up to a structured eval."

---

Where we left off

At the end of Part 1, the caddie agent could answer questions like:

• "Who has the lowest scoring average this season?"
• "What can Jeff Baart do to improve his game?"
• "Who's hot lately?"

That worked. It also barely scratched what the agent could be. A real caddie sees a lot more than your scorecard — they know the course inside and out and they read conditions before they ever hand you a club. So this post tackles two ideas:

1. Add the data a real caddie would have. The Twin Hills course layout (par, yardage, hole handicap) and seven hundred days of weather history. Now we can ask about hole difficulty by yardage, scoring impact of wind, whether heat amplifies the wind effect on a tee shot. Real questions, not toy ones.
2. Stop trusting that the agent is "good" and start measuring. Build a proper CLEARS-style eval, run it, get an honest baseline, and use that baseline to drive tuning.

That's the headline. Below is everything that actually happened.

💬 A word on the role Genie plays here. The Mosaic AI agent doesn't write SQL. The agent reasons in natural language and calls a single tool — Genie — for every data lookup. Genie is the AI glue between English and SQL: it reads the question, picks the right tables out of the space's configured assets, generates a SQL query against the Databricks SQL Warehouse, and returns rows. That separation is what makes the system extensible. Adding course layout and weather data is mostly an exercise in giving Genie new tables to reach for; the agent's prompt then guides when to ask which kind of question. If the lakehouse is the body, Genie is the translator standing between the user and the data.

---

Step 1: Add the course as data

The first new table is the simplest one in the whole project: 18 rows, one per hole at Twin Hills. Par, hole handicap, yardage.

Before I show the code, it's worth stopping for a minute on a question Part 1 glossed over: how do you choose what kind of pipeline to build for a given dataset? Databricks gives you a menu of patterns, and picking the right one is the difference between elegant infrastructure and theatre.

Here's the menu, in plain terms:

Data shape	Cadence	Best fit on Databricks
Streaming events (clicks, telemetry, IoT sensors)	Continuous	Structured Streaming or Lakeflow Declarative Pipelines with Auto Loader
Operational source (a SaaS API, a transactional DB)	Periodic — daily or weekly	Lakeflow Declarative Pipelines (bronze → silver → gold), triggered by a serverless Job
Reference data (a code table, a course layout, a tax-rate sheet)	Rarely — yearly or "when someone decides"	A seed notebook that fully overwrites a static table (safe to rerun)
One-shot backfill (history dump, migration)	Once	A one-time notebook run, then never again
Manual data (a CSV someone produces in a spreadsheet)	When the human gets to it	A notebook that reads from a Volume location, triggered manually or scheduled to look for new files
External enrichment (weather, market data, lookups)	Periodic and append-only	A scheduled notebook or simple Job that pulls and appends

The wrong move in any direction has real cost. Streaming machinery for 18 rows of course data is theatre — the schemas, the checkpoints, the apply_changes calls — for zero benefit. A one-shot notebook for live event data is silently broken because nothing keeps it running. Picking the wrong shape doesn't usually break things on day one; it breaks them slowly enough that nobody connects the symptom back to the original choice.

In Part 1 I leaned on Lakeflow Declarative Pipelines for the league scoring data because it's a SaaS source that changes weekly, has clean primary keys, and benefits from a bronze → silver → gold layering with explicit data quality expectations. That's the classic "operational source" setup.

For course layout the right pick is seed notebook. Reference data changes maybe once a year when the course re-rates. A full bronze → silver → gold pipeline for 18 rows would be overkill. The right shape is a static seed that fully overwrites a Delta table on every run — safe to rerun any number of times, always lands the same result:

HOLES = [
    # course_side 1 = Front 9 — rating 32.5, slope 110
    ("twin_hills", "Twin Hills GC", 1, 1, 5,  5, 505),
    ("twin_hills", "Twin Hills GC", 1, 2, 3,  7, 195),
    # ... 16 more rows ...
]

(df.write
   .mode("overwrite")
   .option("overwriteSchema", "true")
   .saveAsTable("golf_league.silver.course_holes"))

A note worth saying out loud because Part 1 didn't: a notebook is a perfectly legitimate deployment shape.

If you've never used one, a notebook is best thought of as a web-based document that mixes natural language, code, and result tables in a single scrollable page. The page is broken into cells. Each cell is either Markdown (text and headings) or code (Python, SQL, Scala, R — and on Databricks you can mix languages inside the same notebook). When you click ▶ on a code cell, the cell runs, and its output — a table, a chart, a print line, a stack trace — appears immediately below. You can run a single cell, run from a cell down, or run the whole notebook end-to-end in one click. State stays warm between runs, so you can iterate on a bug in cell 12 without re-running cells 1-11. Additional Notebook guidance with Databricks - getting-started HERE.

On Databricks specifically, that notebook is backed by serverless compute by default — a Python (or SQL or Scala) runtime that spins up on demand, scales to your workload, and shuts itself off when you're done. There's no cluster to manage, no environment to configure, no "did I remember to install pandas" friction. You open the notebook, click Run All, and the platform finds you a runtime, executes your code against whatever data lives in Unity Catalog, and shows you the answer.

That model has a few properties that change how you think about deployment:

• The dev loop is the deploy loop. The same notebook you used to develop a transformation IS the thing you run in production. There's no "now translate this into a .py file and a Dockerfile" handoff step.
• You can re-run sections, not just the whole thing. Found a bug in the silver-layer transformation? Re-run just that cell — the cells before it don't need to redo their work. That cuts iteration time on multi-step pipelines from minutes to seconds.
• The notebook is the documentation. Because Markdown cells live alongside code cells, the explanation of why a transformation works lives right next to the transformation itself. Six months from now, when you re-open it, you don't have to reverse-engineer your own intent.
• Notebooks schedule. Any notebook can be wrapped in a Databricks Job to run on a cron schedule, on a trigger, or on demand from an API call. The same notebook you ran by hand becomes the production daily-refresh with three clicks.
• Notebooks version. Git integration is built in. Every notebook can be committed to a branch, code-reviewed in a PR, and deployed via the same Asset Bundles tooling that handles the rest of the lakehouse.

Some folks coming from a software-engineering background think of notebooks as "where you explore" and pipelines as "where you ship." On Databricks, that boundary is fuzzy by design. A notebook can be scheduled by a Job, version-controlled in Git, parameterized via widgets, and rerun safely if you write it that way. For low-frequency, low-volume, well-bounded work — reference seeds, manual backfills, periodic enrichment — a notebook IS the production deployment. The decision is not "notebook vs production"; it's "pick the simplest shape that does the job correctly."

For silver.course_holes, that meant a notebook with three properties:

1. Safe to rerun. mode="overwrite" means re-running it doesn't duplicate rows or leave the table in a half-written state. Run it once or run it ten times — the table looks the same after.
2. Self-documenting. The data is in the notebook, not in some upstream file or API. If you read the notebook you know exactly what landed in the table.
3. Schema-aware. option("overwriteSchema", "true") means if I add a column (a tee_shot_advice field, say), the next run picks it up cleanly.

I added a second tiny table — silver.courses — the same way, to hold per-course metadata: address, lat/lon, tee box, rating/slope per nine. That's the join key I'd need to bolt on more courses if my league ever played a second one.

💡 The deliberate engineer move: match the pipeline shape to the data's change cadence, not to a "we always use streaming here" rule. A weekly-changing source belongs in Lakeflow. A yearly-changing source belongs in a seed notebook. A daily-changing source might be a scheduled job. They all live in the same Unity Catalog; consumers never know the difference, and that's the point. Pick the simplest shape that works, document the choice, move on.

---

Step 2: Pull seven hundred days of weather

Now the fun one. Weather data, joined to every round we have on file.

For historical weather I used Open-Meteo's Archive API: no signup, no key, ERA5 reanalysis-quality data, free for non-commercial use, with one row per day (I pulled from 2024 forward; the archive itself reaches back to 1940).

One note on the seed pattern: the initial backfill does a full overwrite of the table — the right call for a one-time load of a few hundred rows — re-running it can't hurt anything, and simple wins. For the weekly refresh I did exactly what scale would demand: a separate incremental MERGE keyed on (course_id, observation_date) that lands only the new day’s observations and leaves the history untouched. Same pipeline-shape-matching logic from Step 1, just one size up — full-overwrite for the backfill, MERGE for the steady state.

def fetch_course(course_id, lat, lon, start, end):
    r = requests.get(ARCHIVE_URL, params={
        "latitude":  lat,
        "longitude": lon,
        "start_date": start,
        "end_date":   end,
        "daily":      DAILY_VARS,
        "temperature_unit":   "fahrenheit",
        "wind_speed_unit":    "mph",
        "precipitation_unit": "inch",
        "timezone":  "America/New_York",
    }, timeout=60)
    ...

Thirty seconds of HTTP, eight hundred and seventeen rows in silver.weather_observations, one row per day with temp high/low, wind avg + gust, precipitation, and a WMO weather code that I translated into human strings ("clear", "light rain", "thunderstorm").

💡 The pattern that matters: any new data source in a lakehouse should be three things — named, governed, and joinable. Open-Meteo's Archive API became a Unity Catalog table with a comment describing where the data came from, a course_id column to anchor it to the existing schema, and an observation_date column to join cleanly to silver.rounds. The lakehouse pattern means new sources are 30-line scripts, not three-week projects.

The last piece was a pre-bucketed joined table that does the analytics-friendly view in one place — gold.round_with_weather. Every round in silver.rounds joined to that day's weather, with wind_bucket / temp_bucket / precip_bucket columns pre-computed (calm / breezy / windy / gusty; cold / cool / warm / hot; dry / light_rain / wet). The agent could compute those on the fly, but pre-bucketing means natural-language questions like "how do players score in windy conditions?" hit a one-table query instead of a join with case expressions.

---

Step 3: The first finding

With the new tables in place, here's what the data says about wind and scoring in my league (all rounds, 2024-2026):

Wind bucket	Rounds	Avg gross (9 holes)	Avg net
calm (<8 mph)	140	46.18	37.64
breezy (8–15)	432	46.48	36.81
windy (15–22)	68	49.40	38.96
gusty (22+)	20	48.70	39.75

The user asked, in plain English, "How does wind affect scoring in 2026?" Genie translated that into the SQL below — automatically, by reading the question against the configured space tables and the instructions I'd written about how to handle weather questions. No template, no engineered query. The question becomes structured analytics through the natural-language-to-SQL contract:

SELECT wind_bucket,
       COUNT(*)                   AS rounds,
       ROUND(AVG(net_score), 2)   AS avg_net
FROM   golf_league.gold.round_with_weather
WHERE  season_year = 2026
   AND wind_bucket IS NOT NULL
GROUP  BY wind_bucket
ORDER  BY rounds DESC

That's the entire interaction. The user thinks in English. Genie writes the SQL. The Mosaic AI agent narrates the result with context, sample-size caveats, and a "bottom line" sentence. It's a small thing to admire on paper but a real win in practice — there is no DBA in the loop. The lakehouse's natural-language interface is now conversational and low-friction, and any new table I add becomes part of that conversation the moment I declare it.

A few things jumped out in the numbers themselves:

• The effect isn't linear — there's a cliff at 15 mph. Calm vs breezy is noise (0.3 strokes). Breezy vs windy is real (+3 strokes per nine, ~6 per 18).
• Sample sizes matter. The windy bucket at n=68 is what we trust. gusty at n=20 is directional only.
• The contrarian finding: when I split this by temperature and wind together, hot+breezy turned out to be the league's lowest-scoring condition (45.64), and hot+windy was the worst (50.30). The wind cliff is amplified by heat. That's bro-science confirmed by physics — less dense hot air carries the ball further and gives wind more perturbation distance.

If you're a golfer reading this and you're thinking "I knew it," the data says no, you didn't — some of it surprised you. That's why we measure.

---

Step 4: The permission saga (and the invisible service principal)

This one took me the better part of two days to close out — though most of that was my own availability and the time it took to understand which service principals were actually in play, not the platform fighting me. If the section seems long, that's intentional: the point is what I learned about identity on Databricks, not the punchline.

I'd been treating UC permissions as a solved problem — and mostly they were. In Part 1 the original 9 tables, the Genie space, and the agent's runtime identity worked together cleanly because I'd declared them once and the grants propagated through the workspace. When I added four new tables I assumed they'd inherit that same setup automatically. They didn't — and the reason is worth understanding, because it's by design, not a bug.

The agent immediately started returning "It seems the Genie query is failing due to permission issues." The honest reason it took me a while: I'd forgotten how the original grants got there, so I chased the usual suspects by hand first — catalog SHOW GRANTS, Genie space sharing, warehouse Can Use, even the wrong identity (the inference table shows the webapp's managed identity as the requester, which sent me granting permissions to a principal that never needed them). None of that was the mechanism. The smoking gun surfaced in a Genie monitor row I almost missed:

PERMISSION_DENIED: Unable to get space [01f1724975d41883b61bd9700a46aacf]. 
Caused by User 3eb023df-351c-4b69-a987-2bae39783f8e does not have 
read permission for node with acl

That UUID 3eb023df-... wasn't an identity I'd ever granted to by hand — and that's the point. When Mosaic AI deploys an agent with automatic authentication passthrough, it provisions a dedicated system service principal to run the endpoint on your behalf. This is documented, intentional Agent Framework behavior: the runtime SP is how the platform enforces least-privilege at query time instead of baking your personal token into the deployment. It doesn't show up in the workspace SP list or the Genie Share dialog because you're not meant to manage it by hand — you grant it access declaratively, by listing the resources the agent depends on at model-log time and letting the platform propagate the grants to that SP for you:

mlflow.pyfunc.log_model(
    artifact_path="agent",
    python_model=AGENT_FILE,
    resources=[
        DatabricksGenieSpace(genie_space_id=GENIE_SPACE_ID),
        DatabricksServingEndpoint(endpoint_name=LLM_ENDPOINT),
        DatabricksTable(table_name="golf_league.silver.weather_observations"),
        DatabricksTable(table_name="golf_league.silver.courses"),
        DatabricksTable(table_name="golf_league.silver.course_holes"),
        DatabricksTable(table_name="golf_league.gold.round_with_weather"),
        # ... plus the original 9 tables ...
    ],
    ...
)

This is Unity Catalog doing exactly what it's for: because every table, the Genie space, and the endpoint are all governed objects in one catalog, the platform can read the agent's declared dependencies and provision the runtime SP's grants for me — one governance model spanning data, AI assets, and identity, instead of a separate ACL system bolted onto each. I already had this resources list in the deploy notebook — with the original 9 tables from Part 1. I simply hadn't added the 4 new ones. The grants for the original tables had propagated to the runtime SP at the last deploy, so the agent kept answering questions about them fine; the new tables were never in the list, so their grants never propagated. The fix wasn't granting anything by hand — it was re-declaring the full set and redeploying, so the platform re-propagated the grants to the SP. The same mechanism that worked in Part 1; I'd just forgotten I was relying on it.

One other behavior in the same arc is worth surfacing, because it explains why a single missing grant looked so catastrophic:

• Genie validates every table in a space on every call, regardless of which the question needs. When one table's permission validation failed for a principal, the entire space went red. Asking "what's Jeff's handicap?" — which only needs silver.rounds + silver.players — would fail because Genie couldn't validate silver.weather_observations for the SP. The blast radius of a missing grant is the space, not the table.

The declarative model has one corollary that's easy to miss, and it cost me a second round weeks later. The four tables were in resources, redeployed, propagated, working — then one Wednesday the agent went red again on a table I knew was declared. The culprit wasn't the declaration; it was my weekly build job. The notebook that rebuilt that table did a DROP TABLE followed by a CREATE. Unity Catalog governs by object identity, not name — so a drop-and-recreate mints a brand-new object with a new internal table ID, and grants attach to the object, not the name. The runtime SP's grant had been propagated to the old object; the new table of the same name inherited none of it. Every scheduled rebuild was quietly re-breaking the exact permission I'd just fixed.

I confirmed it with a controlled test rather than trusting my hunch: create a throwaway table, grant it, capture its UC table ID, then rebuild it two ways. DROP + CREATE changed the table ID and wiped the table-level grants. CREATE OR REPLACE TABLE — and, it turns out, df.write.mode("overwrite").saveAsTable() on an existing table — kept the same ID and preserved every grant. Same data, same schema outcome, completely different permission fate.

A broader pattern, not a Databricks-specific story

If this whole saga sounds familiar, it should. The identity-and-permission problem at the heart of this post is one of the universal problems in agentic design — every platform has its own version of it, and they all bite the same way.

I've written about the Microsoft side of the same problem more than once. "The Connection Manager Mystery" walked through the identical confusion in Copilot Studio: depending on how the connection manager is configured, a flow can run as the agent author, the end user, or a service connection — and only one usually matches what the designer intended. "Connecting Copilot Studio to Snowflake" made the same point: the connector lights up green, but whose identity flows through to the data layer is what decides whether you ship something secure or accidentally over-permissioned. And the ServiceNow "Graph Connectors vs Custom Agent Connectors" post came down almost entirely to which identity ends up calling ServiceNow at query time. Same problem, three different vendors.

Pattern recognition across all four:

• There is almost always a hidden identity. It might be a Mosaic AI runtime SP, a Copilot Studio connection reference, a Graph connector app registration, or an Azure Logic Apps managed identity. Whatever it's called, the identity that actually executes the data call is rarely the one your dev intuition assumed.
• The permission gates compound. It's never "one grant and you're done." Databricks has UC + Genie space + warehouse. Copilot Studio has connection references + environment security roles + Dataverse + the target system's own auth. ServiceNow connectors have OAuth scopes + Power Platform DLP + the target table's ACLs. Multiple gates, all of which must agree, none of which fail noisily until the user clicks a button.
• Declarative configuration beats hand-grants every time. Whether it's resources=[...] in Mosaic AI, environment variables in a Copilot Studio solution package, or a service principal manifest in Entra — describing what an agent needs at build time and letting the platform provision the grants is the only model that survives the second or third change.

That last point is the real lesson, and it's why I wrote this post the way I did. If you build agents on any platform, the maintenance burden is determined almost entirely by how disciplined you were about declaring dependencies up front. Hand-grant your way through the first six demos and you'll be fine; hand-grant your way through six months of iteration and you'll spend a Saturday afternoon doing what I just did.

I closed the saga, declared all 13 tables in resources=[], redeployed, and the wind question came back with the real answer in about three seconds.

An aside on AI helping AI build AI

Worth surfacing, because it's a genuine Databricks advantage: throughout the saga I was working alongside Genie Code — the in-workspace AI coding and data assistant (part of the same Genie family as the Genie One BI experience and the Genie Agents that power my caddie, all brought under one name in 2026). It reasons over the workspace's own state — catalog grants, endpoint config, served-entity versions, query history — so I used it the way you'd use a senior engineer paired next to you: "show me the runtime SP for this endpoint", "what tables does this space see", "are there any DENYs on this catalog". And the same governed-data foundation powers both the assistant helping me build and the agent I'm building — which is hard to pull off on a stack that wasn't designed for it.

You still verify what it tells you — once it reported grants were in place when they weren't — but as a fast index into the platform's own state it dropped time-to-first-clue from minutes to seconds: pulling traffic-routing tables, inspecting served-entity env vars, diffing v8 vs v9 config.

That, more than anything, is what I expect the day-to-day of working with Databricks to feel like going forward: the platform reasons about itself, in natural language, alongside the human and the agent doing the build. The same governed foundation that powers the caddie's analytics also powers Genie Code, the assistant that helps me build the caddie. The recursion is, frankly, kind of cool.

---

Step 5: Evaluating the agent honestly

This is where the post gets quietly important. Up to this point I'd been judging the agent by vibes — "ask it a question, eyeball the answer." That works for one or two questions. It doesn't work for fourteen test cases across a season of real scores.

Worth naming what’s running underneath all of this: the endpoint has MLflow Tracing switched on (ENABLE_MLFLOW_TRACING=true), so every agent call is already captured end-to-end — the reasoning step, the Genie tool call, the generated SQL, latency, and token counts, all as a nested trace tied to a request ID. That’s the difference between an eval score and an explanation: when a CLEARS case comes back red, the trace tells me whether the agent reasoned wrong, Genie translated wrong, or the SQL simply returned nothing. Tracing is how you debug the why behind the number.

I built a small CLEARS-style eval set. Fourteen golden Q&A pairs — leaderboards, handicap trend, weekly recap, advisory question, weather effects, course-layout questions. Each row has an expected_facts list: things the response must include for the correctness judge to mark it green. Then I wired it into mlflow.evaluate(model_type="databricks-agent") which runs Mosaic AI Agent Evaluation's built-in judges (correctness, safety, relevance, groundedness).

The journey from "no eval" to "honest baseline" went through four readings, all on the same agent:

Run	Correctness	Why
Eval v1	0%	Rubric was wrong. My expected_facts described SQL internals ("filters to course_side=1") that the judge can't observe from the response text.
Eval v2	36%	Rubric fixed but evaluation hit Genie's rate limit. 9 of 14 responses were the agent saying "rate limit has been exceeded."
Eval v3	57%	Pacing and retry added inside the eval harness. Real signal — but the rubric was strict about coverage.
Eval v4	79%	One paragraph added to the Genie space instructions telling the agent to explicitly address every named entity in the question. Big jump from a small change.

That trajectory is the post. Not the 79%. The trajectory.

The eval set is checked in. It runs in twelve minutes. After any prompt change I re-run it. The honest baseline is the foundation of every future improvement. Without it I would have shipped any of the three earlier versions and called it done.

💡 The eval lesson: the first eval result is always the rubric, not the agent. Build a rubric the judge can actually verify from response text. Then iterate.

💬 A note on the rate-limit detour. I solved the eval rate-limiting with pacing and retries inside the harness, which is fine for a personal project. The platform-native answer is Mosaic AI Gateway — it puts rate limiting, usage tracking, and safety guardrails in front of a serving endpoint as configuration rather than code. For anything multi-user I’d put the caddie behind a gateway and delete my hand-rolled retry loop entirely.

One more thing worth noting before we move on. The CLEARS score grades the agent's answer — but the agent's answer only exists if Genie's natural-language-to-SQL translation got the right tables, the right joins, and the right filters in the first place. If a question maps to ambiguous SQL, the agent's response will reflect that ambiguity downstream. The 79% we landed on is really two scores multiplied together: the agent's reasoning quality and Genie's translation quality on this specific set of questions against this specific set of tables. That's part of why an honest eval matters — it surfaces both layers at once, and instruction tweaks in the Genie space (not just the agent prompt) move the number.

---

Where we land

So at the end of this post:

• The agent knows about every hole at Twin Hills, every day of weather for three seasons, and the full history of every round our league has played.
• It can answer questions about wind, heat, and course layout that it couldn't answer two weeks ago.
• It has an eval baseline of 79% correctness / 100% safety on a 14-question CLEARS-style set, with reproducible methodology checked into the repo.
• I've found and fixed two things in Part 1's architecture that would have bitten me at scale: an invisible runtime service principal whose grants only propagate when you declare every dependency at log time, and a warehouse setting that was costing me $100/month I wasn't using.

That's two things I now know about Databricks that I didn't a month ago — and that I'd bet money I'll see again in real customer engagements over the next year. Which was kind of the point.

---

Why a golf league looks a lot like a factory floor

This whole project is personal. The data set is two dozen guys, two seasons, and a Wednesday-night nine-hole league. But the shape of the problem — and every architectural decision I made — maps almost one-for-one onto a real enterprise workload. Swap "rounds at Twin Hills" for "parts coming off line 4," and the lakehouse looks identical.

Think about a manufacturing operation. The shop floor has:

• A scoring SaaS (a production execution system or MES) that owns the operational data and has a perfectly good UI, just no good way to get the data out for analytics.
• A course layout (a bill of materials or routing definition) that changes rarely but is the static reference data every analytical question joins back to.
• Weather (machine sensors, ambient conditions, supplier lead times) that's a separate external feed nobody owns end-to-end but everybody wants in the analysis.
• Hand-tuned spreadsheets that operators use today because they grew up with Excel and there was no other path.
• A handful of power users who know which sheet is real, which one is stale, and which formula breaks if you sort the rows.

That's exactly what my golf league looks like, minus a few decimal places. And the answer for both is the same: pull the operational data into a governed lakehouse, add the static and external sources alongside it, put a natural-language layer on top, and put an agent in front of the natural-language layer. Now your shift supervisor can ask "which line is running below average on Tuesdays when we run from third-shift inventory" the way I ask "how does Jeff Baart score on the longest par 4s." Same architecture. Same building blocks. Different stakes.

If you're reading this from a manufacturing job — or any operations role where you've been exporting CSVs and pivoting in Excel for years — the most important thing this post is trying to show you is that you don't need to become a software engineer to do this work. I'll be honest about what you just read: a two-day permission chase, some base64 spelunking, a rubric I rewrote four times. The platform lowers the floor a long way, but it doesn't remove the thinking — it removes the undifferentiated engineering (the clusters, the checkpoints, the auth plumbing) so the effort that's left is the part that was always the real work. You need:

• A platform that handles the storage, the pipelines, and the governance so you don't have to (Unity Catalog + Lakeflow).
• A natural-language interface so you can ask questions without writing SQL (Genie).
• An agent layer that turns questions into multi-step reasoning instead of one-shot lookups (Mosaic AI).
• A measurement framework so you can trust the answers (Mosaic AI Agent Evaluation).

Each of those took me a weekend or less to wire up, and none required code you couldn't read. Most of the time I spent wasn't writing software — it was deciding which building block to reach for, and why. That's an analyst's skill set, not a developer's. So if your current "data platform" is a folder of Excel exports and a few PowerBI dashboards someone built three years ago, you are not behind — you are exactly the audience for this. The work I did for my two-dozen-guy golf league is the same work you'd do for a twenty-line factory, and the same first ten findings will surprise you the way the wind-cliff surprised me. The on-ramp is shorter than you probably believe.

---

What's next

A few threads from this post are worth pulling on next:

• Move the business logic out of the prompt. The caddie’s domain rules — gross vs. net vs. match-play, "sum match_play_points for standings" — currently live in the agent’s system prompt, which means every prompt edit risks the semantics. The durable home is a Unity Catalog metric view: define those rules once in UC and let the agent, Genie, and the dashboards read the same definition instead of each re-encoding it. It’s also the thing that most reduces how often I have to redeploy the agent just to change a definition.
• Real per-user identity in the agent path. Today the webapp invokes the agent as its own Container App managed identity. For a multi-user app — or a Teams/Copilot bot — the right pattern is on-behalf-of-user token exchange, so per-user Unity Catalog ACLs are actually enforced. That’s a governance demo I want to write up on its own.
• Multi-tool agents and Agent Bricks. This agent has a single tool (Genie). The natural next step is a second — a weather forecast lookup, say, so the caddie can tell me "play long off the tee, the wind drops at 6 pm" instead of only doing retrospective analytics. That’s also where the line between hand-assembled agents and Agent Bricks gets interesting.

If any of those sound interesting — drop a comment or DM me. I'm just landing at Databricks and the lens through which I'm looking at all of this is "what would my team help a customer build next, and what would we tell them to avoid?" Both halves of that question have answers worth writing down.

---

Thanks for reading. If you want to start from the beginning, Part 1 is here.

References

• Mosaic AI Agent Framework — declaring resources at log time
• Mosaic AI Agent Evaluation — built-in judges
• AI/BI Genie spaces — sharing and permissions
• Unity Catalog — privilege model
• Lakeflow Declarative Pipelines
• Open-Meteo Archive API