Real agents, real edge cases: tuning, evaluating, and breaking my golf lakehouse

Part 2 of a personal-scale data + AI series. In which I add new data, break almost everything, learn the actual production-shaped lessons, and end up with an agent that's measurably smarter and a saga I'll tell at every dinner party for the next year.

Personal-lakehouse series: Part 1 · From spreadsheets to a real Lakehouse · Part 1.5 · Genie learns · Part 2 · Real agents, real edge cases (you are here)

A quick note on why

A few weeks before I started this post, I accepted a Field Engineering leadership role at Databricks. After nearly nine years at Microsoft, where most of my time was spent helping customers run AI workloads on Azure, the move felt like a natural alignment of two things I've cared about for a long time: agentic AI that genuinely earns its keep and structured data done with discipline.

I wrote Part 1 of this series in the seam between those two chapters — partly to refresh my platform knowledge end to end before showing up to my new team, partly because the best way I know to learn something is to build something real with it. The first post took a weekly golf league SaaS, extracted its data, and stood up a personal lakehouse on Azure Databricks with Unity Catalog, Lakeflow, a Genie space, and a Mosaic AI agent on top.

This post is what happens after that. I added two new data sources to make the agent smarter, broke about six things in the process, learned exactly the lessons I needed to learn before I get my Databricks badge, and ended up with a working agent and a real eval baseline I can actually defend in a code review.

If you're new here — Part 1 sets up the lakehouse from scratch. This post assumes that foundation exists and goes from "the agent works on the easy questions" to "the agent stands up to a structured eval."

Where we left off

At the end of Part 1, the caddie agent could answer questions like:

"Who has the lowest scoring average this season?"
"What can Jeff Baart do to improve his game?"
"Who's hot lately?"

That worked. It also barely scratched what the agent could be. A real caddie sees a lot more than your scorecard — they know the course inside and out and they read conditions before they ever hand you a club. So this post tackles two ideas:

Add the data a real caddie would have. The Twin Hills course layout (par, yardage, hole handicap) and seven hundred days of weather history. Now we can ask about hole difficulty by yardage, scoring impact of wind, whether heat amplifies the wind effect on a tee shot. Real questions, not toy ones.
Stop trusting that the agent is "good" and start measuring. Build a proper CLEARS-style eval, run it, get an honest baseline, and use that baseline to drive tuning.

That's the headline. Below is everything that actually happened.

💬 A word on the role Genie plays here. The Mosaic AI agent doesn't write SQL. The agent reasons in natural language and calls a single tool — Genie — for every data lookup. Genie is the AI glue between English and SQL: it reads the question, picks the right tables out of the space's configured assets, generates a SQL query against the Databricks SQL Warehouse, and returns rows. That separation is what makes the system extensible. Adding course layout and weather data is mostly an exercise in giving Genie new tables to reach for; the agent's prompt then guides when to ask which kind of question. If the lakehouse is the body, Genie is the translator standing between the user and the data.

Step 1: Add the course as data

The first new table is the simplest one in the whole project: 18 rows, one per hole at Twin Hills. Par, hole handicap, yardage.

Before I show the code, it's worth stopping for a minute on a question Part 1 glossed over: how do you choose what kind of pipeline to build for a given dataset? Databricks gives you a menu of patterns, and picking the right one is the difference between elegant infrastructure and theatre.

Here's the menu, in plain terms:

Data shape	Cadence	Best fit on Databricks
Streaming events (clicks, telemetry, IoT sensors)	Continuous	Structured Streaming or Lakeflow Declarative Pipelines with Auto Loader
Operational source (a SaaS API, a transactional DB)	Periodic — daily or weekly	Lakeflow Declarative Pipelines (bronze → silver → gold), triggered by a serverless Job
Reference data (a code table, a course layout, a tax-rate sheet)	Rarely — yearly or "when someone decides"	A seed notebook that fully overwrites a static table (safe to rerun)
One-shot backfill (history dump, migration)	Once	A one-time notebook run, then never again
Manual data (a CSV someone produces in a spreadsheet)	When the human gets to it	A notebook that reads from a Volume location, triggered manually or scheduled to look for new files
External enrichment (weather, market data, lookups)	Periodic and append-only	A scheduled notebook or simple Job that pulls and appends

The wrong move in any direction has real cost. Streaming machinery for 18 rows of course data is theatre — the schemas, the checkpoints, the apply_changes calls — for zero benefit. A one-shot notebook for live event data is silently broken because nothing keeps it running. Picking the wrong shape doesn't usually break things on day one; it breaks them slowly enough that nobody connects the symptom back to the original choice.

In Part 1 I leaned on Lakeflow Declarative Pipelines for the league scoring data because it's a SaaS source that changes weekly, has clean primary keys, and benefits from a bronze → silver → gold layering with explicit data quality expectations. That's the canonical "operational source" pattern.

For course layout the right pick is seed notebook. Reference data changes maybe once a year when the course re-rates. A full bronze → silver → gold pipeline for 18 rows would be overkill. The right shape is a static seed that fully overwrites a Delta table on every run — safe to rerun any number of times, always lands the same result:

HOLES = [
    # course_side 1 = Front 9 — rating 32.5, slope 110
    ("twin_hills", "Twin Hills GC", 1, 1, 5,  5, 505),
    ("twin_hills", "Twin Hills GC", 1, 2, 3,  7, 195),
    # ... 16 more rows ...
]

(df.write
   .mode("overwrite")
   .option("overwriteSchema", "true")
   .saveAsTable("golf_league.silver.course_holes"))

A note worth saying out loud because Part 1 didn't: a notebook is a perfectly legitimate deployment shape.

If you've never used one, a notebook is best thought of as a web-based document that mixes prose, code, and result tables in a single scrollable page. The page is broken into cells. Each cell is either Markdown (text and headings) or code (Python, SQL, Scala, R — and on Databricks you can mix languages inside the same notebook). When you click ▶ on a code cell, the cell runs, and its output — a table, a chart, a print line, a stack trace — appears immediately below. You can run a single cell, run from a cell down, or run the whole notebook end-to-end in one click. State stays warm between runs, so you can iterate on a bug in cell 12 without re-running cells 1-11.Additional Notebook getting-started guidance HERE.

On Databricks specifically, that notebook is backed by serverless compute by default — a Python (or SQL or Scala) runtime that spins up on demand, scales to your workload, and shuts itself off when you're done. There's no cluster to manage, no environment to configure, no "did I remember to install pandas" friction. You open the notebook, click Run All, and the platform finds you a runtime, executes your code against whatever data lives in Unity Catalog, and shows you the answer.

That model has a few properties that change how you think about deployment:

The dev loop is the deploy loop. The same notebook you used to develop a transformation IS the thing you run in production. There's no "now translate this into a .py file and a Dockerfile" handoff step.
You can re-run sections, not just the whole thing. Found a bug in the silver-layer transformation? Re-run just that cell — the cells before it don't need to redo their work. That cuts iteration time on multi-step pipelines from minutes to seconds.
The notebook is the documentation. Because Markdown cells live alongside code cells, the explanation of why a transformation works lives right next to the transformation itself. Six months from now, when you re-open it, you don't have to reverse-engineer your own intent.
Notebooks schedule. Any notebook can be wrapped in a Databricks Job to run on a cron schedule, on a trigger, or on demand from an API call. The same notebook you ran by hand becomes the production daily-refresh with three clicks.
Notebooks version. Git integration is built in. Every notebook can be committed to a branch, code-reviewed in a PR, and deployed via the same Asset Bundles tooling that handles the rest of the lakehouse.

Some folks coming from a software-engineering background think of notebooks as "where you explore" and pipelines as "where you ship." On Databricks, that boundary is fuzzy by design. A notebook can be scheduled by a Job, version-controlled in Git, parameterized via widgets, and rerun safely if you write it that way. For low-frequency, low-volume, well-bounded work — reference seeds, manual backfills, periodic enrichment — a notebook IS the production deployment. The decision is not "notebook vs production"; it's "pick the simplest shape that does the job correctly."

For silver.course_holes, that meant a notebook with three properties:

Safe to rerun. mode="overwrite" means re-running it doesn't duplicate rows or leave the table in a half-written state. Run it once or run it ten times — the table looks the same after.
Self-documenting. The data is in the notebook, not in some upstream file or API. If you read the notebook you know exactly what landed in the table.
Schema-aware. option("overwriteSchema", "true") means if I add a column (a tee_shot_advice field, say), the next run picks it up cleanly.

I added a second tiny table — silver.courses — the same way, to hold per-course metadata: address, lat/lon, tee box, rating/slope per nine. That's the join key I'd need to bolt on more courses if my league ever played a second one.

💡 The deliberate engineer move: match the pipeline shape to the data's change cadence, not to a "we always use streaming here" rule. A weekly-changing source belongs in Lakeflow. A yearly-changing source belongs in a seed notebook. A daily-changing source might be a scheduled job. They all live in the same Unity Catalog; consumers never know the difference, and that's the point. Pick the simplest shape that works, document the choice, move on.

Step 2: Pull seven hundred days of weather

Now the fun one. Weather data, joined to every round we have on file.

There's a slightly annoying market quirk worth knowing: NOAA's free public API only serves forecasts, not history. For historical observations you have to use NOAA Climate Data Online, which requires a token and is rate-throttled. So I went somewhere else: Open-Meteo's Archive API. No signup, no key, ERA5 reanalysis-quality data, free for non-commercial use, with one row per day (I pulled from 2024 forward; the archive itself reaches back to 1940).

One honest caveat on the seed pattern: it does a full overwrite of the table on every run. That’s the right call here because the dataset is tiny (a few hundred rows) and idempotence beats cleverness. At real scale you’d switch to an incremental MERGE keyed on (course_id, observation_date), or an Auto Loader streaming table — same pipeline-shape-matching logic from Step 1, just one size up.

def fetch_course(course_id, lat, lon, start, end):
    r = requests.get(ARCHIVE_URL, params={
        "latitude":  lat,
        "longitude": lon,
        "start_date": start,
        "end_date":   end,
        "daily":      DAILY_VARS,
        "temperature_unit":   "fahrenheit",
        "wind_speed_unit":    "mph",
        "precipitation_unit": "inch",
        "timezone":  "America/New_York",
    }, timeout=60)
    ...

Thirty seconds of HTTP, eight hundred and seventeen rows in silver.weather_observations, one row per day with temp high/low, wind avg + gust, precipitation, and a WMO weather code that I translated into human strings ("clear", "light rain", "thunderstorm").

💡 The pattern that matters: any new data source in a lakehouse should be three things — named, governed, and joinable. Open-Meteo's Archive API became a Unity Catalog table with a comment describing where the data came from, a course_id column to anchor it to the existing schema, and an observation_date column to join cleanly to silver.rounds. The lakehouse pattern means new sources are 30-line scripts, not three-week projects.

The last piece was a pre-bucketed joined table that does the analytics-friendly view in one place — gold.round_with_weather. Every round in silver.rounds joined to that day's weather, with wind_bucket / temp_bucket / precip_bucket columns pre-computed (calm / breezy / windy / gusty; cold / cool / warm / hot; dry / light_rain / wet). The agent could compute those on the fly, but pre-bucketing means natural-language questions like "how do players score in windy conditions?" hit a one-table query instead of a join with case expressions.

Step 3: The first finding

With the new tables in place, here's what the data says about wind and scoring in my league (all rounds, 2024-2026):

Wind bucket	Rounds	Avg gross (9 holes)	Avg net
calm (<8 mph)	140	46.18	37.64
breezy (8–15)	432	46.48	36.81
windy (15–22)	68	49.40	38.96
gusty (22+)	20	48.70	39.75

The user asked, in plain English, "How does wind affect scoring in 2026?" Genie translated that into the SQL below — automatically, by reading the question against the configured space tables and the instructions I'd written about how to handle weather questions. No template, no engineered query. The question becomes structured analytics through the natural-language-to-SQL contract:

SELECT wind_bucket,
       COUNT(*)                   AS rounds,
       ROUND(AVG(net_score), 2)   AS avg_net
FROM   golf_league.gold.round_with_weather
WHERE  season_year = 2026
   AND wind_bucket IS NOT NULL
GROUP  BY wind_bucket
ORDER  BY rounds DESC

That's the entire interaction. The user thinks in English. Genie writes the SQL. The Mosaic AI agent narrates the result with context, sample-size caveats, and a "bottom line" sentence. It's a small thing to admire on paper but a real win in practice — there is no DBA in the loop. The lakehouse's natural-language interface is now genuinely conversational, and any new table I add becomes part of that conversation the moment I declare it.

A few things jumped out in the numbers themselves:

The effect isn't linear — there's a cliff at 15 mph. Calm vs breezy is noise (0.3 strokes). Breezy vs windy is real (+3 strokes per nine, ~6 per 18).
Sample sizes matter. The windy bucket at n=68 is what we trust. gusty at n=20 is directional only.
The contrarian finding: when I split this by temperature and wind together, hot+breezy turned out to be the league's lowest-scoring condition (45.64), and hot+windy was the worst (50.30). The wind cliff is amplified by heat. That's bro-science confirmed by physics — less dense hot air carries the ball further and gives wind more perturbation distance.

If you're a golfer reading this and you're thinking "I knew it," the data says no, you didn't — some of it surprised you. That's why we measure.

Step 4: The permission saga (and the invisible service principal)

This one took me the better part of two days to close out — though most of that was my own availability and the time it took to understand which service principals were actually in play, not the platform fighting me. If the section seems long, that's intentional: the point is what I learned about identity on Databricks, not the punchline.

I'd been treating UC permissions as a solved problem — and mostly they were. In Part 1 the original 9 tables, the Genie space, and the agent's runtime identity worked together cleanly because I'd declared them once and the grants propagated through the workspace. When I added four new tables I assumed they'd inherit that same setup automatically. They didn't — and the reason is worth understanding, because it's by design, not a bug.

The agent immediately started returning "It seems the Genie query is failing due to permission issues." The honest reason it took me a while: I'd forgotten how the original grants got there, so I re-checked every layer of the stack by hand before remembering the mechanism:

Catalog grants — confirmed via SHOW GRANTS, explicit per-table SELECT, schema-level SELECT, account users SELECT, the works
Genie space sharing — the SP listed with Can Run, removed and re-added to force a cache flush, no improvement
SQL Warehouse permissions — Can Use granted, warehouse rightsized from Small to 2X-Small for cost (more on this in a minute)
Identity confusion — the inference table showed the webapp's MI as the requester (f03fa488-...), so I spent hours granting permissions to that principal that didn't actually need them

The smoking gun finally surfaced in a Genie monitor row I almost missed:

PERMISSION_DENIED: Unable to get space [01f1724975d41883b61bd9700a46aacf]. 
Caused by User 3eb023df-351c-4b69-a987-2bae39783f8e does not have 
read permission for node with acl

That UUID 3eb023df-... wasn't an identity I'd ever granted to by hand — and that's the point. When Mosaic AI deploys an agent with automatic authentication passthrough, it provisions a dedicated system service principal to run the endpoint on your behalf. This is documented, intentional Agent Framework behavior: the runtime SP is how the platform enforces least-privilege at query time instead of baking your personal token into the deployment. It doesn't show up in the workspace SP list or the Genie Share dialog because you're not meant to manage it by hand — you grant it access declaratively, by listing the resources the agent depends on at model-log time and letting the platform propagate the grants to that SP for you:

mlflow.pyfunc.log_model(
    artifact_path="agent",
    python_model=AGENT_FILE,
    resources=[
        DatabricksGenieSpace(genie_space_id=GENIE_SPACE_ID),
        DatabricksServingEndpoint(endpoint_name=LLM_ENDPOINT),
        DatabricksTable(table_name="golf_league.silver.weather_observations"),
        DatabricksTable(table_name="golf_league.silver.courses"),
        DatabricksTable(table_name="golf_league.silver.course_holes"),
        DatabricksTable(table_name="golf_league.gold.round_with_weather"),
        # ... plus the original 9 tables ...
    ],
    ...
)

I already had this resources list in the deploy notebook — with the original 9 tables from Part 1. I simply hadn't added the 4 new ones. The grants for the original tables had propagated to the runtime SP at the last deploy, so the agent kept answering questions about them fine; the new tables were never in the list, so their grants never propagated. The fix wasn't granting anything by hand — it was re-declaring the full set and redeploying, so the platform re-propagated the grants to the SP. The same mechanism that worked in Part 1; I'd just forgotten I was relying on it.

One other behavior in the same arc is worth surfacing, because it explains why a single missing grant looked so catastrophic:

Genie validates every table in a space on every call, regardless of which the question needs. When one table's permission validation failed for a principal, the entire space went red. Asking "what's Jeff's handicap?" — which only needs silver.rounds + silver.players — would fail because Genie couldn't validate silver.weather_observations for the SP. The blast radius of a missing grant is the space, not the table.

A broader pattern, not a Databricks-specific story

If this whole saga sounds familiar, it should. The identity-and-permission problem at the heart of this post is one of the universal problems in agentic design — every platform has its own version of it, and they all bite the same way.

I've written about the Microsoft side of the same problem more than once. In "The Connection Manager Mystery: Who Actually Runs Your Agent's Flow?" I walked through exactly the same confusion in Copilot Studio: an agent calls a Power Automate flow, and depending on how the connection manager is configured, that flow can run as the agent author, the end user, or a service connection — three different identities, three different security postures, and only one of them usually matches what the designer intended. Same problem, completely different vendor. In "Connecting Copilot Studio to Snowflake: Why Agent Architecture Matters More Than the Connector" the lesson was the same again: the connector lights up green, but the architectural question of whose identity flows through to the data layer is what determines whether you ship something secure or something accidentally over-permissioned. And in the ServiceNow "Graph Connectors vs Custom Agent Connectors" post, the choice between the two patterns came down almost entirely to which identity ends up calling ServiceNow at query time — the user's, the connector app's, or a service account.

Pattern recognition across all four:

There is almost always a hidden identity. It might be a Mosaic AI runtime SP, a Copilot Studio connection reference, a Graph connector app registration, or an Azure Logic Apps managed identity. Whatever it's called, the identity that actually executes the data call is rarely the one your dev intuition assumed.
The permission gates compound. It's never "one grant and you're done." Databricks has UC + Genie space + warehouse. Copilot Studio has connection references + environment security roles + Dataverse + the target system's own auth. ServiceNow connectors have OAuth scopes + Power Platform DLP + the target table's ACLs. Multiple gates, all of which must agree, none of which fail noisily until the user clicks a button.
Declarative configuration beats hand-grants every time. Whether it's resources=[...] in Mosaic AI, environment variables in a Copilot Studio solution package, or a service principal manifest in Entra — describing what an agent needs at build time and letting the platform provision the grants is the only model that survives the second or third change.

That last point is the real lesson, and it's why I wrote this post the way I did. If you build agents on any platform, the maintenance burden is determined almost entirely by how disciplined you were about declaring dependencies up front. Hand-grant your way through the first six demos and you'll be fine; hand-grant your way through six months of iteration and you'll spend a Saturday afternoon doing what I just did.

💡 The lesson — and the real reason this post exists: when you build an agent on top of governed data, the permission surface is not what your dev intuition tells you. On Databricks there are at least four identities involved (the user, the webapp MI, the agent's invisible runtime SP, the Genie space owner) and three permission gates (UC grants, space sharing, warehouse Can Use). The clean answer is declarative resources at log time — list every dependency in the resources=[...] block and let Mosaic AI auto-grant the invisible SP. The hand-grant path will work, until you add a new table six months from now and don't remember which of the four identities needed which of the three grants. The exact same lesson, in different vocabulary, applies in Copilot Studio, in M365 Agent 365, in any agentic system worth shipping. Declare your dependencies. Trust the platform to grant. Don't build a permission graph in your head — it will leak.

I closed the saga, declared all 13 tables in resources=[], redeployed, and the wind question came back with the real answer in about three seconds.

An aside on AI helping AI build AI

Worth surfacing: throughout the saga I was working in two windows. One was the Databricks workspace itself. The other was the Databricks Assistant — the in-product Genie-powered assistant that ships inside the Databricks UI and can query the workspace's own metadata (catalog grants, endpoint config, served entity versions, query history). I used it the way you'd use a senior engineer paired in next to you: "show me the runtime SP for this endpoint", "what tables does this space see", "are there any DENYs on this catalog".

Occasionally, I needed to coach it for accuracy. It confidently told me grants were correctly in place when they weren't, and it suggested storing a PAT in a secret to fix what was actually a resources=[] declaration problem. But it was also genuinely useful for things that would have been a slog otherwise — pulling traffic-routing tables, inspecting served entity env vars, diffing v8 vs v9 config. The right way to think about it isn't "second opinion" or "fact source," it's a fast index into the platform's own state. You still verify everything it tells you, but the time-to-first-clue dropped from minutes to seconds.

That, more than anything, is what I expect the day-to-day of working with Databricks to feel like going forward: the platform reasons about itself, in natural language, alongside the human and the agent doing the build. The same Genie capability that powers the caddie's analytics also powers the assistant that helps me build the caddie. The recursion is, frankly, kind of cool.

Step 5: Cost — the meter I didn't expect

While I was debugging permissions I happened to check Azure Cost Management and saw a spike for the resource group over five days. For a personal lakehouse that should be running at $5-10 a month, that's an order of magnitude off.

The breakdown was educational:

Meter	$	What it is
Premium Serverless SQL DBU	~$100	The SQL Warehouse the Genie space uses
Interactive Serverless DBU	$6	Notebook runs (seeds, eval, deploy)
Serverless Realtime Inferencing DBU	$5	The agent endpoint
Automated Serverless DBU	$4	The Lakeflow pipeline

The Genie warehouse was 88% of my bill. Not the agent endpoint (which I'd assumed). Not the pipeline (which only runs once a week). The warehouse that wakes up every time I — or the agent, or some testing notebook — fires a query.

Three fifteen-second changes cut it dramatically:

Cluster size: Small → 2X-Small (halves DBU per hour for my data size)
Auto-stop: 10 min → 5 min (minimum allowed; halves idle time)
Confirmed only one warehouse exists (the system-default that Genie spaces attach to by default)

Projected new monthly: ~$25-30, headed lower once I stop poking it daily.

💡 The cost lesson nobody publishes: the most expensive part of a small agent workload isn't the LLM and it isn't the inference endpoint. It's the data warehouse the agent talks to. Right-size that warehouse on day one or you'll find out in week two.

Step 6: Evaluating the agent honestly

This is where the post gets quietly important. Up to this point I'd been judging the agent by vibes — "ask it a question, eyeball the answer." That works for one or two questions. It doesn't work for fourteen test cases across a season of real scores.

Worth naming what’s running underneath all of this: the endpoint has MLflow Tracing switched on (ENABLE_MLFLOW_TRACING=true), so every agent call is already captured end-to-end — the reasoning step, the Genie tool call, the generated SQL, latency, and token counts, all as a nested trace tied to a request ID. That’s the difference between an eval score and an explanation: when a CLEARS case comes back red, the trace tells me whether the agent reasoned wrong, Genie translated wrong, or the SQL simply returned nothing. Tracing is how you debug the why behind the number.

I built a small CLEARS-style eval set. Fourteen golden Q&A pairs — leaderboards, handicap trend, weekly recap, advisory question, weather effects, course-layout questions. Each row has an expected_facts list: things the response must include for the correctness judge to mark it green. Then I wired it into mlflow.evaluate(model_type="databricks-agent") which runs Mosaic AI Agent Evaluation's built-in judges (correctness, safety, relevance, groundedness).

The journey from "no eval" to "honest baseline" went through four readings, all on the same agent:

Run	Correctness	Why
Eval v1	0%	Rubric was wrong. My `expected_facts` described SQL internals ("filters to course_side=1") that the judge can't observe from the response text.
Eval v2	36%	Rubric fixed but evaluation hit Genie's rate limit. 9 of 14 responses were the agent saying "rate limit has been exceeded."
Eval v3	57%	Pacing and retry added inside the eval harness. Real signal — but the rubric was strict about coverage.
Eval v4	79%	One paragraph added to the Genie space instructions telling the agent to explicitly address every named entity in the question. Big jump from a small change.

That trajectory is the post. Not the 79%. The trajectory.

The eval set is checked in. It runs in twelve minutes. After any prompt change I re-run it. The honest baseline is the foundation of every future improvement. Without it I would have shipped any of the three earlier versions and called it done.

💡 The eval lesson: the first eval result is always the rubric, not the agent. Build a rubric the judge can actually verify from response text. Then iterate.

💬 A note on the rate-limit detour. I solved the eval rate-limiting with pacing and retries inside the harness, which is fine for a personal project. The platform-native answer is Mosaic AI Gateway — it puts rate limiting, usage tracking, and safety guardrails in front of a serving endpoint as configuration rather than code. For anything multi-user I’d put the caddie behind a gateway and delete my hand-rolled retry loop entirely.

One more thing worth noting before we move on. The CLEARS score grades the agent's answer — but the agent's answer only exists if Genie's natural-language-to-SQL translation got the right tables, the right joins, and the right filters in the first place. If a question maps to ambiguous SQL, the agent's response will reflect that ambiguity downstream. The 79% we landed on is really two scores multiplied together: the agent's reasoning quality and Genie's translation quality on this specific set of questions against this specific set of tables. That's part of why an honest eval matters — it surfaces both layers at once, and instruction tweaks in the Genie space (not just the agent prompt) move the number.

Where we land

So at the end of this post:

The agent knows about every hole at Twin Hills, every day of weather for three seasons, and the full history of every round our league has played.
It can answer questions about wind, heat, and course layout that it couldn't answer two weeks ago.
It has an eval baseline of 79% correctness / 100% safety on a 14-question CLEARS-style set, with reproducible methodology checked into the repo.
I've found and fixed two things in Part 1's architecture that would have bitten me at scale: an invisible runtime service principal whose grants only propagate when you declare every dependency at log time, and a warehouse setting that was costing me $100/month I wasn't using.

That's two things I now know about Databricks that I didn't a month ago — and that I'd bet money I'll see again in real customer engagements over the next year. Which was kind of the point.

Why a golf league looks a lot like a factory floor

This whole project is personal. The data set is two dozen guys, two seasons, and a Wednesday-night nine-hole league. But the shape of the problem — and every architectural decision I made — maps almost one-for-one onto a real enterprise workload. Swap "rounds at Twin Hills" for "parts coming off line 4," and the lakehouse looks identical.

Think about a manufacturing operation. The shop floor has:

A scoring SaaS (a production execution system or MES) that owns the operational data and has a perfectly good UI, just no good way to get the data out for analytics.
A course layout (a bill of materials or routing definition) that changes rarely but is the static reference data every analytical question joins back to.
Weather (machine sensors, ambient conditions, supplier lead times) that's a separate external feed nobody owns end-to-end but everybody wants in the analysis.
Hand-tuned spreadsheets that operators use today because they grew up with Excel and there was no other path.
A handful of power users who know which sheet is real, which one is stale, and which formula breaks if you sort the rows.

That's exactly what my golf league looks like, minus a few decimal places. And the answer for both is the same: pull the operational data into a governed lakehouse, add the static and external sources alongside it, put a natural-language layer on top, and put an agent in front of the natural-language layer. Now your shift supervisor can ask "which line is running below average on Tuesdays when we run from third-shift inventory" the way I ask "how does Jeff Baart score on the longest par 4s." Same architecture. Same building blocks. Different stakes.

If you're reading this from a manufacturing job — or any operations role where you've been exporting CSVs and pivoting in Excel for years — the most important thing this post is trying to show you is that you don't need to become a software engineer to do this work — though I'll be honest about what you just read: there was a two-day permission chase, some base64 spelunking, and a rubric I had to rewrite four times. The platform lowers the floor a long way, but it doesn't remove the thinking. What it removes is the undifferentiated engineering — the clusters, the checkpoints, the auth plumbing — so the effort that’s left is the part that was always the real work: deciding which building block to reach for, and why. That’s an analyst’s judgment, not a developer’s toolchain. You need:

A platform that handles the storage, the pipelines, and the governance so you don't have to (Unity Catalog + Lakeflow).
A natural-language interface so you can ask questions without writing SQL (Genie).
An agent layer that turns questions into multi-step reasoning instead of one-shot lookups (Mosaic AI).
A measurement framework so you can trust the answers (Mosaic AI Agent Evaluation).

Each of those took me a weekend or less to wire up. None of them required code you couldn't read. Most of the time I spent on this project wasn't writing software — it was making decisions about which building block to reach for, and why. That's an analyst's skill set, not a developer's, and the entry ramp into real production-shaped data analytics is shorter than most people think.

If your current "data platform" is a folder of Excel exports and a few PowerBI dashboards that someone built three years ago, you are not behind. You are exactly the audience for this architecture. The work I did to wire up my two-dozen-guy golf league is the same work you'd do to wire up a twenty-line factory — and the same first ten findings will surprise you the same way the wind-cliff surprised me.

That's the move from spreadsheets to a lakehouse. It is genuinely worth making, and the on-ramp is shorter than the version of you reading this post probably believes.

What's next

A few threads from this post are worth following independently:

Move the business logic out of the prompt. Right now the caddie’s domain rules — gross vs. net vs. match-play points, "sum match_play_points for standings" — live in the agent’s system prompt. The more durable home is a Unity Catalog metric view: define those semantics once in UC and let the agent, Genie, and the dashboards consume the same definition instead of each re-encoding it. Pair that with Genie’s real accuracy levers — certified queries, example SQL, and general instructions (the one-paragraph instruction tweak that moved my eval from 57% to 79% was exactly this) — and the natural-language layer gets both smarter and more consistent without touching agent code.
Real per-user identity in the agent path. Right now the webapp invokes the agent as its own Container App Managed Identity. For a multi-user app — or a Teams/Copilot bot — the right pattern is on-behalf-of-user token exchange so per-user UC ACLs are enforced. That's an interesting governance demo I want to write up next.
Multi-tool agents and Agent Bricks. This agent uses a single tool (Genie). The natural next step is a second tool — a weather forecast lookup, say, so the caddie can recommend "play long off the tee, the wind is dropping at 6 pm" rather than only doing retrospective analytics. That's also where the line between hand-assembled agents and Agent Bricks gets interesting.
Eval as a CI gate. The eval notebook runs in twelve minutes. The natural next move is to wire it into a Databricks Workflow that runs after every model version is logged and compares correctness against the previous baseline. Regression-as-a-test, not vibes-as-a-test.
Genie Agent mode. Databricks recently introduced Agent mode on Genie spaces — the space itself acts as an agent, reasoning across its instructions, certified queries, and tables without a separate serving endpoint. Same tables, same curation loop from Part 1.5 — but the custom Mosaic AI agent, the base64-embedded code, the hidden runtime service principal, and the version-dance from earlier in this post all collapse out for read-mostly conversational flows. Currently UX-only (no API yet), so I'm not rebuilding today, but every hour I put into space curation now compounds when Agent mode's API surface ships. It's the shape most of the golf-agent traffic probably wants to live in eventually.

If any of those sound interesting — drop a comment or DM me. I'm just landing at Databricks and the lens through which I'm looking at all of this is "what would my team help a customer build next, and what would we tell them to avoid?" Both halves of that question have answers worth writing down.

Thanks for reading. If you want to start from the beginning, Part 1 is here.