Genie learns: making a natural-language layer smarter without more code

A middle chapter in the personal-lakehouse series. In Part 1 I built the platform. In Part 2 I add data and evaluate the agent. This post sits in between — and asks a question I didn't answer well the first time: how does the natural-language layer get smarter over time?**

The gap I want to close

If you read Part 1 of this series, you saw me stand up a Genie space over a governed lakehouse and give it a Genie agent as its natural-language front door. The way I made the space "good" was to write instructions, glossary terms, and a handful of trusted example SQL queries by hand — a YAML file in source control called ontology.yaml that I pasted into the Genie UI at setup time. Then I moved on.

That works. It also leaves a huge chunk of Genie's actual value on the table.

Here's the thing I glossed over in Part 1: Genie is designed to learn from the questions users actually ask. Every question that goes through the space, every SQL statement it generates, every thumbs-up or thumbs-down, every correction lives in the space's history. If you treat that history as material a steward can promote into trusted examples — instead of just a log to browse — the space you shipped on day one is not the space you'll have on day ninety. It's genuinely smarter. And the work you do to make that happen is mostly review and certification, not engineering.

A quick reality check up front: Genie does not silently self-train on unlabeled history. No background process watches your Monitor tab and promotes queries on its own — and it shouldn't, because it has no reliable signal for whether a query was actually correct. What Genie does do is turn every human-certified answer into a few-shot example that guides all future similar questions. So the learning isn't magic; it's platform-native pattern of "you certify, everyone benefits from that certification forever after." That's a much more defensible engineering pattern than "auto-learn," and it's the one that scales.

This post is the middle chapter I owe the series: what the learning mechanisms actually are, what I did (and didn't do) with them the first time through, and what curation looks like on a small league's-worth of usage.

What Genie DOES adapt to on its own

Before I get to the human-in-the-loop part, it's worth calling out what the space picks up passively — because these are real intelligence benefits you get without any curation work at all:

Sample data awareness. Genie samples rows from every table it queries and uses those samples as context when generating SQL. So as your data changes over time — a new player joins the league, a new record heat day gets recorded, a new course gets added — Genie's SQL adapts on its own without any human involvement.
Schema drift. Add a new column, rename an old one, change a type. Next query, Genie's aware. No re-configuration required.
Table scope. Adding or removing a table from the space's Data list changes Genie's reasoning space immediately. You don't have to re-train anything.
Platform model improvements. Databricks periodically upgrades the LLM behind Genie. Your space benefits automatically the day that happens, on questions you've never even seen.

None of these are "learning from users" in the strict sense, but they're all reasons the space you ship today isn't a fixed artifact. It adapts to the data it's operating over, whether you review the Monitor tab or not.

The rest of this post is about the deliberate learning — the part you get from certification, which compounds on top of everything above.

Two ways to make a Genie space better

Any natural-language-over-SQL system faces the same core question: how do you close the gap between what a user asks and the query they actually needed? Genie gives you two knobs, and they're complementary, not competing.

1. Static tuning (what I did in Part 1)

Before anyone even opens the space, you can:

Write a description of what the space is for, so Genie knows the domain.
Add instructions — defaults, terminology, formatting rules ("this season = MAX(season_year)"; "front 9 = course_side 1"; "always exclude the placeholder Twin Hills sub records").
Publish sample / trusted queries — pre-authored SQL for common questions, which Genie uses as few-shot examples when it generates new queries.
Maintain a glossary that maps business terms to columns ("handicap" → silver.players.current_handicap).

This is the shape Part 1 focused on. It's fast to set up, it's version-controllable (mine lives in ontology.yaml), and it gets you a competent day-one space.

The catch: every new question your users ask is either already covered by your static examples or it isn't. If it isn't, you're relying on Genie's LLM to figure out the right join, filter, and grain on its own.

2. Dynamic tuning (what this post is about)

The complement to hand-authored examples is curating what users actually ask. Genie captures every interaction in the space — the natural-language question, the SQL it generated, the result it returned, and whatever feedback the user left on the answer. That log is a goldmine for making the space smarter without guessing:

Verified answers. When Genie generates the right SQL for a question, a data steward can mark that Q&A pair as "verified." From that point forward, whenever a user asks a similar question, Genie recognizes the pattern and reuses the certified query instead of re-generating from scratch.
User feedback (thumbs up/down). Right in the Genie UI, users mark responses as helpful or unhelpful. Thumbs-up on a novel question is a candidate for verification. Thumbs-down is a signal that the SQL missed the intent — worth reviewing, correcting, and either re-certifying or turning into a new instruction.
Historical review. The Monitor tab lists every question in chronological order with the generated SQL and status. A weekly review cycle — "what did people ask that we didn't already have a good answer for?" — turns actual usage into new trusted assets.
Query correction. If Genie generated almost the right SQL, you can edit it directly in the interaction record and save the corrected version as the new canonical example.

Put together, these give you a feedback loop: users ask → Genie tries → users signal → stewards curate → Genie improves. The natural-language layer becomes something that starts good and gets better as your organization uses it, not something that stays fixed until the next dev sprint.

There's also a strategic reason to invest in this loop now, even if you're currently fronting the space with a custom Mosaic AI agent (as I am). Databricks recently introduced Agent mode on Genie spaces — a mode where the space itself acts as an agent, reasoning across your tables, instructions, and certified queries without a separate serving endpoint. Every hour you put into space curation today pays double: your custom agent gets sharper, and the eventual Agent-mode flip is a configuration change instead of a rebuild. More on that below.

What "day one done" leaves out — and why a product owner matters

If you're building your first Genie space, the static path is what you have to do to get an answer at all. Instructions, glossary, a handful of trusted queries, table selection, permissions. That's the day-one build. It's what Part 1 walked through, and it's what any team stands up first.

What day-one done doesn't include — and what most teams underestimate — is the fact that the agent has a lifecycle. It ships, it starts collecting real interactions, users start asking things nobody predicted, the data underneath it drifts, and instructions that were fine on day one stop landing correctly six weeks later. The static configuration is the shell. Someone has to operate the inside.

That "someone" is a role most teams don't staff on day one, and it's the biggest gap I see with customers moving from proof-of-concept to production: the agent needs a product owner.

Not the builder. Not the platform team. A product owner — the person responsible for the agent's effectiveness over time. Someone whose job is to:

Watch what real users actually ask and how the agent responds
Certify the responses that were correct so they become permanent examples
Correct the ones that were wrong and add clarifying instructions
Notice when the space's answer quality is drifting and take action before users complain
Own the eval set — what "correct" means for this agent, in this domain, for these users
Speak for the users when there's tension between "the agent can do X" and "the agent should do X"

In every discipline that ships software to end users, there's a role like this. In consumer product, it's a PM. In enterprise BI, it's a data steward or a semantic layer owner. In modern data platforms, it's the person who owns the agent's KPIs — accuracy, adoption, latency, cost per query — and treats them like they own a P&L.

Without this role, the space you shipped on day one is exactly as smart today as it was on day one. No amount of platform investment fixes that. Genie can give you every mechanism to learn from users, but if nobody's reviewing and certifying, none of those mechanisms activate. The Monitor tab fills up with unpromoted questions. The thumbs collect signal that nobody reads. The eval never gets updated. The space stagnates while your users' questions evolve.

The pattern that separates a demo from a production agent has almost nothing to do with technology and almost everything to do with staffing that role. Someone owns the effectiveness of this agent. It is their job. When that's true, the rest of the machinery I describe below works. When it's not, no machinery matters.

Trust is the shipping constraint

Worth being blunt about, because it's the difference that catches most teams off guard: when an agent gives users bad answers, they don't file a bug — they leave.

Traditional tools train users to be forgiving. Search engines miss all the time, and people scan the second and third results without thinking twice. Spreadsheets return wrong numbers routinely, and users cross-check them against another sheet. Dashboards go stale, and analysts adjust. That forgiveness is baked into how users interact with those tools — no single output is expected to be the final word.

Agentic solutions come in with a different implicit contract. When a user asks a natural-language question and gets a natural-language answer back, the answer they get is the answer, in their head. If that answer is wrong or vague three times in the first week, they don't tune their prompts and try again — they go back to Excel, they open the SQL editor themselves, or they ask the analyst who used to do this by hand. And once they've reverted to a workaround, getting them back is very hard. You've moved a habit, and habits are sticky.

This is why the product-owner role isn't a luxury — it's a change-management function. Platform teams and builders (myself included) love shipping fast, and Databricks genuinely makes shipping easy — a Genie space can be up in an afternoon, an agent wrapper in a weekend. But shipping is not landing. Landing is when users trust the tool enough to rely on it for real work. And that trust is what the product owner is protecting on every certification, every clarification instruction, every eval pass. Every one of those actions is a small deposit into the trust account users hold with the agent.

If you're a developer or platform lead reading this, the temptation is to treat the product-owner work as optional polish that "we'll do later." It isn't. In agentic systems, quality is adoption. Sloppy answers early on don't just lose one user's confidence — they set the ceiling on how far the whole solution ever spreads inside your organization. Ship fast, but build trust faster. That's the actual constraint you're operating under, whether or not the roadmap admits it.

For my golf league, that role is me. It's a fifteen-minute-a-week job at league scale. But the shape of the work — walk the Monitor, certify what was right, add instructions for what was ambiguous, review the eval numbers — is identical to what a full-time agent product owner would do for a 200-user shop-floor deployment. The volume changes; the workflow doesn't.

What that shift means for how I'm working the golf agent

I'm mid-development on the golf agent, so this isn't a "here's what I already did" section. It's a "here's what I'm adopting now that I understand the role" section. Concretely, as I move this agent toward being something my league buddies actually use week-over-week:

I'm treating myself as the product owner, not just the builder. That means a fifteen-minute weekly review is now on my calendar — even though the space is technically working today.
I'm turning on user feedback (thumbs) before the buddies get access. The signal only starts collecting when it's enabled, and the sooner it starts, the sooner curation has material to work with.
I'm going to certify from my own dev-and-demo history first. Every question I've asked the space during development is sitting in the Monitor — most of it produced correct SQL. Promoting those is free intelligence I already earned.
I'm writing a lightweight "definition of correct" — what counts as a good answer for this domain, so future-me (or a future steward) doesn't have to re-derive that judgment every review cycle.
I'm going to treat the CLEARS eval from Part 2 as the regression gate — every material change to the space triggers a re-run, and I don't ship a change that regresses the score without a documented reason.

Those five things don't need new engineering. They're all product-owner discipline. And they're what separates building an agent from shipping one.

What the curation loop actually looks like

Here's the workflow I'm implementing now, sized for a personal lakehouse with light traffic. The same shape scales to a hundred users; you just add reviewers and cadence.

Step 1 — Enable user feedback

In the space's settings, turn on the thumbs-up/thumbs-down icons on every response. This is the cheapest signal you can collect, and unlike an eval framework, it happens for every real interaction — not just the fourteen questions in your golden set.

For my golf league, "user" is a small handful of people: me, my brother, maybe two league buddies who want to poke around. Even at that scale, the feedback signal is more useful than it sounds — because when any of them asks a question that isn't in my head, that's a new question I would never have written into the trusted-queries list.

Step 2 — Weekly Monitor review

Open the Monitor tab, sort by most recent, and walk through the interactions from the last seven days. For each one:

Was the generated SQL right? If yes, was the response wording right too? (The agent narrates on top of Genie's data; both layers can independently be wrong.)
If the SQL was right but the question is one people are likely to ask again, certify it as a verified answer. That's a two-click promotion in the UI.
If the SQL was almost right — wrong join grain, missing a filter, ordered the wrong way — edit the SQL directly on the interaction record and save the corrected version as a verified answer.
If the question is genuinely ambiguous ("show me Jeff" when there are two Jeffs), add a clarification instruction to the space so future asks get resolved.
If a question is off-topic or malicious, ignore it.

This is not a huge time investment at league scale — maybe 10-15 minutes a week. At enterprise scale it's a role — the data steward who owns the space. Same activity, different volume.

Step 3 — Promote trusted assets from your own history

The ontology.yaml file I hand-wrote for Part 1 contained eight sample queries. Genie's Monitor tab has hundreds of real interactions from my dev-and-demo period. Instead of only using the eight I guessed at up front, I can walk backward through the Monitor and promote the ones that actually got asked. That's a much better source of trusted examples than my imagination.

Step 4 — Feed learnings back into instructions

Some patterns you see in the Monitor aren't about specific SQL — they're about the space's overall behavior. If three users in a row ask a variant of "who's playing well lately" and get slightly different windows (some Genie runs use 30 days, some 60, some "this season"), the fix isn't to certify one query — it's to add an instruction that says "'lately' means the last 30 days by default, unless a longer window is specified." Now every future ask lands the same way.

Step 5 — Repeat forever

The static content (instructions, glossary, hand-written examples) gets updated maybe once a month, if that. The verified-answer library grows every week. Together they form the space's institutional memory — the part that carries forward even as staff change and questions evolve.

How this pairs with CLEARS-style evaluation

Part 2 is going to talk in detail about building a proper CLEARS-style eval — a fixed golden set that scores the agent on correctness, safety, and relevance. Verified answers and CLEARS are complementary, and it's worth being clear about which does what:

Mechanism	Ground truth from	Runs when	Best for
Verified answers	The steward reviewing actual usage	Every user interaction	Ongoing curation. Making the space smarter as new questions surface, using traffic you already have.
CLEARS-style eval	The rubric author writing expected facts	On demand, and before every deploy	Regression testing. Catching cases where a prompt change or new table broke something that used to work.

Neither replaces the other. Verified answers make the space drift toward good through usage. CLEARS eval catches you when a change drifts it away from good against your fixed contract. You want both. Personal-scale, weekend-only projects can get away with either; anything with real users needs both.

Agent mode: when the space becomes the agent

Everything in this post so far has assumed the shape I built in Part 1: a custom Mosaic AI agent that acts as the natural-language front door and calls the Genie space as one of its tools. That's a valid pattern, and it's still where you land if you need multiple tools, custom pre- and post-processing, or on-behalf-of-user token exchange.

But there's a newer shape worth calling out — one that changes the argument for why space curation matters. Databricks recently introduced Agent mode on Genie spaces. In Agent mode, the space itself reasons agentically over its own configuration: your instructions, your glossary, your certified queries, and your tables' sample data become the runtime brain of a conversational agent, without a separate serving endpoint or custom agent code.

An honest caveat first: as of this writing, Agent mode is UX-only. The Databricks docs are explicit that programmatic API access isn't yet available; you invoke it from the Genie space UI, not from a webapp calling a REST endpoint. So for anyone reading this planning to embed a conversational agent in an application right now, the custom-Mosaic-AI path (what I built in Part 1) is still the shipping path. Treat Agent mode as an experimentation surface today, and as a foreseeable evolution of the same space you're already curating.

That framing matters, because it tightens the argument this whole post has been making. When your Genie space is just a tool called by a Mosaic AI agent, curation quality shows up indirectly — in the SQL Genie generates for the agent's tool calls. In Agent mode, curation quality is the agent's quality. Instructions aren't a hint anymore; they're the reasoning substrate. Certified queries aren't examples; they're the operating manual the agent uses to answer users. Same investment, direct payoff.

The architectural picture also collapses. With a custom agent in front, the read path looks like:

webapp → Mosaic AI agent (custom code) → Genie space → UC tables

Four moving parts, each with its own identity, permission surface, deploy notebook, and version. In Agent mode, the read path shortens to:

UX → Agent-mode Genie space → UC tables

Two moving parts. The custom agent, the base64-embedded code snapshot, the hidden system service principal, the serving endpoint version dance from Part 2 — all of it goes away for the read-mostly cases where a single well-tuned space is enough. You keep the custom-agent path for the complex ones (multi-tool orchestration, on-behalf-of-user auth, external system integration).

If you're planning to try Agent mode on a space, Databricks' own setup guidance is the right anchor:

Genie — Set up a Genie space — initial configuration and required permissions
Genie — Best practices — how to structure an effective space for agentic use
Genie — Tune space quality — the same instructions, SQL examples, and knowledge-store levers this post has been arguing for, from the platform team's own words

The prep work isn't different from what a well-curated Mosaic AI-fronted space needs. It's the same prep work — just now with a first-class runtime that consumes it directly. That's exactly the compounding return the "static vs dynamic tuning" section was pointing at: every curated asset earns interest from the day you certify it.

Testing an agent is not testing a search engine

I want to spend a moment on a testing failure mode I've seen more than once with customers evaluating natural-language interfaces for the first time. It's the reason I have three layers in the table above instead of one.

The trap looks like this. A team stands up a Genie space (or a Copilot, or any conversational data interface). To evaluate it, they compare it to what they already know — web search. So they test it the way they'd test Google or Bing:

Type in three or four keywords, no context ("golf handicap Jeff")
Look at the response
Thumbs up if it "worked," thumbs down if it didn't
Compare pass rates

Then they conclude the agent is worse than search, because for that testing style… it is. Search engines are optimized for the exact behavior they're testing: keyword-lookup, retrieval, ranked links, and a user who can scan ten results and pick the one they wanted. Conversational agents are optimized for the opposite behavior — natural questions with context, multi-step reasoning, and a shared understanding of what a "handicap" is and whose you're asking about.

The failure mode isn't the agent; it's the testing model. When you test an agent with three keywords and no context, you're evaluating it on the least favorable input the interface supports. That's a bit like judging a lawyer by handing them a phone number and asking them to argue a case.

Test agents the way they're meant to be used. Full sentences, natural context, follow-up turns when the first answer isn't quite right. And when you build the evaluation harness, do it in three overlapping layers instead of relying on one:

User feedback (thumbs up/down). Cheap, real, comes for free with every interaction — but noisy. A thumbs-down often signals "I didn't like the answer" rather than "the SQL was wrong." Valuable as a filter for review, not a source of truth.
Automated eval (CLEARS-style). A fixed golden set with expected facts per response, run before every deploy. Catches regressions cleanly. Requires investment to build, but it's the only signal that survives prompt-tuning changes.
Human oversight. A steward or data expert who periodically walks the Monitor tab, sanity-checks what the feedback and eval are telling you, and catches the cases neither signal captured — including cases where a thumbs-down was undeserved because the user tested with keyword-search expectations.

Any one of those signals alone is insufficient. All three together are what makes an agent's quality defensible — to your users, to your leadership, and to yourself when someone asks "how do you know it's any good?"

The customers I've watched succeed at this treat their agent quality the same way a factory treats process quality: sample-based inspection, statistical process control, and a formal review cadence. The customers who struggle are the ones who thumbs-down a keyword-search test on Monday and declare the pilot a failure on Tuesday. Framing matters as much as tooling.

Trying it on the golf space

I want to end this post with what I'm actually doing this week, so it's not just theory.

Turn on user feedback in the space so future asks capture thumbs data.
Walk the Monitor tab from the last thirty days and note the top ten questions that either haven't been asked before or that Genie handled well without an example. Certify those.
Find two or three "same intent, different phrasing" clusters and add an instruction rule for the canonical interpretation.
Retire any hand-authored sample queries that turned out to be too specific and never matched a real question — they add noise, not signal.
Prep the space for Agent mode. Turn it on in the space UI, walk through the readiness checklist in the Databricks setup and best-practices docs (see References), and see whether the same curated space can answer the golf-league questions directly, without the Mosaic AI custom agent in front. Even if you don't cut over, the readiness pass surfaces gaps the custom-agent path was hiding.

Then I'll come back and measure. If I re-run the CLEARS eval from Part 2 after this curation pass, does correctness go up? Does the agent land the right SQL on the first try more often? I don't know yet. The whole point of this post is to start the loop, not report a finish.

The enterprise parallel

Everything above scales to a real organization by changing exactly one thing: who does the review.

At personal scale, I'm the sole steward — I write the questions, I review the answers, I certify the good ones. In an enterprise data team, that role is a data steward or domain expert, sitting between the analyst users and the platform team. The review cadence gets more formal (a weekly working meeting instead of a Sunday-afternoon browse), but the workflow is the same: users ask → Genie tries → users signal → stewards curate → Genie improves.

The reason this matters more at scale is that you never have enough hand-authored examples to cover the questions a real org asks. A shop floor has hundreds of ways to phrase "why did we scrap so many parts on Tuesday" — you cannot pre-write them all. What you can do is set up the loop so the space learns what your specific plant's specific supervisors actually ask, and every certified answer becomes tribal knowledge that future users get for free.

That's the compounding return you don't get from a static configuration file. The natural-language layer becomes a living knowledge base, not a fixed API contract.

Where this fits in the series

Part 1 — From spreadsheets to a real Lakehouse. Build the platform. Ship a day-one Genie space and agent.
Part 1.5 — this post. Turn on the feedback loop. Curate what users actually ask. Make the space genuinely improve over time.
Part 2 — Real agents, real edge cases. Add new data sources, evaluate the agent honestly, break and fix the permission model.

The natural-language interface on top of a lakehouse is not a thing you configure once. It's a thing you operate. This post is the operating manual I wish I'd written between build and evaluate — because the sooner you close the loop, the faster the platform gets to "actually indispensable" rather than "cool demo."