Agent Proposes, DQX Disposes

01Two design principles

Everything in the system falls out of two rules we decided not to break.

The agent never writes to Silver or Gold. Not once. Every agent-curated row is re-run through the same data-quality engine — we use DQX, the Databricks Labs quality framework — that quarantined it. If the “fix” doesn’t actually make the row pass the checks, it doesn’t get in. This keeps a hallucinated correction from silently corrupting the warehouse, and it makes the whole thing auditable: the gate is code, not vibes.

If a rule owner has already written down exactly how a violation should be remediated — “primary-key collision on customer_id: keep the row with the latest updated_at, drop the rest” — then that fix is applied deterministically and the LLM never sees the row. The agent is the fallback for rules nobody has written a playbook for yet. It is not the default path.

That second principle matters more than it sounds. The cheapest, most reliable, most auditable model call is the one you never make.

02The shape of the pipeline

The whole thing runs as a Databricks Workflow. Because a For Each task can only wrap a single task, not a sub-graph, the per-table pipeline is a child job that the parent fans out to — one child run per quarantine table.

Each stage writes only to a control table — a review queue with one row per quarantined row per pass. Nothing touches Silver until the final merge, and that merge only ever sees rows that have already passed re-validation.

03Triage

Pull the unprocessed rows from one quarantine table, dedupe to the latest quarantined version per row, and flatten the failure metadata. DQX records failures as an array of structs — {name, message, …} — and triage flattens those names into a plain array<string> of violated rule names, then seeds the review queue with status pending.

It also captures the business key at this moment. Reconstructing a natural key later, after the row has been patched, is guesswork; capturing it at quarantine time is not.

04Playbook remediation — the deterministic half

This is where documented fixes get applied. Each dataset has a YAML file — not in the codebase, but in a Unity Catalog Volume, so adding or changing a rule needs no deploy:

playbooks:

  # structurally broken — not a recoverable formatting slip
  - rule_name: code_not_matching_regex
    priority: 10
    where: "code rlike '^[A-Za-z]'"
    strategy: reject_rows
    params: { reason: "alphabetic prefix on a numeric code" }

  # documented tie-breaker: keep the most recently updated row
  - rule_name: struct_customer_id_order_ts_is_not_unique
    priority: 20
    strategy: dedupe_keep_latest
    params:
      partition_by: [customer_id, order_ts]
      order_by: updated_at
      order_direction: desc

Entries run in priority order; the first one that matches a row wins. Three things we learned to bake in:

• where predicates let one rule route by value. status = 'Duplicate' goes to dedupe; status = 'Invalid' has no entry, so it falls through to the agent. Same rule, different handling, driven by the data.
• reject_rows is a terminal state, not a failure. Some rows genuinely can’t be recovered. Dropping them is a decision the rule owner made on purpose, and it is recorded as such — not retried, not escalated, not seen by the agent.
• Dedup is configured only here. The agent never deduplicates. Choosing which of two near-identical rows survives is a business call with a documented tie-breaker, not something to infer.

The strategy functions — dedupe_keep_latest, standardize_value, fill_default, reject_rows — are code. A new kind of fix needs a deploy. A new instance of an existing kind is just a file upload. Behavior in code, policy in config: that split is what keeps the playbook maintainable by the people who own the rules rather than the people who own the pipeline.

05Agent curation — the fallback

Only rows with no playbook match reach the agent. And here is the decision that makes this affordable:

If 40,000 rows all failed the same check on the same column in the same way, that is one call. The model sees the rule name, the check function, the message, a sample of failing values, and up to five passing examples per flagged column. It returns a single deterministic transform per column, chosen from a fixed vocabulary:

Spark then applies that transform to every row in the signature. The model does not see 40,000 rows, does not write 40,000 fixes, and does not get to invent an expression. It picks lower_trim for a casing problem or regexp_replace to strip a known prefix, and that is the entire surface area. Token cost is independent of quarantine volume.

Transforms can be chained — strip a prefix, then zero-pad — which covers most of the real formatting damage: typos, column shifts, date formats, casing, padding, regex-strippable junk. And we guard against no-ops: if the model proposes left_pad on a string already at the target length, that is not a fix, and it gets downgraded to an escalation.

06The cross-table probe — no model at all

Referential-integrity checks — a row in table A must match a row in table B — are common, and the LLM is bad at them. So before the agent is involved, a deterministic probe runs: on a sample, normalize the compared column (trim, lower_trim, upper_trim) on both sides of the join and check whether the mismatch clears. If a normalization resolves, say, 95% of the sample, propose that transform. If it doesn’t, escalate.

A surprising fraction of “broken foreign keys” are just one side of a join carrying trailing whitespace.

07The confidence gate

The agent attaches a confidence score to every fix decision. Between curation and application, a gate checks the minimum confidence across the batch against a threshold. Above it: proceed to auto-apply. Below it: skip straight to audit, and those rows wait for a human.

One low-confidence signature holds back the whole batch for that table. We would rather under-automate than merge a shaky fix.

08Apply and re-validate

For rows that clear the gate: apply the patch, strip every bookkeeping column, and re-run the row through approved_checks_for(silver_table) — the same rules table that quarantined it. Not a copy of the checks. Not a checks file that might have drifted. The same source. “Re-run the exact checks that quarantined it” is only true if both sides read from one place.

Rows that pass are staged. Rows that fail get retry_count += 1 and go back into the queue. After three failed attempts a row auto-escalates regardless of confidence. Nothing loops forever.

09Reingest

A dynamic MERGE INTO Silver, keyed on the business key that triage captured, columns intersected with the target schema. Every reingested row is tagged with lineage: agent_curated = true, the confidence score, the run id, the timestamp. Downstream consumers who don’t trust agent-curated data can filter it out. That is their call to make, and we give them the column to make it with.

10Dry runs

Because “an LLM will now edit rows that flow into our warehouse” is a sentence that makes people nervous, there are two ways to see what would happen without committing anything.

• A read-only estimate that computes the deterministic half for real — match, apply strategy, re-validate — and reports the agent half as a count only. The model is never called. One row per table into a report table, nothing else.
• A full dry run that runs the whole pipeline, calls the agent, writes the queue and the staging table — but holds the final MERGE and instead reports the exact insert/update split against Silver.

Run the first before you turn the system on. Run the second in staging when you want to see precisely what will land.

11What this bought us

The quarantine table stopped being a graveyard. On the first real run, the large majority of quarantined rows resolved deterministically — dedup and structural rejects that rule owners had already documented. A few hundred went to the agent. A couple dozen genuinely needed a human, and those are the ones a person should be looking at.

The lesson we keep relearning: the LLM is the least important component. The value is in the scaffolding — the deterministic-first routing, the fixed transform vocabulary, the re-validation gate, the retry cap, the confidence threshold, the dry runs. The agent is a fallback that turns “a human triages every weird row” into “a human triages the genuinely ambiguous ones.”

Agent proposes. DQX disposes. Silver stays clean.