Picking an open table format bought you exactly one guarantee: any engine that speaks the format can read the files.
Writing is a different promise, and so is governance. The format makes neither. Whether a table stays correct depends on who commits to it, and who sees which rows depends on where the policies live. A second engine that writes, or reads around the catalog, can break both without the table’s owner seeing an error.
When a second engine commits
Commits to an open-format table are optimistic. A writer builds new data files, then tries to swap the table’s current snapshot for its own in one atomic step. If someone committed in between, the writer refreshes and retries. Plain appends usually survive that. When both writers touch the same files or rows, say a MERGE against a stream that’s upserting the same keys, the conflict check rejects the MERGE or its retries run out. The job fails, and the alert lands with that engine’s team, not the table’s owner.
The second engine also brings its own MERGE behavior and, unless someone stops it, its own maintenance jobs. And an engine that reads files by path never asks the catalog anything, so the row filters and column masks defined there don’t apply to it.
Before a second engine touches a table, it has to clear two rules.
Exactly one writer per table
One engine and one service identity commit to each table. Every other engine gets read-only grants through the catalog.
Maintenance counts as writing. Compaction, snapshot expiry and orphan-file cleanup all rewrite or delete files, so the owner runs them. Late corrections go in as another input to the same writer, never as a second MERGE from a different engine.
The CRM sync post made the same call field by field. A table is a bigger unit with the same failure: leave the owner unnamed and the conflict check settles it, one failed job at a time.
Policies defined once, in front of the files
Row filters and column masks get defined in one place: the catalog, or a policy layer in front of it. Not once per engine. An engine that can’t enforce them gets no access to governed tables, which beats leaving it a path to the raw files.
That rule has a hard dependency. Lock down the storage paths. If anything can read the bucket directly, the catalog’s masks are decorative.
Late returns and an unmasked email column
Picture a retailer, made up for this post, whose orders table is written by a streaming ingestion job while a nightly warehouse job runs a MERGE on it to apply late returns.
On nights when the two overlap, the MERGE loses the conflict check and fails. The failure sits in the warehouse’s job log, which nobody on the orders team reads, and return totals drift for a week. Separately, an analyst’s notebook engine reads the files by path, so the email column, masked everywhere else, shows up in plain text.
Most of the fix is taking things away. Returns become a correction stream that the ingestion writer consumes next to new orders. The warehouse keeps read and loses write. The notebook engine reads through the catalog with masking applied, or it doesn’t read the table at all.
A month later somebody ships a new job that commits to orders under the ingest identity. The grants look fine, since that identity may write, but the next morning’s commit check flags commits whose writer-id isn’t in the manifest.
The tables manifest
Rules only help if a machine can check them, so these live in git:
# tables.yml: one entry per governed table
retail.orders:
writer:
engine: stream-ingest
identity: svc-orders-ingest
writer_id: orders-ingest
readers:
- { engine: warehouse, identity: svc-warehouse-read }
- { engine: notebooks, identity: grp-analysts }
maintenance:
owner: orders-data-team
runs_as: svc-orders-ingest
writer_id: orders-maint
snapshot_expiry_days: 7
policy_tags:
email: pii-mask
region: row-filter-by-region
exit_notes: >
To move the writer, change engine, identity and runs_as, then
move the write grant. Readers and policies stay put.
The manifest doubles as the exit plan: swapping the writing engine is a few changed lines and one grant, reviewed like any other diff.
Reading the commit history
Every writer job stamps a custom snapshot property, writer-id, on each commit. The main writers accept one per write, though some only through their API, not plain SQL. A nightly check reads the table’s snapshots metadata table, whose name and shape vary by format and engine:
-- Generated per table from tables.yml
SELECT committed_at, snapshot_id, operation,
summary['writer-id'] AS writer_id
FROM lake.retail.orders.snapshots
WHERE committed_at >= current_timestamp() - INTERVAL 1 DAY
AND (summary['writer-id'] IS NULL
OR summary['writer-id'] NOT IN ('orders-ingest', 'orders-maint'))
ORDER BY committed_at;
Any row goes to the maintenance owner. Keep snapshot expiry well past the check window, or there’s nothing left to read.
Grants are the other half. The same nightly run diffs the catalog’s grants against the manifest:
import sys
import yaml
# Adjust to your catalog's privilege names.
WRITE_PRIVILEGES = {"INSERT", "UPDATE", "DELETE", "MODIFY", "OWNERSHIP"}
def fetch_grants(table):
"""Return (principal, privilege) pairs from your catalog's grants API."""
raise NotImplementedError
def check(table, spec):
owner = spec["writer"]["identity"]
readers = {r["identity"] for r in spec["readers"]}
for principal, privilege in fetch_grants(table):
if privilege.upper() in WRITE_PRIVILEGES and principal != owner:
yield f"{table}: {principal} has {privilege}; only {owner} may write"
elif principal not in readers | {owner}:
yield f"{table}: {principal} has access but is not in the manifest"
with open("tables.yml") as f:
manifest = yaml.safe_load(f)
problems = [p for table, spec in manifest.items() for p in check(table, spec)]
for problem in problems:
print(problem)
sys.exit(1 if problems else 0)
What the pull request has to say
Adding an engine means a pull request to tables.yml, and its description answers five things: which table it will write, if any, and which engine gives that table up; whether it enforces the catalog’s filters and masks on every governed table it reads; whether it holds any credential for the storage paths; whether every job it runs stamps writer-id; and which lines and grants change if it’s dropped next year.
The jobs that have to move
The first rule has a price. The retailer’s returns logic gets rebuilt inside the streaming job, real work for a team that only wanted a nightly MERGE.
Locking down raw paths will also break someone’s notebook, usually someone who never did anything wrong. Hand them a connection through the catalog the same day the bucket credential goes away, and most of those notebooks keep running, just with the email column masked.