Data checks

What the console can work out by reading the data, as opposed to Findings, which never opens a data file. These cost real bytes, so none of them runs unless somebody asks — and every one is priced before it does.

Three properties, and they are the whole design.

Quoted first. POST /scan/tables/{name}/plan weighs each check from the data-file footers, without opening a page. On a media table the quote carries the more interesting half: reading every video's descriptor costs kilobytes, and the gigabytes those descriptors point at are not read.

Stoppable, and it means it. Cancelling a scan stops the work between batches and reports the bytes it had spent. This is the one place in the product where cancel means cancel — a Lance query cannot be interrupted, and the query panel says so rather than pretending otherwise.

Measured afterwards. Every result carries what it actually read, drained from the same handle every other panel drains, so the quote can be checked against the outcome rather than believed.

missing-content

Rows with nothing in them.

Rows whose content is absent — a null field, or a blob of zero bytes.

The cheapest useful thing this layer does, and on a media table it is close to free. A Blob V2 column projects to its descriptor — position and size — not to its bytes, so asking whether 900,000 videos are all actually there reads the descriptors and none of the video. On this repository's own corpus that is 43.4 KB against the 2.65 GB it describes.

A null caption and a zero-byte video are the same failure to a training run: a row that will be batched, embedded and learned from with nothing in it.

class-balance

How the rows fall across classes.

How the rows fall across the classes of one column.

A model trained on a corpus that is 94% one label learns to say that label, and reports excellent accuracy for doing it. The row count never shows this and neither does anything in a manifest.

Reads one column. It reports the distribution and says nothing about whether the labels are correct, which is not visible from here and never will be.

exact-duplicates

Rows that are the same row twice.

Rows that are identical across the columns you named.

Grouped rather than compared: a group-by over the projection, so this is one pass and no pairwise anything. Exact — two rows either agree on those columns or they do not — which is the half of deduplication that can be answered without an opinion. The other half is near-duplicates, and it is approximate on purpose.

A duplicate that survives into a split is a row the model sees twice and an evaluation that scores it twice.

split-leakage

Items on both sides of the split.

Items that appear in more than one split.

Give it an identity column and a split column. It groups by the identity and counts how many distinct splits each one lands in; anything above one is an item the model trained on and was then evaluated against.

It checks the split column it was handed. It cannot see a leak that is semantic — the same photograph under two filenames is one identity to a human and two here — and saying so is part of the answer.

vector-health

Embeddings that are dead or broken.

Embeddings that are dead, broken, or not the shape the index expects.

Three failures that produce a table which looks entirely fine and retrieves nothing useful: an all-zero vector, where the embedder returned a blank and nobody checked; a NaN or an infinity, which poisons every distance it takes part in; and a mix of normalised and un-normalised rows, where a cosine index is being fed two different geometries.

Reads the vector column. That is the most expensive column in most tables, and the quote says so before this runs.

near-duplicates

Rows near enough to be the same item.

Rows whose embeddings are close enough to be the same item twice.

Approximate, because it asks the index — which is approximate — and because it asks about a sample rather than every row. Both are in the evidence, and neither is a defect to be fixed later: an exact answer is every pair compared to every other pair, and nobody is running that over a million rows to find out whether their corpus needs a closer look.

Refused where the column has no vector index, rather than falling back to a brute scan. The fallback would be one full pass over the vectors per sampled row, which is a bill nobody consented to when they pressed a button that says check.

The distance threshold is in the index's own units, and the metric it was built with is in the evidence — an l2 index and a cosine index do not mean the same thing by 0.02.

When a check will not run

A check reports the same three states a connection does — available, unsupported, unverified — with the reason attached. Two of them matter often:

• No default column. Which column holds a label, and which holds a split, are not visible in a schema. Guessing produces a real-looking answer to a question nobody asked, so these checks ask instead.
• No vector index. near-duplicates refuses rather than falling back to a brute scan, because the fallback is one full pass over every vector per row sampled — a bill nobody agreed to by pressing a button labelled check.

What none of these can tell you

Said here because a panel of green ticks implies otherwise. They cannot tell you whether a label is right; they cannot find a duplicate the embedding does not encode; they cannot attribute drift to a cause; and near-duplicate results are approximate by construction, because the index they ask is.