A reader read my benchmark better than I did

I published a LoRA fine-tuning run two weeks ago. The headline was that my first test set had lied to me: on data I generated myself the fine-tune scored 100% and prompting scored 94%, so fine-tuning looked pointless. On a test set rebuilt from real public corpora the fine-tune scored 95% and prompting collapsed to 66%.

A reader called Max Quimby left a comment that reframed the whole thing:

Five points and twenty-eight points. Your fine-tune lost the first moving to the rebuilt set, prompting lost the second, and the distance between those two drops tells you more than either one does. It says the original set was differentially easy for the prompted model rather than uniformly easy for both.

He then named a mechanism. The few-shot examples are part of the prompted method's input, so any kinship between the example pool and the test set inflates one arm and not the other. And he closed with a rule: when two methods move by very different amounts after a test-set swap, suspect the set before you suspect the methods.

I spent an afternoon checking it. He was right that my table was misleading, wrong about which arm was inflated, and the direct test of his mechanism produced a third answer that neither of us predicted.

The error in my table

Here is the comparison as I published it.

Read that as three methods measured on two sets and his conclusion follows immediately. But one row is not what it appears to be.

The few-shot and zero-shot rows are the same method twice: the same base model, the same prompt, the same six hand-written examples, evaluated on two different sets. Clean.

The fine-tune row is two different adapters. The 100% is an adapter trained on v1 data scoring on the v1 set. The 95% is a different adapter, trained on v2 data, scoring on the v2 set. I changed the training data and the test data at the same time, put both numbers in one row, and labelled the difference a drop.

That five was never a measure of set difficulty. Retraining on harder data was quietly absorbing the loss, and my table gave a reader no way to see it.

Holding the artifact fixed

The fix is to evaluate one artifact against both sets. The v1 adapter still exists in the repo, so this is one command:

.venv/bin/python evaluate.py --mode lora \
  --adapter-path ./adapters_v1_synthetic \
  --limit 400 --tag _v1adapter_realset

Thirty-three points. The same weights, the same code, the same prompt shape. Only the test set changed.

So the corrected picture, every row now a single method or artifact measured twice:

His instinct was right and his conclusion was backwards. The differential is real, but the fine-tune is the most inflated arm, not the least. And the ordering is the part worth keeping:

The more a method had been fitted to the v1 distribution, the more it lost when that distribution went away. Trained on v1 data: 33 points. Merely prompted with v1-shaped examples: 28. Never shown v1 at all: 22.

That monotonicity is the cleanest statement of the whole affair. It also rescues the 95%: retraining the same recipe on representative data took it from 67 back to 95, which makes that number a statement about the data, not about the method.

Testing his mechanism directly

The 22-point floor is what the set costs any model. The 6 points above it that few-shot paid are the candidate for his example-pool kinship, and my examples make the case look strong. Here are three of the six, next to the v1 generator templates they instantiate:

Same templates, different instances. Git history confirms I never touched them between the two evaluations, so the prompted arm carried v1-shaped hints into a v1-shaped test set, and then carried the same hints into a set they no longer matched.

If that kinship is what bought the 6 points, swapping the pool for one drawn from the v2 training data should recover them. So I built a second set of six: same count, same 3 positive / 3 negative balance, same three PII types, but every example lifted from data/train.jsonl and verified absent from the test set by exact string match.

One and a half points. Both arms ran on the identical test set, so this is a paired comparison and deserves a paired test rather than a glance at the margin of error. McNemar's exact test on the 400 prediction pairs: 21 items only the v1 pool got right, 27 only the v2 pool got right, p = 0.47.

Nothing. Matching the example pool's provenance to the test set recovers no measurable accuracy.

Then where did the 6 points go?

Back to the v1 set, where the examples were worth something, and compare them on the v2 set with the same paired test:

On the real set, adding six examples to the prompt does exactly nothing: 53 items only zero-shot got right, 54 only few-shot got right. A coin flip.

So the six examples were worth 6 points on the set that shared their provenance and 0 points on the set that did not. That is his differential inflation, confirmed. But his implied remedy, using examples of matching provenance, does not recover the loss, because the deficit was never really about provenance. Six examples of any origin cannot express this task's rules. On the easy v1 set they were enough. On the real one, no pool of six helps, so there is nothing for a better-chosen six to win back.

The per-type breakdown shows what in-domain examples actually do:

They redistribute rather than add. The types the examples demonstrate get better, the types they omit get worse, and the total barely moves. Few-shot prompting teaches what it shows and distracts from everything else.

What survives all of this

The fine-tune's advantage on the real set is not in doubt. Same paired test, fine-tune against the better of the two prompted arms: 4 items only few-shot got right, 113 only the fine-tune got right, p < 0.0001. A 113-to-4 split is not a benchmark artifact.

The rules I would keep

Report same-artifact numbers, or say plainly that you are not. My five-point drop was two different adapters wearing one row. Nobody could have caught that from the article, which is my fault and not the reader's.

Keep an arm with no exposure to the training distribution. Zero-shot was the only reason I could decompose 28 into 22 plus 6. Without a control that has never seen your data, a drop is just a drop.

A differential drop points at the set, but does not tell you which arm to blame. Rank your methods by how much they were fitted to the old distribution and check whether the drops follow that order. Here they did, exactly.

Use paired tests when arms share a test set. Two of the three differences that looked meaningful at a glance (1.5 points, 0.2 points) are indistinguishable from noise under McNemar, while the one that mattered came back at p < 0.0001. Margins of error on independent proportions would have let me argue for all three.

And the one that stings: a perfect score gets audited, a merely good one gets a slide. Max's point, and he is right. My 100% is what made me rebuild the set. If it had been 0.94 I would have shipped it, and if it had been 0.94 nobody would ever have found the two-adapters error in my table either.

Reproduce it

Everything is in the repository, including both example pools, the new results JSONs and the paired predictions the McNemar tests read:

git clone https://github.com/jguillaumesio/lora-pii-detection-mlx
cd lora-pii-detection-mlx
python3 -m venv .venv && .venv/bin/pip install mlx-lm datasets
.venv/bin/python build_dataset.py
.venv/bin/python evaluate.py --mode few-shot --few-shot-set v2 --limit 400 --tag _v2examples

Thanks to Max Quimby for the comment. It cost me an afternoon and a correction, which is the best possible outcome for a comment.

Originally published on jguillaumesio.com. Follow-up to My fine-tuned model scored 100%. The benchmark was lying.