Synthetic Data and LLM Eval Sets

I've been building promptcrucible, a multi-agent workbench for optimizing prompts against real eval harnesses. (It's in active development, so treat this as notes from the trenches, not a launch post.) Going in, I assumed the interesting problem would be the optimization loop — the search over prompt variants, the agent orchestration, the scoring. That part was tractable.

The hard part, every time, was the eval set. Specifically: how do you get enough labeled examples to trust a score? And the seductive answer everyone reaches for is generate them. Have a strong model synthesize test cases. It's fast, it's cheap, and it will quietly ruin your evaluation.

Synthetic data tells you what you want to hear

Here's the trap. You use a capable model to generate eval examples. Those examples inherit the generating model's distribution — its phrasings, its blind spots, its idea of what a "typical" case looks like. Then you optimize your prompt to score well on them. You've built a closed loop that rewards agreement with the generator, not correctness in the world.

The scores go up. Everyone's happy. Then production traffic arrives — with its typos, its adversarial users, its weird edge formats, its long tail the generator never imagined — and the gains evaporate. You didn't measure quality. You measured similarity to synthetic data, and optimized straight into it.

The specific failure modes

Distribution shift. Synthetic examples cluster around the generator's notion of normal. Real inputs have a fat, ugly tail. Optimizing on the cluster leaves you blind to the tail — which is where production failures actually live.
Overfitting to the eval. Any fixed eval set can be gamed by enough optimization pressure, and automated optimization applies a lot of pressure. Synthetic sets are especially easy to overfit because they're internally consistent in ways real data never is.
Label leakage via LLM-as-judge. If the same family of model both generates the cases and grades the answers, you've built a hall of mirrors. The judge rewards outputs that look like what the generator would produce — not what's correct.

What I do instead

None of this means "never use synthetic data." It means don't let it be your ground truth. The practices that survived contact with production:

Anchor on real examples, even a few. A small set of genuine, human-labeled cases from actual traffic beats a large synthetic set for measuring anything that matters. Quality and provenance over volume.
Use synthetic data for coverage, not scoring. It's good for stress-testing edge cases you can then verify by hand — a way to find hard cases, not to grade them.
Keep a held-out set the optimizer never sees. If your loop can see it, your loop will overfit it. Guard a real, untouched set for the final honest read.
Be paranoid about LLM-as-judge. It's a useful tool with a real bias. Spot-check the judge against human labels, and never let the generator and the judge be the same model on the same task. Treat "the judge agreed" as evidence, not proof.
Watch for the generalization gap. Eval score up, production quality flat is the tell that you've optimized into your eval set rather than into reality.

The uncomfortable conclusion

The bottleneck in evaluation is, and remains, honest labeled data — and there's no synthetic shortcut around it, only synthetic ways to fool yourself faster. The whole point of an eval set is to be an adversary to your optimizer: to not move when you're gaming it and to only move when you've genuinely improved. Synthetic data, generated by the same kind of system you're evaluating, is a poor adversary. It's on your side, which is exactly the problem.

Automated prompt optimization is real and it works. But it's only as trustworthy as the eval it optimizes against — and building an eval you can trust is the actual engineering. The optimizer is the easy part.