Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.

A friendly, deep-ish dive into Consistency-based Self-adaptive Prompting (COSP), where LLMs build their own in-context examples with zero labels.

If you have ever tried to squeeze better reasoning out of a large language model, you have probably lived through one of these two nightmares.

Nightmare one: you go full few-shot.

You handcraft a few beautiful, worked-out examples, paste them into the prompt, and the model suddenly gets smart.

Except now you have to do that for every single task, and if you pick slightly different examples the accuracy wobbles like a shopping cart with one bad wheel.

Nightmare two: you go full zero-shot with the classic "Let's think step by step" trick.

No examples, no labor, very clean.

But without guidance the model sometimes wanders off into a reasoning path that sounds confident and is completely wrong.

So one path costs you effort. The other path costs you accuracy. Pick your poison.

Today I want to talk about a paper that basically said "why not neither" and got away with it.

It is called Consistency-based Self-adaptive Prompting, or COSP, from Wan and colleagues at Google. And the core idea is delightfully cheeky: let the model write its own in-context examples, using nothing but its own answers.

Let me walk you through it, because it is genuinely clever and you can steal the mental model even if you never touch the exact implementation.

The problem, illustrated by my own suffering

Before COSP, the "make my LLM reason better" toolbox mostly looked like a choice between doing a lot of manual work or accepting mediocre results.

The few-shot route is powerful but high maintenance.

You are not just writing examples once.

You are writing examples per task, re-checking them when the task shifts, and quietly praying your handpicked demos are representative.

It scales about as well as folding a fitted sheet.

The zero-shot route dodges all that labor, but it is flying blind.

There is no anchor telling the model what a good answer looks like, so it is easy for it to commit to a bad line of reasoning early and never recover.

COSP looks at both of these and asks a very good question: what if the examples do not have to come from us at all?

The big idea: the model is already generating great examples, we just do not use them

Here is the insight that makes the whole thing click.

When you ask a model a question several times, it does not always give you the same answer.

Sometimes it gives you the same answer four times out of five.

Sometimes it gives you five completely different answers.

That spread is not noise. That spread is information.

If the model keeps landing on the same answer across independent attempts, it is quietly telling you "I am pretty sure about this one."

If it is all over the place, it is telling you "I am guessing, please do not trust me here."

COSP uses that signal to pick which of the model's own responses are trustworthy enough to become in-context examples for a second pass.

The model produces the raw material, and consistency acts as the quality filter.

It is the best of both worlds fused into one move: zero-shot's zero-labor plus few-shot's guided accuracy.

How COSP actually works

COSP runs in two stages, then finishes with a vote. Here is the whole thing at a glance.

Let me break down the three scoring criteria, because this is where the paper earns its lunch.

1. Consistency. For each question, the model samples several reasoning paths.

COSP looks at how much the final answers agree.

Strong agreement means low uncertainty, which is a good sign the reasoning is solid.

The paper quantifies this with entropy over the answer distribution: low entropy means the answers cluster tightly, high entropy means chaos.

Low entropy responses are the candidates worth promoting to example status.

2. Repetition. Ever seen a model get stuck in a loop and repeat the same phrase like a scratched record? That kind of degenerate, repetitive output correlates with bad reasoning.

COSP penalizes it, so those responses do not sneak into the example set just because they happened to be "consistent" with themselves.

3. Diversity. If all your chosen examples are basically the same question in a trench coat, they do not teach the model much.

COSP encourages the selected demos to be varied, so the final prompt covers different flavors of the problem rather than four clones.

Put those together and you get a set of self-generated, self-vetted examples. COSP prepends them to the original questions, runs the model again, and takes a majority vote across the reasoning paths for the final answer.

It is essentially self-consistency wearing a nicer coat, with the added twist that the model's own confident answers become the teaching material.

Consistency as a confidence meter, with a worked example

Let me make the consistency idea concrete, because it is the heart of everything.

Say we ask this the classic way: "Henry had 11 dollars, got 18 more, then spent 10 on a game. How much does he have now?" We sample the model a few times and get a spread of answers.

Three out of four attempts land on 19.

That tight cluster is low entropy, which reads as high confidence, which makes this a trustworthy example to reuse.

The lone 27 gets outvoted and does not derail things.

Consistency did the quality control that a human labeler would normally have to do by hand.

But wait, is consistency the same as correctness?

No. And this is the part I want you to keep in the back of your head, because it is easy to get starry-eyed here.

Consistency is a proxy for confidence, not a guarantee of truth.

A model can be consistently, enthusiastically, repeatedly wrong.

If it has a systematic misconception, it will produce the same wrong answer over and over with total conviction, and COSP will happily hand that a gold star.

The reason COSP still works well in practice is that its criteria stack the odds in your favor rather than trusting any single signal.

Consistency, plus the repetition penalty, plus diversity, plus a final majority vote, together filter out a lot of the junk. It is not magic.

It is good statistics dressed as a prompting trick.

Does it actually help? The numbers

The honest answer: yes, meaningfully, and mostly for free.

Across the reasoning benchmarks in the paper, COSP reliably beats plain zero-shot chain-of-thought, and it often matches or edges past few-shot, without any handcrafted examples or labels.

The headline figure the authors report is an improvement of up to about 15 percent over zero-shot baselines, tested across three different LLMs.

A rough sense of the shape of the results on reasoning tasks:

Notice the pattern.

On several tasks COSP quietly overtakes even the few-shot setup that required human effort.

On the harder ones it lands in the same neighborhood.

Either way you are getting few-shot-tier results while paying zero-shot prices, which is a pretty good deal.

The takeaway

COSP's whole personality is that the model is already doing most of the work, we were just throwing the useful part away.

Instead of us curating examples, the model generates candidates, consistency filters them, and a vote seals the deal.

The mental model is worth keeping even outside this exact method: treat agreement across samples as a confidence signal, and let your best self-generated outputs bootstrap your next prompt.

That idea shows up all over modern LLM tooling now, and COSP is one of the cleaner places to first meet it.

If you take one thing away, let it be this: your model has opinions about which of its own answers are good. It would be rude not to ask.

---

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

⭐ Star it on GitHub: