LLM Confidence Score: 4 Recipes That Ship (2026)

Quick Answer

An LLM confidence score is a number between 0 and 1 that tells you how sure the model was about what it just said. In 2026 there are only two honest ways to get one: read the token log probabilities the provider returns, or make the model produce a score you can parse. Which one is available to you depends entirely on the provider. OpenAI, Gemini and Together return logprobs. Claude returns none at all. The four recipes below cover both cases.

The 2026 support matrix

Checked against each vendor's own API reference on August 19, 2026. This is the part every blog post skips.

Two things worth noticing. Gemini will hand you 20 alternative candidates per position where OpenAI caps you at 5, and Gemini is the only one of the four that also returns a ready made avgLogprobs for the whole candidate so you do not have to average it yourself. And Claude, the model many teams run in production, gives you nothing. The Messages API parameter list has temperature, top_k, top_p, stop_sequences and tool_choice, and no log probability field anywhere in the request or the response.

Recipe 1: Confidence from logprobs on OpenAI

Claim. One boolean gets you a real, model-internal confidence number. This is the only recipe here that is not a proxy.

Receipt.

r = client.chat.completions.create(
    model="gpt-5",
    logprobs=True,
    top_logprobs=5,
    messages=[{"role": "user", "content": "Sentiment of 'shipped late but it works'. One word."}],
)

import math
tok = r.choices[0].logprobs.content[0]
print(tok.token, round(math.exp(tok.logprob) * 100, 2))

Why it works. A logprob is the natural log of a probability, so it runs from negative infinity up to 0, where 0 is 100 percent. Exponentiating gives you the probability back. The OpenAI cookbook uses exactly this conversion.

The trick most people miss. For a classification, do not average across the whole response. Read the logprob of the first content token, because that is the token carrying the decision. Averaging a 40 token answer buries the one number you care about under 39 boring ones.

Recipe 2: The same thing on Gemini

Claim. Two fields instead of one, and you get a whole candidate average for free.

Receipt.

{
  "contents": [{"parts": [{"text": "Sentiment of 'shipped late but it works'. One word."}]}],
  "generationConfig": {
    "responseLogprobs": true,
    "logprobs": 5
  }
}

Why it works. logprobs is only valid when responseLogprobs is true, which is the error everyone hits first. Set both. The response carries logprobsResult with topCandidates and chosenCandidates per decoding step, each candidate exposing token, tokenId and logProbability, plus a top level avgLogprobs on the candidate. All of it is documented in the generateContent reference.

When to reach for this over OpenAI. When you need to see the runners up. Twenty candidates per position is enough to tell "the model was torn between two labels" apart from "the model had no idea," and those two cases deserve different handling in your code. Five is often not.

Recipe 3: Claude has no logprobs, so force a score you can parse

Claim. On Claude you cannot read confidence, so you make the model write it, and you use prefill so it cannot wander off format.

Receipt.

{
  "model": "claude-sonnet-5",
  "max_tokens": 100,
  "messages": [
    {"role": "user", "content": "Classify sentiment. Reply with JSON: label, confidence 0.00 to 1.00, and a 5 word reason."},
    {"role": "assistant", "content": "{\"label\": \""}
  ]
}

Why it works. The prefill puts the model mid key, so it has to continue the object rather than open with prose. You get a parseable score on the first call with no retry loop. This is the same one move covered in assistant prefill recipes, pointed at a different job.

Read this before you trust the number. A verbalized score is the model's opinion of itself, not a measurement. The honest finding, from On Verbalized Confidence Scores for LLMs (Yang, Tsai and Yamada, revised May 2026), is more nuanced than the usual "it is worthless" take: reliability "strongly depends on how the model is asked," and well calibrated scores are extractable with certain prompt methods. So the prompt wording is not cosmetic here. It is the whole experiment. Ask for a bare 0 to 1 float and pin the scale in the prompt, then check the calibration on your own labelled set before you wire it to anything.

Recipe 4: The n-vote agreement score, which works on any model

Claim. Sample the same prompt k times and let disagreement be your confidence signal. No logprobs required, so it works identically on Claude, on Together, on anything.

Receipt.

from collections import Counter

votes = [classify(text) for _ in range(5)]   # temperature 1.0
label, n = Counter(votes).most_common(1)[0]
confidence = n / len(votes)                   # 5/5 = 1.0, 3/5 = 0.6

Why it works. It measures the thing you actually want. A logprob tells you the model was confident about a token; a 5 out of 5 vote tells you the answer is stable under resampling, which is much closer to "will this be right." This is self consistency used as a meter instead of an accuracy booster.

The cost. Five times the calls. Only run it on the rows that need it, which in practice means gating it behind Recipe 1 or 3 and only escalating the ambiguous ones.

Where confidence scores break

• Calibrated is not the same as correct. A model can be confidently wrong, and on a bad prompt it usually is. A confidence score routes work. It does not verify anything.
• Do not average across a long answer. Mean logprob over 400 tokens is dominated by punctuation, articles and whitespace, all of which the model is extremely sure about. Scope the score to the decision tokens.
• Thresholds do not transfer. The 0.85 cutoff you tuned on one model and one dataset is meaningless on the next one. Retune per task, on labelled data, every time.
• Do not use it as a hallucination detector. It is a fluency signal, not a truth signal. Fluent nonsense scores high. If you need to judge whether an answer is actually good, that is a different tool: score the output with a rubric, the way an LLM as a judge harness does.
• Watch for the free tier trap. top_logprobs and responseLogprobs inflate response size, not input cost. The bill moves less than people expect. The latency moves more.

When NOT to use one at all

If every output gets reviewed by a human anyway, skip it. A confidence score earns its keep only when it changes routing: auto approve above the line, queue for review below it. If nothing branches on the number, you are paying tokens for a decoration.

FAQ

Does Claude have logprobs?
No. As of August 2026 the Anthropic Messages API exposes no logprobs or top_logprobs parameter and returns no token probabilities in the response. Use a prefilled JSON confidence field or an n-vote agreement score instead.

How do you convert a logprob to a probability?
Exponentiate it. probability = math.exp(logprob). A logprob of 0 is 100 percent, -0.7 is about 50 percent, and -2.3 is about 10 percent.

What is the difference between logprobs and logits?
Logits are the raw unnormalized scores the model produces before the softmax. Logprobs are those scores after softmax and after taking a log, so they are actual probabilities expressed on a log scale. APIs return logprobs, not logits.

Can you just ask the model how confident it is?
Yes, and it works better than its reputation suggests, but only with care. Peer reviewed work in 2026 found verbalized confidence can be well calibrated depending heavily on how you ask. Pin the scale, force the format with a prefill, and validate on your own labelled data.

What is a good confidence threshold?
There is no portable answer. Pick it from a labelled sample by choosing the cutoff that gives the precision your workflow needs, and retune it whenever you change model or task.

Does asking for top_logprobs cost extra tokens?
No. Logprobs are metadata about tokens that were generated anyway, so they do not add to your billed token count. They do make the response payload considerably larger, which shows up as latency and bandwidth.

Cost to test: about $0.03. All four recipes, under twenty calls, on the cheapest tier of each provider.