Is it also post-trained to know when it's wrong about when it's wrong?
> "Every response comes with a confidence score between 0 and 1"
How confident is it in its confidence?
Please, I'm sure that what you're doing is very neat and useful, but use other language to describe it. I beg you. You can't know when you're wrong. You can only know when you're unsure or inconsistent. You can be absolutely certain and still wrong and uncertain and still correct.
A few years ago, before RLVR became ubiquitous, people were using generate -> classify loops to get models to improve math reasoning. Turns out classification is sometimes easier to do and more accurate than generating coherent traces. So you'd get a model to "think step by step", then run each step (.split('/n/n')) through a classifier, and tell the model "this step might be wrong, re-check". If you're interested I could probably dig up some papers on this.
Now there might be even better ways of getting feedback, I saw some interesting trials using "J-spaces" (from Anthropic's latest research in this area). Turns out you can kinda see when the model uses deep seeded concepts in early layers, or it's "riffing on an idea" with shallow signals in those same layers. And youc an get some signal out of it, and have it re-do a step.
It's also worth mentioning that LMs are wrong in different ways from humans. It's not impossible that we'll eventually find ways of better understanding this (through mechanistic interpretation or something else) and have easy "classifiers" for "this comes from the training data" vs. "this comes from early context" (i.e. supplied materials from prompts/discovery) vs. "this comes from later tokens, likely the model doom-looped itself".
None of what you've said has anything to do with correctness, only certainty. And it's funny to me in context if you're certain that the opposite is true.
> "It's also worth mentioning that LMs are wrong in different ways from humans."
This to me only shows a poor and incomplete knowledge of humans and of wrongness and its causes. If you look at all of the humans and all of the times, you find exhibitions of all the same ways of being wrong. At best you can say that some (<all) LMs are wrong in different ways than some (<all) humans at some (<all) times, but that's as good as saying that different humans are wrong in different ways than different other humans at different times.
This is an old problem in philosophy, one that the pre-LLM systems were interested in, which got completely bypassed by the present approach. What is knowledge? It's not a simple question even for humans, we don't have a magic truth detector in the brain.
This is why LLMs are more successful in programming than other disciplines: it's easier to assess correctness of results, through proofs, typechecking, and testing. Similarly for maths and applied maths. Much harder in the humanities.
> successful in programming than other disciplines
It seems successful specifically in "computer science", for "programming" which encompasses a much wider field, "successful" is still up for debate. Also, we haven't even figured out yet what "success" even means in a software development process, for some it's that the program does everything the users want, in the way they expect, for other's, it's about correctness, reliability, performance, composability, maintainability and/or a whole host of other "measures".
IMHO this also depends on the type of assertion.. For example, one system defines three broad classes of assertion : pure opinion (I like green); factual references (the current temperature here is 20c) ; and the rest in a catch-all about "reasoned judgements"
Complex assertions and their compounded forms can be filtered by these classes fairly easily. I rarely see this referenced in general comments.
opinion is ephemeral, cannot be independently confirmed or denied; purely single viewer information sounds like not-a-fact to me
Opinion is a ghost in the machine—fleeting, unanchored, and vanishing upon inspection. It leaves no footprint for another mind to verify, no independent trail of evidence to confirm or deny its existence. To rely on such solitary perception is to drift in a void where facts do not exist; it is not merely uncertainty, but an absence of reality itself—a whisper that the universe refuses to echo back.
--AI Carl Sagan
"Ephemeral" isn't relevant - a statement of fact can be true at one time and false at another.
Confirmation also is somewhat orthogonal: there can be true facts that we can't confirm.
Conventionally, "I like green" is considered a factual claim about the speaker's preferences. Even if you refuse to acknowledge it as a fact because of the restrictions you've given, that doesn't make it an opinion. It's subjective, but it's not an opinion.
Would you say that if someone is experiencing a headache, and they say "I have a headache", that that's also merely an opinion?
Exactly. A fact is something that can be objectively confirmed or rejected. Exactly what a specific person likes is a fact, because it can be verified/falsified by simply asking them. An opinion, by contrast is 'Elephants are large.' The reason that's an opinion is because while it seems obviously true, it's subjective. Similarly, grass is purple with pink polka-dots is a fact. It's obviously false, but of course right/wrong has nothing to do with fact/opinion.
_All_ statements are only valid inside the window within which their understandable context stays constant, facts or otherwise.
> "Verifying an opinion X at time T does not entail X at time T+1."
And verifying that my cat weighs 9.5 lbs now does not entail that he will weigh 9.5 lbs tomorrow, but "my cat weighs 9.5 lbs" is not an opinion; it's a statement of fact from measurement/observation and implies "extrapolated from the last specific check done in the usual way" unless explicitly stated otherwise because that's how language works in polite society. "I like green" is the same.
"My cat weighs 9.5 lbs" is only true within a context of a particular moment and location and measurement apparatus and understanding of cat and ownership and 9.5 and lbs.
"1+1=2" is only true within a context of what "1" and "2" and "+" and "=" mean.
> "Opinions can only be seen as facts when modelled as time series data."
Opinions can never be seen as facts, timestamps or otherwise. Only facts about opinions can be seen as facts, not the opinions themselves.
"Green is good" is never a fact regardless of whether you timestamp it. "Green is good right now" is not a fact. "Green was good yesterday" is not a fact. "BugsJustFindMe thinks green is good" is a (false) fact, "BugsJustFindMe said green is good" is a (true) fact, and neither one is an opinion at any time.
> "Verifying an opinion X at time T does not entail X at time T+1."
A constraint binding timeliness specifically to opinions is misplaced and also not extremely relevant in the context of whether something is a statement of opinion or fact.
An opinion is a particular subject's belief [1]. That's literally the definition. "X believes Y at time T", is a fully specified fact and completely follows from the definitions of all terms involved.
None of that has anything to do with whether opinions can ever be considered facts. X believes Y at T is not the same as Y at T, and a statement about an opinion is not the opinion itself. Anyway, we're going in circles now and that's uninteresting to me.
The catch-all is a problem. We've been studying fallacies in arguments since before the time of Plato, and do not have a complete answer that always works, to prevent the logical failure.
Complex assertions and compounding arguments, can still result in circular reasoning - the petitio principii, or worse.
Most LLMs are guided by strong statements like it must do this or that. Which assumes that it won't apply the fallacy of accident, even though people regularly do. (Don't agree? Argue a definition with a BA for a day and come back to me.)
Interesting question and I don’t share your pessimism. Feels like it’s a matter of some plain old Bayesian stats, almost a homework assignment given all experimental data is available and known.
1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This would let you turn your raw probe score into a threshold with a guaranteed bound on the rate of wrongly-kept on-device answers. Source: https://en.wikipedia.org/wiki/Conformal_prediction
2. The best indicators of confidence in ML come from multiple independent methods. What was the result if you combine the token entropy and verbal confidence reporting methods? Does this improve the result?
3. I noticed you didn't mention the assessment method of rerunning the model and judging whether outputs are consistent. How does that method compare in terms of AUROC?
> the hidden state for different layers carry meaningful self-awareness signal for various situations.
Is it plausible to wonder if some developer judgement feels, like maybe "the code I just wrote is clean/crufty", or "things came together smoothly/janky", might have extractable signals in some models?
If so, might one create a shopping list of desired signals to check for in a model, as with activation steering concepts, where one checks whether and how hard each concept can usefully be nudged?
> So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations.
Neat! Just to make sure I understand - you trained your probe layer to take this hidden state and predict p(wrong)?
Curious to learn more. Any more info on your approach (esp the mechanistic study)?
Correct, the study is verbose, we will compile into a neat shareable report and publish once we solve the pending caveats. Interesting username btw haha.
Was actually pulling on a similar thread as I saw announcement so I integrated it just for fun. Have been only running this on my Framework Desktop but should be runnable elsewhere
https://github.com/olafura/gemma-4-mic-transcribe
I don't really get why you need handoff if your score is accurate. If it is, and it is low for a given response, just let the harness re-run the prompt with a different seed until the score is high enough. If this approach doesn't work, your score is most likely garbage.
> "post-trained to know when it's wrong"
Is it also post-trained to know when it's wrong about when it's wrong?
> "Every response comes with a confidence score between 0 and 1"
How confident is it in its confidence?
Please, I'm sure that what you're doing is very neat and useful, but use other language to describe it. I beg you. You can't know when you're wrong. You can only know when you're unsure or inconsistent. You can be absolutely certain and still wrong and uncertain and still correct.
A few years ago, before RLVR became ubiquitous, people were using generate -> classify loops to get models to improve math reasoning. Turns out classification is sometimes easier to do and more accurate than generating coherent traces. So you'd get a model to "think step by step", then run each step (.split('/n/n')) through a classifier, and tell the model "this step might be wrong, re-check". If you're interested I could probably dig up some papers on this.
Now there might be even better ways of getting feedback, I saw some interesting trials using "J-spaces" (from Anthropic's latest research in this area). Turns out you can kinda see when the model uses deep seeded concepts in early layers, or it's "riffing on an idea" with shallow signals in those same layers. And youc an get some signal out of it, and have it re-do a step.
It's also worth mentioning that LMs are wrong in different ways from humans. It's not impossible that we'll eventually find ways of better understanding this (through mechanistic interpretation or something else) and have easy "classifiers" for "this comes from the training data" vs. "this comes from early context" (i.e. supplied materials from prompts/discovery) vs. "this comes from later tokens, likely the model doom-looped itself".
None of what you've said has anything to do with correctness, only certainty. And it's funny to me in context if you're certain that the opposite is true.
> "It's also worth mentioning that LMs are wrong in different ways from humans."
This to me only shows a poor and incomplete knowledge of humans and of wrongness and its causes. If you look at all of the humans and all of the times, you find exhibitions of all the same ways of being wrong. At best you can say that some (<all) LMs are wrong in different ways than some (<all) humans at some (<all) times, but that's as good as saying that different humans are wrong in different ways than different other humans at different times.
> And youc an get some signal out of it, and have it re-do a step.
Or the opposite https://arxiv.org/abs/2606.06574v1
I always thought it was a pretty noisy signal, was there any empirical work showing what the SNR was in these step by step classification loops?
This is an old problem in philosophy, one that the pre-LLM systems were interested in, which got completely bypassed by the present approach. What is knowledge? It's not a simple question even for humans, we don't have a magic truth detector in the brain.
https://plato.stanford.edu/entries/knowledge-analysis/
This is why LLMs are more successful in programming than other disciplines: it's easier to assess correctness of results, through proofs, typechecking, and testing. Similarly for maths and applied maths. Much harder in the humanities.
> successful in programming than other disciplines
It seems successful specifically in "computer science", for "programming" which encompasses a much wider field, "successful" is still up for debate. Also, we haven't even figured out yet what "success" even means in a software development process, for some it's that the program does everything the users want, in the way they expect, for other's, it's about correctness, reliability, performance, composability, maintainability and/or a whole host of other "measures".
IMHO this also depends on the type of assertion.. For example, one system defines three broad classes of assertion : pure opinion (I like green); factual references (the current temperature here is 20c) ; and the rest in a catch-all about "reasoned judgements"
Complex assertions and their compounded forms can be filtered by these classes fairly easily. I rarely see this referenced in general comments.
> pure opinion (I like green)
"I like green" is a statement of fact about what you like. "Green is good" would be an opinion. It's funny to me how often people mix the two.
> forms can be filtered by these classes fairly easily
Obviously not so easily. :)
opinion is ephemeral, cannot be independently confirmed or denied; purely single viewer information sounds like not-a-fact to me
Opinion is a ghost in the machine—fleeting, unanchored, and vanishing upon inspection. It leaves no footprint for another mind to verify, no independent trail of evidence to confirm or deny its existence. To rely on such solitary perception is to drift in a void where facts do not exist; it is not merely uncertainty, but an absence of reality itself—a whisper that the universe refuses to echo back. --AI Carl Sagan
"Ephemeral" isn't relevant - a statement of fact can be true at one time and false at another.
Confirmation also is somewhat orthogonal: there can be true facts that we can't confirm.
Conventionally, "I like green" is considered a factual claim about the speaker's preferences. Even if you refuse to acknowledge it as a fact because of the restrictions you've given, that doesn't make it an opinion. It's subjective, but it's not an opinion.
Would you say that if someone is experiencing a headache, and they say "I have a headache", that that's also merely an opinion?
It seems to me trivial to verify: just ask.
Exactly. A fact is something that can be objectively confirmed or rejected. Exactly what a specific person likes is a fact, because it can be verified/falsified by simply asking them. An opinion, by contrast is 'Elephants are large.' The reason that's an opinion is because while it seems obviously true, it's subjective. Similarly, grass is purple with pink polka-dots is a fact. It's obviously false, but of course right/wrong has nothing to do with fact/opinion.
Opinions can only be seen as facts when modelled as time series data. Verifying an opinion X at time T does not entail X at time T+1.
_All_ statements are only valid inside the window within which their understandable context stays constant, facts or otherwise.
> "Verifying an opinion X at time T does not entail X at time T+1."
And verifying that my cat weighs 9.5 lbs now does not entail that he will weigh 9.5 lbs tomorrow, but "my cat weighs 9.5 lbs" is not an opinion; it's a statement of fact from measurement/observation and implies "extrapolated from the last specific check done in the usual way" unless explicitly stated otherwise because that's how language works in polite society. "I like green" is the same.
"My cat weighs 9.5 lbs" is only true within a context of a particular moment and location and measurement apparatus and understanding of cat and ownership and 9.5 and lbs.
"1+1=2" is only true within a context of what "1" and "2" and "+" and "=" mean.
And so on.
This seems like a very long winded way to agree with my statement that opinions can only be considered facts when modelled as time series data.
> "This seems like a very long winded way"
I think you could do without the invective.
My view is specifically thus:
> "Opinions can only be seen as facts when modelled as time series data."
Opinions can never be seen as facts, timestamps or otherwise. Only facts about opinions can be seen as facts, not the opinions themselves.
"Green is good" is never a fact regardless of whether you timestamp it. "Green is good right now" is not a fact. "Green was good yesterday" is not a fact. "BugsJustFindMe thinks green is good" is a (false) fact, "BugsJustFindMe said green is good" is a (true) fact, and neither one is an opinion at any time.
> "Verifying an opinion X at time T does not entail X at time T+1."
A constraint binding timeliness specifically to opinions is misplaced and also not extremely relevant in the context of whether something is a statement of opinion or fact.
An opinion is a particular subject's belief [1]. That's literally the definition. "X believes Y at time T", is a fully specified fact and completely follows from the definitions of all terms involved.
[1] a view, judgment, or appraisal formed in the mind about a particular matter, https://www.merriam-webster.com/dictionary/opinion
None of that has anything to do with whether opinions can ever be considered facts. X believes Y at T is not the same as Y at T, and a statement about an opinion is not the opinion itself. Anyway, we're going in circles now and that's uninteresting to me.
The catch-all is a problem. We've been studying fallacies in arguments since before the time of Plato, and do not have a complete answer that always works, to prevent the logical failure.
Complex assertions and compounding arguments, can still result in circular reasoning - the petitio principii, or worse.
Most LLMs are guided by strong statements like it must do this or that. Which assumes that it won't apply the fallacy of accident, even though people regularly do. (Don't agree? Argue a definition with a BA for a day and come back to me.)
Interesting question and I don’t share your pessimism. Feels like it’s a matter of some plain old Bayesian stats, almost a homework assignment given all experimental data is available and known.
Cool idea, some feedback:
1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This would let you turn your raw probe score into a threshold with a guaranteed bound on the rate of wrongly-kept on-device answers. Source: https://en.wikipedia.org/wiki/Conformal_prediction
2. The best indicators of confidence in ML come from multiple independent methods. What was the result if you combine the token entropy and verbal confidence reporting methods? Does this improve the result?
3. I noticed you didn't mention the assessment method of rerunning the model and judging whether outputs are consistent. How does that method compare in terms of AUROC?
1. Thanks for this!
2. We combine both on Cactus; rolling entropy method helps with per-token signal for early exit, while the probe runs on complete output.
3. Users hate this in production, drains battery on edge devices, but ultimately primarily works ok-ish on 7B+ models, we will share the results too.
> the hidden state for different layers carry meaningful self-awareness signal for various situations.
Is it plausible to wonder if some developer judgement feels, like maybe "the code I just wrote is clean/crufty", or "things came together smoothly/janky", might have extractable signals in some models?
If so, might one create a shopping list of desired signals to check for in a model, as with activation steering concepts, where one checks whether and how hard each concept can usefully be nudged?
YOu are thinking along the right direction, we are going deeper into the signals.
Is this in any way similar to Goodfire's work? https://www.goodfire.ai/research/rlfr#
Thats an interesting outlook, loosely similar.
> So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations.
Neat! Just to make sure I understand - you trained your probe layer to take this hidden state and predict p(wrong)?
Curious to learn more. Any more info on your approach (esp the mechanistic study)?
Correct, the study is verbose, we will compile into a neat shareable report and publish once we solve the pending caveats. Interesting username btw haha.
Nice, looking forward to the report.
And thanks, huge pasta fan :)
anytime!
Was actually pulling on a similar thread as I saw announcement so I integrated it just for fun. Have been only running this on my Framework Desktop but should be runnable elsewhere https://github.com/olafura/gemma-4-mic-transcribe
Have you benched this for coding tasks, with a fallback to a larger local model, for example Qwen-3.6-27B?
Or using it for sub-tasks, where a framework with a larger primary model dispatches simpler jobs ("summarize ...", etc.) to it?
We are currently working on larger models atm! your suggestion is cool btw
I don't really get why you need handoff if your score is accurate. If it is, and it is low for a given response, just let the harness re-run the prompt with a different seed until the score is high enough. If this approach doesn't work, your score is most likely garbage.
Smaller models in production can produce quite interesting outputs.
Interesting how that once again matches human intuition: It's easier to know what you don't know, than to know what is true.
haha, interesting analogy
Love what you're doing at cactus, keep it up!!
Does the model quality become degraded in other ways?
no because we freeze the main weights and only train our added weights.
It's impossible to add weights which degrade the quality?
Not if the new weights don't influence the old outputs.
Aha! I thought this was feeding the confidence info back to Gemma for some reason.