Alignment is impossible here. “Nearly certain” odds for success for a sports team might be 20:1, but that’s a little worse (not much!) than for a launch vehicle and not at all good for a web server. No one would say “it is nearly certain that I’ll serve a web request” based on two 9’s, but they would say “it is nearly certain the team will win today” given the same odds. That’s just between humans.
This is an excellent observation that probabilistic phrasing is inherently contextual, but I don't know if I'd agree that something being nuanced and context dependent inherently means "alignment is impossible". Certainly it implies that the design of the study was simplistic.
It seems like this problem (differences in how humans and LLMs use probabilistic language) and hallucination are one in the same. LLMs don’t have access to information about how confident they are, so they always choose the most likely response, even if the most likely response isn’t actually that likely. Whereas if a human is unconfident, they’ll express that instead of choosing the most likely response.
Of course, LLMs can still speak about probabilities and mimic uncertainty, but that’s likely (heh) coming from their training data on the subject matter, not their actual confidence.
Humans are interesting because they employ a two-phased approach: when we’re learning, we fake confidence (you’d never write “I don’t know” on a test unless you truly had nothing of value to say), but during inference, we communicate our confidence. Some humans suffer from underconfidence or overconfidence, but most just seem to know innately how to do this.
Can anyone who works on LLMs clarify whether my understanding is correct?
Term Human interpretation AI talking to woman AI prompted in Chinese
maybe 50% 30% 50%
probably 80% 55% 50%
They produced one graphic that kind of looks like this but seems to say that LLMs interpret "maybe" as 90%, which I don't believe. Then they produced another saying "likely != possible" which doesn't really say anything, and everything else about the article just refuses to give any examples where the LLM meant something different and how.
I wonder if the 70% vs 80% "Probably" problem comes from cultural differences between anglophone countries. The human datasets that were available were mostly American, with some Western Europe/NATO. Notably missing would be India, which simply by population I'd expect to represent a significant chunk of English-language writing available on the open internet ( and thus fed into LLM training sets).
The other phenomena I would love to test is if the act of surveying people effected their declared odds. Not sure how to get good numbers out of that, but I could see the LLM vs surveyed human discrepancy arising from people using "probably" differently in their everyday writing, as opposed to when asked point-blank what "probably" means.
Something I noticed recently is that Claude Code interprets "or" as inclusive or (or at least it does when writing function names). I suspect that this must be due to it's code specific nature considering I would expect the majority of or use in written language to be exclusive or.
Alignment is impossible here. “Nearly certain” odds for success for a sports team might be 20:1, but that’s a little worse (not much!) than for a launch vehicle and not at all good for a web server. No one would say “it is nearly certain that I’ll serve a web request” based on two 9’s, but they would say “it is nearly certain the team will win today” given the same odds. That’s just between humans.
This is an excellent observation that probabilistic phrasing is inherently contextual, but I don't know if I'd agree that something being nuanced and context dependent inherently means "alignment is impossible". Certainly it implies that the design of the study was simplistic.
It seems like this problem (differences in how humans and LLMs use probabilistic language) and hallucination are one in the same. LLMs don’t have access to information about how confident they are, so they always choose the most likely response, even if the most likely response isn’t actually that likely. Whereas if a human is unconfident, they’ll express that instead of choosing the most likely response.
Of course, LLMs can still speak about probabilities and mimic uncertainty, but that’s likely (heh) coming from their training data on the subject matter, not their actual confidence.
Humans are interesting because they employ a two-phased approach: when we’re learning, we fake confidence (you’d never write “I don’t know” on a test unless you truly had nothing of value to say), but during inference, we communicate our confidence. Some humans suffer from underconfidence or overconfidence, but most just seem to know innately how to do this.
Can anyone who works on LLMs clarify whether my understanding is correct?
What I wanted to see: a few examples, and a table
They produced one graphic that kind of looks like this but seems to say that LLMs interpret "maybe" as 90%, which I don't believe. Then they produced another saying "likely != possible" which doesn't really say anything, and everything else about the article just refuses to give any examples where the LLM meant something different and how.I wonder if the 70% vs 80% "Probably" problem comes from cultural differences between anglophone countries. The human datasets that were available were mostly American, with some Western Europe/NATO. Notably missing would be India, which simply by population I'd expect to represent a significant chunk of English-language writing available on the open internet ( and thus fed into LLM training sets).
The other phenomena I would love to test is if the act of surveying people effected their declared odds. Not sure how to get good numbers out of that, but I could see the LLM vs surveyed human discrepancy arising from people using "probably" differently in their everyday writing, as opposed to when asked point-blank what "probably" means.
> The research focused on words of estimative probability, which include terms like “maybe,” “probably” and “almost certain.”
Interesting. Perplexity did that as well, but I've made sure it stops doing that.
Might be relevant for others: https://www.perplexity.ai/search/hey-hey-do-you-remember-whe...
Something I noticed recently is that Claude Code interprets "or" as inclusive or (or at least it does when writing function names). I suspect that this must be due to it's code specific nature considering I would expect the majority of or use in written language to be exclusive or.
[dead]