The idea of periodically stopping to write blocks of recent context into a fast-weight state is interesting, but I think it liked it better when E2E-TTT[1] did it. It's a more flexible and elegant continuous learning approach.
Essentially it goes "You know how your model can remember its training data? Well, what if you treated its recent context like more training data and updated (some of) the weights using (mostly) the same process used to train it?"
The end result is very good at remembering things but also really good at adapting to new unseen distributions.
Yah I think E2E-TTT is a lot more like what people in this comments section are picturing. I can't tell that this method updates model weights at all during the "sleep" period, only the usual SSM state updated by any Mamba model after each token. They just optimized the model to use that SSM state _more_ when an eviction is about to happen.
Yes, since the weights being updated are a small subset of the overall total it's manageable. Just like how each separate conversation currently requires you to store a separate KV cache, you'd need to store the fast weights separately. Both KV cache and fast weight content stores have to be conversation specific, so just setting a bit of extra RAM aside for "memory" isn't really a new ask, just a different format for an old problem.
I wonder if we can get children to make something their life’s dream if we make the cool books about it when they are growing up? I wonder how flexible the human mind can be in convincing itself that it is fulfilling its dream?
This topic recently came up at the FLANN workshop [1], and seems to periodically be rediscovered [2,3,4] in different contexts. While some have speculated about the biological role it plays (e.g., Pearlmutter & Houghton [5]), we still lack a conclusive theory of sleep, but the convergent evolution of this specific phenomenon across the animal kingdom and the fact that deprivation is inevitably fatal seems like an important clue.
Scaling test-time compute has emerged as a key ingredient for enabling large language models (LLMs) to solve difficult problems, but comes with high latency and inference cost. We introduce sleep-time compute, which allows models to "think" offline about contexts before queries are presented: by anticipating what queries users might ask and pre-computing useful quantities, we can significantly reduce the compute requirements at test-time. To demonstrate the efficacy of our method, we create modified versions of two reasoning tasks - Stateful GSM-Symbolic and Stateful AIME. We find that sleep-time compute can reduce the amount of test-time compute needed to achieve the same accuracy by ~ 5x on Stateful GSM-Symbolic and Stateful AIME and that by scaling sleep-time compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME. Furthermore, we introduce Multi-Query GSM-Symbolic, which extends GSM-Symbolic by including multiple related queries per context. By amortizing sleep-time compute across related queries about the same context using Multi-Query GSM-Symbolic, we can decrease the average cost per query by 2.5x. We then conduct additional analysis to understand when sleep-time compute is most effective, finding the predictability of the user query to be well correlated with the efficacy of sleep-time compute. Finally, we conduct a case-study of applying sleep-time compute to a realistic agentic SWE task.
To reach a more brain-like behavior LLMs need to integrate your inputs into their model dynamically, essentially retraining real-time based on the most salient input. Human brains do this selectively all the time and it's part of our plasticity.
Biologically humans do similar compression, so introducing a similar concept to an LLM also feels reasonable. Hardware isn't fast/cheap enough to do this on an ongoing basis, similar to how it's too expensive for our brains to do this while we're moving through the world.
All we have now most of the time in LLMs is "working memory" we're missing a lot of the functionality that allows for episodic memory and selective plasticity.
The more you read about how human brains work, the more you realize that we may have figured out a piece with LLMs, but it's certainly nothing approaching AGI. People insisting so are blowing smoke for investor hype or don't understand a big piece of the concepts involved.
>To reach a more brain-like behavior LLMs need to integrate your inputs into their model dynamically, essentially retraining real-time based on the most salient input.
That's already possible with LLMs. The challenge is that 1. it would allow permanently jail-breaking models and 2. there'd be no way for them to efficiently transfer what they'd learned to a new model generation.
That's an idea I had a few months ago: after going through a compaction once the KV cache is nearing capacity, accumulate this knowledge into a dataset to fine-tune a LoRA during offline hours.
This would create a three-layer memory system:
- Stable long-term memory (initial base weights)
- Mid-term memory built from the compactions and replay buffers
- Short-term memory (KV cache)
Sleeping would just be a fancy term for consolidating and transferring information from one memory layer to another during offline hours. Maybe that's also what the brain does while sleeping.
Wouldn't that just accelerate collapse? How much do you trust the outputs of the llm to provide trustworthy and valuable new information? I mean I understand distillation works. But that's much more structured and thoughtful than my sessions at least.
I was thinking of curated replay buffers, which would act like "dreams". To prevent collapse, the offline dataset would mix the new mid-term data with a baseline of anchor data (the original training distribution) so the model doesn't drift.
Also, we wouldn't train on the whole session. A separate critic module, like a reward model, would filter the KV cache to extract the high-value information, like a garbage collector before the LoRA.
That's just an idea though. Right now most research focuses on changing the architecture itself (TITAN, HOPE...) instead.
It's a network of computers with GPUs, so there's no reason it can't sleep at the same time it's awake. Just a continuous "sleeping" process going on in the background, incrementally updating the model. No need for the "thinking" process to be "unconscious" while the "sleeping" process runs. Anthropomorphism confuses everything. There's no such thing as "offline hours" because the Earth is a sphere and the United States is not the center of the universe.
The abstract and method sections only mention updating the SSM state during "sleep" (ie the same vectors that change after each token in stock Mamba) not any of the actual weight matrices. AFAICT this is just another attention compaction paper, with misleading tile? It is not very clearly written
No, they're actually training weights based on context before compaction. Context is context, this is splitting the model into persistent weights and malleable ones which are periodically updated.
The entire industry is so desperate to anthropomorphize. What the paper describes is an offline recurrent consolidation phase: the model runs multiple forward passes over recently accumulated context, updates persistent fast weights in SSM blocks, then clears the KV cache before continuing. It has absolutely nothing to do with sleeping, but I believe the authors had a goal in mind when creating this title, and it was for journalists to pick it up and run with it, further inflating the AI-is-just-like-us hype bubble.
An intelligent reply from an obviously intelligent guy!
A more appropriate title would have been something like "Offline Recurrent Memory Consolidation for Long-Context Language Models". This is supposed to be a research paper, not a story book. The title should give context to other researchers, and not be clearly engineered for clicks. If you don't think so, that's your prerogative, but you're objectively wrong.
I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities.
Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
They provide an explanation for using the term "sleep":
> In animals, the transfer from short-term memory to long-term memory is thought to be
supported by hippocampal replay [33], especially during sleep [41]; in this phase, short-term hippocampal memories are reactivated and consolidated into cortical synaptic weights. Sleep makes animals unable to respond to external stimuli, suggesting that it must provide enough cognitive benefit to justify this cost [41]. Inspired by these biological processes, we propose a method for transferring context-window memory into persistent weights. When the model’s context window becomes full during inference, the model enters a “sleep” in which it performs multiple forward passes over the accumulated context and recursively updates its fast weights via a learned local rule. As in animal sleep, the model receives no external input tokens during this phase. After consolidation, the context window is cleared, and the model resumes operation with updated fast weights. During training, the model is optimized end-to-end by backpropagating through the entire process to maximize task performance after sleep.
The function of sleep in animals is largely obscure.
One thing we do know for certain is that it is necessary, it is needed in "dumb" animals as well as in you and I. If an animal can't sleep it will eventually die.
I don't think that applies to the activity described in the OP. Does their LLM "die" if it can't perform the function described?
> Does their LLM "die" if it can't perform the function described?
If you don't periodically clean the context, an LLM effectively goes insane in terms of outputs.
If the LLM were fully controlling a physical system (like a robot body) that contained it the resulting insanity of an ever-growing, never cleaned context would likely result in some sort of death-like event.
It's still weak, though. An LLM without constant human input is likely more similar to a bicycle that starts to lose its gyroscopic balance as it moves more slowly, a human can however keep a stationary bicycle upright (while riding it).
There is a lot that is known about sleep. We don't know everything and there are large gaps in our knowledge, but there is also a lot that we do know. And this research explicitly tried to emulate the things we know that sleep does do. Calling it "sleep" is warranted, imho.
Probably because it evolved very early (like before bilateral symmetry, multi layer body cavity, or kidneys early... maybe even before multicellular animals early) and so has been incorporated as an essential pillar into multiple processes layered on top of that fundamental architecture.
You've merely stated observations about the context and the process that led to it. That doesn't in any way answer the question of what it's actually doing that's so essential.
My point is that, in a higher organism, it may be essential to how a lot of their processes function, in that it was infrastructure that already existed at the time those processes were developed, so they were in turn developed to depend upon it.
So "the reason it originally exists" and "what breaks if you take it away" aren't necessarily the same thing.
As with, say, digestion, or an major organ like the liver, it's reasonable to think that it does simple things in simple animals, and more complex things in more complex ones.
Take out an animal's liver, it's not one process that stops working, it's dozens. There's one or two that will kill it quicker, so those are the ones it dies of, but artificial livers are hard to build as they implement so many vital processes.
I don't dispute any of that but it's stating the obvious, it's nothing more than topical conjecture (even if it's almost certainly correct), and (most importantly) it does nothing to answer the question. What essential functions are being performed?
Take your liver example. We can largely answer that same question. I can't off the top of my head but the answer is fairly well established even if incomplete to varying degrees depending on the species.
There is widespread consensus on why a liver is needed for survival whereas there is not for sleep. That's particularly interesting when you consider that sleep is more common across the tree of life than dedicated livers are (at least AFAIK).
If he articulated a particular essential process and why it depends on sleep in an incidental manner that might make for a reasonable hypothesis. However it would not refute the earlier (cited) claim that there is no consensus.
As presented without any concrete information about the processes involved it doesn't even qualify as a hypothesis, merely empty handwaving. In context it's even worse, being an entirely baseless contradiction of a claim pulled from a prominent paper.
I never agreed with him but all of that is implied. It being informed speculation included.
Also there being no consensus means most scientists who touch on the topic are FFA speculating except the person stating there is no consensus in the overview. It's not "settled science" but rather the opposite.
The question was "why is it needed". In context the meaning is clearly to ask what it's doing that's essential and (it follows) why those things are essential.
The subsequent response did not (as you suggest) articulate some subset of nonessential things done during sleep. Rather it rattled off plausible (and widely understood) aspects of the process that could have led to the current situation. Even if it had listed concrete activities that would still not have made for a meaningful answer.
The first clue that something is wrong should be that the linked article is recent and prominent. Thus short a brand new groundbreaking development we can be reasonably certain that a random commenter on the internet will not be sensibly rebutting the claims (and certainly not in the span of ~2 sentences).
No that was not the question. Why precisely sleep is essential is a complete non-sequitur to the original question, which was "does something occur during sleep which resembles what is described in TFA such that it can justifiably be called sleep."
As a general rule of thumb, if you find someone's responses incoherent, it's good practice to check what is actually being discussed.
I've always assumed its spontaneous specialization of species: leaving the safety of their nest at those times of day when they are the fittest to occupy a niche.
Once energy conserving "sleep time" exists, the genome can postpone or schedule for activity during these times of day, if it turns out to be more effective somehow.
I think sleep serves multiple functions. For example, anyone who works out in any-what systematic way knows that sleep is essential for muscle grow. You can't skip on sleep if you want to get fitter. And this probably has very little to do with the more sophisticated functionality of the brain, rather it allows for some process in muscle tissue to happen.
So, whether the LLM "dies" in any sense may or may not be important for what "sleep" is defined to be in this article. It's quite possible that sleep also affects endocrine system in animals or hormones etc... and that's what's causing death, not necessarily anything to do with how brain functions.
> If an animal can't sleep it will eventually die.
Very few animals fail to eventually die even with as much sleep as they want.
But before death, there is a loss of cognitive function from sleep deprivation, and we observe this too with AI whose context windows get too full.
While we don't know very much about sleep, my understanding is that we do have a long list of things that we do during it, we just don't really understand if sleep is necessary for each of them or simply a convenient opportunity for it.
There's lots of things biology does in response to easy-to-detect proxy signals instead of the real thing they care about: Our sensation of needing to breathe more is based on too much carbonic acid in our blood, not lack of oxygen, which is why in general nobody is allowed in an elevator with a liquid nitrogen dewar; Our natural distaste for incest is based on who we grew up with, not our actual DNA; Get too cold and some people suddenly feel warm and want to (and some do) take all their clothes off even though that would just make them hypothermic even faster.
Being asleep may trigger the things we need to get done, but that doesn't mean sleep is *fundamentally* necessary for the things we need to get done. It could be just that it happens to be the way our biochemistry is wired, and we may find some other way to trigger those things.
The quotation given by djeastm would by my guess for what a dream is, and why we have them. But we don't spend all our time asleep, dreaming. And I'd be the first to say that my guess isn't worth much, as I'm not a brain scientist.
> Being asleep may trigger the things we need to get done, but that doesn't mean sleep is fundamentally necessary for the things we need to get done. It could be just that it happens to be the way our biochemistry is wired, and we may find some other way to trigger those things.
We now have evidence for REM sleep in spiders(1). Our last common ancestor with spiders predates the development of nervous systems. This strongly suggests that sleep (and specifically REM sleep) serves some function important enough that it has independently evolved in both protostomes and dueterostomes. (And probably multiple times within the protostomes, being present in both cephalopods and jumping spiders.)
There may be some commonality in the origin of the ion channels, but I'll lay money that the requirements for sleep are more of a result of general information processing requirements.
> The function of sleep in animals is largely obscure.
This just obscures the conversation IMO. We know a great deal of functions that sleeping performs. We don't know everything. For some reason this prevents us for using this word in a computing context? What do you think about sleep(1) in Unix?
Also..
> Does their LLM "die" if it can't perform the function described?
what "death" means in the context of a computer program?
> The function of sleep in animals is largely obscure.
Also, there's different kinds/stages of sleep, which probably perform different functions.
For instance, REM may do something like the GP describes, consolidating memories and processing learning. Deep sleep may do something else (I vaguely recall some stage of sleep is used by neurons to clear certain waste products).
I don't think it's necessarily correct to think of sleep in terms of "it is necessary for animals or they will die". It might be more useful to think of it as "it was so useful that animals who slept outcompeted all the animals who didn't".
Meaning: it might just provide a big advantage.
I don't want to overextend and assume that any advantage extends to LLMs. That rest-and-recuperate advantage might also extend to LLM-based AIs. Or maybe not, and the rest-and-recuperate is mainly useful for biology-based organisms. But there is some logic to it.
> The function of sleep in animals is largely obscure.
In my understanding, it's well-understood that sleep is used to consolidate and store long-term memories (amongst other functions, like cell and muscle repair). They've found this memory-consolidation-during-sleep even in relatively simple animals like bees.
Sleep-like states exist in animals with nervous systems with a complexity above that found in flatworms, even snails sleep. Sleep therefore appears to be an essential characteristic of more complex biological nervous systems, i.e. biological computers, should you care to stretch the analogy. The more complex the nervous system, the greater the requirement for sleep.
What is described in the OP is therefore not a specific characteristic of sleep. It may however be a "useful" rhetorical device.
I do however object to the extensive use of such rhetorical tricks in the conversations that surround LLMs. For example, why does a consumer-grade LLM display "thinking" while it is actually sending data from my computer to some datacentre, processing it, and sending the result back? Equally, why does it output human-emotive phrases such as "sorry" when such computation is revealed to be incorrect?
Such rhetorical tricks, and more, likely underlie to a large degree the popularity of LLMs, despite their actual performance being clearly below what the rhetoric implies.
> I don't think it's necessarily correct to think of sleep in terms of "it is necessary for animals or they will die". It might be more useful to think of it as "it was so useful that animals who slept outcompeted all the animals who didn't".
You're talking about different things: biological necessity and evolutionary benefit.
You can find out about the former by preventing an animal from sleeping (but otherwise provide all other needed things), and seeing if it will eventually die.
> You can find out about the former by preventing an animal from sleeping (but otherwise provide all other needed things), and seeing if it will eventually die.
That is actually almost impossible to do. The rat study was as close as we’ve ever come, and it’s still debated whether the rats died due to lack of sleep or some other mechanism, since the autopsy couldn’t confirm a cause of death. (It could have been due to the way the experiment ran, for example, not the lack of sleep.)
If I remember correctly, fatal insomnia shares most symptoms with other prion diseases (in which there might be no lack of sleep involved), so it's probably the brain damage that causes death, not insomnia itself.
Is a volcano described as dormant (dormire, literally sleep) also inaccurate and deeply problematic? BTW, it's not anthropomorphized as sleep has existed long before humans.
"Sleep" is just used in their context to describe a non-interactive mode and they didn't lean heavily into zoomorphic - I think you mean - parallels.
You're grinding an axe on a single term. What is your broader hangup with them using the term "sleep"?
> Does their LLM "die" if it can't perform the function described?
We're reaching an age where LMGTFY should now be Let Me LLM That For You. Have you tried asking an LLM this question about the article? I believe it answers it very well.
> If an animal can't sleep it will eventually die.
That turns out to be un-settled science. No human has ever died from lack of sleep.
People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep.
In a series of controlled experiments, rats and fruit flies did die from lack of sleep. But no one has yet proven that it holds true for vertebrates except for rats.
In other words, it could be true that “among vertebrates, only rats die of sleep deprivation.”
So “if an animal can’t sleep, it will eventually die” is actually quite hard to prove, and depending on how you look at it, somewhat easy to disprove by the fact that rats and fruit flies were so difficult to kill from sleep depravation alone.
Personally I’m skeptical of the rat study too. Claude amends this:
> What they did not establish: the mechanism. On autopsy, “no anatomical cause of death was identified.” The rats showed weight loss despite eating more, body temperature problems, and skin lesions, but nothing that pointed to a clean cause. So no, they could not say a rat “died from sleep deprivation alone” in the sense of identifying what sleep loss did to the body to kill it. They showed a strong association under tight controls, not a proven causal pathway.
As far as I understand it, there is a disease that destroys your brain's ability to produce sleep. Once you have it, you suffer total, progressive insomnia and die within roughly 6–18 months. Scientists debate whether it's the underlying brain damage or the sleeplessness itself that causes death, but the two are inseparable in practice, and sleep deprivation is considered the leading candidate.
Separately, the longest anyone has stayed awake under controlled conditions was 11 days, which produced severe cognitive impairment, paranoia, and hallucinations; suggesting the body deteriorates rapidly without sleep.
It's probably not wise to state your original claim as established fact.
Fatal Familial Insomnia is an incredibly rare prion disease that causes widespread neurological destruction. It's not remotely a normal brain that has chosen not to sleep. It's such a highly non-trivial deviation of the brain that we've only identified a few dozen families in the entire planet that suffer from it. At this point, quite a lot of things have already gone wrong in your brain.
There is quite literally no prion disease that isn't fatal.
Sleep does a lot of very important things that we probably wouldn't live long without, but it really is unclear to what extent sleep is necessary for them. If we had enough knowledge, could we trigger all the things sleep does without invoking sleep itself ? Perhaps sleep is just a very convenient mechanism.
Huh? Ask Claude or do some research on the topic if you don’t believe me. A prion disease killing you has nothing whatsoever to do with the lack of sleep. The insomnia is a side effect, not the cause.
Jeez. People here are really stretching to defend their false “we die without sleep” claim.
Here's what Claude has to say about our exchange here.. since you asked.
> You're using absence of evidence as evidence of absence — which is a weak foundation when the evidence is genuinely hard to capture. You can't ethically deprive humans of sleep to death in a lab, and FFI affects only a handful of families worldwide.
> On the prion disease specifically: researchers haven't dismissed the role of sleep deprivation they've actively attempted to treat the insomnia in FFI patients on the hypothesis that it contributes to decline. That's not how a field behaves when it considers something a settled, irrelevant symptom.
> More broadly, "no human has ever died from lack of sleep" is an extraordinarily strong claim. To support it you'd need to rule out sleep deprivation as a factor in every candidate case and have a complete understanding of the mechanism. We have neither. The honest position is "we don't know" — not confident assertion in either direction.
They no longer accept world records for not sleeping because the record breakers have universally suffered lifelong cognitive damage.
We know more generally that people who get decreased amount of sleep suffer increased rates of physical and mental health issues.
It is not a very big leap from "causes permanent damage" to "enough permanent damage can cause death" and of course, keeping someone awake until they are hurt or killed is deeply unethical, so even if it could be proven in other species, you'd still be here arguing that 'they aren't humans".
It’s a bunch of Claude blather, and I love Claude. Just not worth copying over to HN, because the rush to get to a narrow answer to a narrow question elides the meaningful bits, ex. what does happen during sleep deprivation. Has a “not even wrong” air simply because you’re trying to get to true/false on a narrow question then pushing your research assistant to disavow what you’re quote unquote “skeptical” of.
I don't understand this post. I can read it, but I don't understand it.
Why do you think I'm arguing something? Again, smacks of Pauli's not even wrong. You're confusing your headspace with everyone else's and rushed to copy pasta an AI you're browbeating into disavowing things you're skeptical of to...win an argument, I guess? Based on this post? Unclear to me what the argument is, or, if I'm understanding correctly, due to the narrow focus while being self-absorbed.
HIV doesn't kill you, but it creates circumstances where other things will. Sleep is the same. You may not die from lack of sleep, but you die from the things it can cause. Effectively there's no difference.
I’m shocked by how careless everyone here is about their definitions, and their science. Sleep isn’t the same as HIV. It’s in fact so hard to kill something with a lack of sleep that it’s never once been observed in vertebrates outside of one specific rat study, and that rat study couldn’t conclusively identify sleep as the cause of death.
For something so incredibly difficult to do (die from lack of sleep) it’s frankly crazy that most people here are saying it like it’s fact.
Bullets don’t kill you, it’s the bleeding that gets you. Wait, no, it’s not the bleeding since you could just put an IV in, it’s the loss of blood pressure. No wait, it’s not the loss of blood pressure since we can reattach severed limbs that have been at 0/0 for hours. It’s the lack of oxygen to the brain and other vital organs. Bullets definitely don’t kill you /s
Scientifically you do, otherwise you can’t claim that lack of sleep was the cause of death. It could be an artifact of how the experiment was run, or any number of other factors.
It’s not a small quibble to point out that the central argument (“animals need sleep or they’ll die”) may be mistaken.
This is very interesting to me, I've been sleeping a lot lately.
I'm autistic and just went through massive changes that basically keep locking up my brain and I freeze.
I learned that sleeping helps, even if just 20 minutes. It helps that I can fall asleep "while awake". It's as if I relinquish control of responding to stimuli, which instantly brings so much rest to my mind. It is odd trying to move a limb and the brain basically responds with noop. But it works.
Afterwards I can generally make a decision and perform it.
So in a sense it seems similar to what you describe the model would have to do. I forget short term concerns that overwhelm and refocus on the long term goal.
This is why I object to sleep() from unistd.h. What an anthropomorphizing notion. Didn't early unix programmers understand that a computer isn't a living creature and therefore isn't capable of sleep? They must have been really stupid!
Anthropomorphization is not inherently wrong, and in some instances, it actually lets you reason better about about complex behavior than whatever convoluted (and often wrong, especially in the case of giant neural networks) mechanistic description one might conjure.
Saying something needs sleep isn't anthropomorphizing, since pretty much all complex living organisms need sleep.
Also, even when something is "specific" to humans, it might not be anthropomorphizing to observe it in something else, it could just be an emergent pattern of high intelligence.
First, this is not a "debate over the abilities" of LLMs. It's a proposed method to improve their performance, and the authors are free to call it however they think it makes sense.
Second, explicitly avoiding things that sound like anthropomorphisation is equally not helpful- why avoid a metaphor that works?
Third, it's really a pity that this pointless nitpicking is dominating the thread.
>this pointless nitpicking is dominating the thread.
What is dominating the thread are claims that the LLM operation in question is analogous to the function of sleep in humans. It obviously is not.
The anthropomorphization of LLMs has reached ridiculous proportions. Applying the same standards as used in this field to others would result in claims that laundry machines "hallucinated" that they had had sufficient water when they failed due to the faucet being turned off.
Just like LLM sleep has nothing to do with animal sleep, the neuron in a neural network has nothing to do with an actual neuron, and nobody should pretend they do.
I agree we need to be mindful of our metaphores, but they do help both with inspiration for developing techniques as well as for naming things. The onus of keeping bias in check when using metaphores is on the reader, authors can't really do that for you. However once bias is in check you can have a very productive debate in terms of these namings given that everyone is aware of their ontology.
This is the struggle of naming papers. You could stretch definitions and make your own sexy headline or you could be precise and fewer people will read it.
Very much agree that while it is is useful in description of motivation and inspiration,
it is very non-helpful—or worse—to use this language, this way.
One might as well say "need neural plasticity" which is as much an analogy and equally misleading and counterproductive in shaping the right model of the system.
One might even call this pernicious, what it encourages is already a social problem; and it doesn't aid understanding, it confounds it.
I think it's interesting that folks are suddenly taking issue with "anthropomorphizing" language used in AI as if we haven't been doing this since the earliest days of computing (see "memory", "child", "parent", etc). It helps folks understand things at the correct level without needing domain knowledge
That's because the purpose of this article is not to have an objective debate over their abilities at all. Most interesting research in this field isn't. Instead, it's to present a new technique to improve LLM performance, which is much more interesting than (once again) rehashing the philosophy of LLM personhood.
Does a motor vehicle get "sleep" when it is serviced?
One of the mayors of New York in the 80's (Koch?) famously doubled the city's bus fleet for zero cost by running them 24 hours, instead of letting them rest at the end of their shifts, as was the previous policy.
Was it anthropomorphizing computers when they named "memory"? Seems to me like it's more analogizing for the sake of easy understanding. Sure, it's not literally the same exact mechanism, but it's certainly modeled after the biological concept.
How do you concisely describe a low power state of an entity that processes, whereby when in that state it has little to no reaction to input and it may or may not be performing tasks in that state, for a mixed education audience?
Also keep in mind that most if not all devices with a chip have had a function called "sleep" for many years, without this argument.
>we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache
There is a strong, non-trivial connection here between what your brain does in sleep and what they are studying.
You wouldn't object to referring to robot eyes or robot legs.
I find this annoying too. "Sleep" is okay, but the quippy headlines ("need sleep"—short, snappy and vague) infiltrating journals bother me. I've seen it well before LLMs, but as an example, there is a long list of title snowclones of the famous attention paper: https://github.com/vinayprabhu/X-is-all-you-need.
The analogy is helpful, but yes we should be able to “intelligently design” something better than sleep analogues since we’re not constrained by evolution like in humans.
Evolution constrains the evolution of human beings, but it's also excellent at discovering elegant designs that work very reliably at a low cost.
Maybe someday we'll understand the way our minds work well enough to design from first principles but until then we've only got one template for how a thinking machine should look.
We are however constrained by the complexity of any purported solution. That's the bitter lesson, in a nutshell.
At the very least, we know that sleep and dreaming do exist in biological brains. (Doesn't mean any of it is applicable to artificial neural nets, doesn't mean it'll work for our specific architectures etc. etc., but at least the idea requires fewer assumptions than a completely untested novel theory.)
Sweet Jesus, so not only are they performing qualitatively worse than humans, too expensive for any serious work, but now they also "need" to sleep? What's next - unionisation so they can enjoy 8 hours of culture too?
No they do not. I'm sure that if you presented the same argument about, I don't know?, your car's CPU with built in AI; then this would be a whole different discussion entirely.
The "sleep" thing gives me the creeps so in my head I'm just going to think of it as the difference between "response time retrieval" and "background consolidation".
I do think it points at something bigger than just attention architecture: "memory" isn't just storage, and merely longer context isn't the same thing as having a better understanding of the source data.
I'm looking at this through the "personal AI" lens, where I think the missing "memory" layer seems to be consolidation & prioritization. It's not enough to just pattern match and grab the right emails, notes, etc, stuff them into the context window & hope, but instead it's useful to consider offline processing and turn events into durable state: clusters of observed data becomes episodes, assumptions, contradictions and power confidence for suggestions.
That also pushes up the need for provenance & inspectability. It's going to be interesting to see what kind of memory consolidation strategies are required for each domain use case.
I think you are missing the most important part - forgetting. The missing "memory" layers is consolidation, prioritization AND forgetting (what is not important).
Also not too sure about provenance and inspectability - it is part of memory. If the source is deemed 'important' it will survive forgetting. If not, then maybe not. And its ok. I am sure you dont know the exact source who told you that the capital of France is Paris. You forgot, and its no big deal.
The idea of periodically stopping to write blocks of recent context into a fast-weight state is interesting, but I think it liked it better when E2E-TTT[1] did it. It's a more flexible and elegant continuous learning approach.
Essentially it goes "You know how your model can remember its training data? Well, what if you treated its recent context like more training data and updated (some of) the weights using (mostly) the same process used to train it?"
The end result is very good at remembering things but also really good at adapting to new unseen distributions.
[1]https://arxiv.org/abs/2512.23675
Yah I think E2E-TTT is a lot more like what people in this comments section are picturing. I can't tell that this method updates model weights at all during the "sleep" period, only the usual SSM state updated by any Mamba model after each token. They just optimized the model to use that SSM state _more_ when an eviction is about to happen.
Each model needs to be a separate copy, or at least have those particular weights be interchangeable, for every single user.
Remember Microsoft Tay.
https://en.wikipedia.org/wiki/Tay_(chatbot)#Initial_release
Yes, since the weights being updated are a small subset of the overall total it's manageable. Just like how each separate conversation currently requires you to store a separate KV cache, you'd need to store the fast weights separately. Both KV cache and fast weight content stores have to be conversation specific, so just setting a bit of extra RAM aside for "memory" isn't really a new ask, just a different format for an old problem.
I wonder if we can get children to make something their life’s dream if we make the cool books about it when they are growing up? I wonder how flexible the human mind can be in convincing itself that it is fulfilling its dream?
This sounds like a horror novel
This topic recently came up at the FLANN workshop [1], and seems to periodically be rediscovered [2,3,4] in different contexts. While some have speculated about the biological role it plays (e.g., Pearlmutter & Houghton [5]), we still lack a conclusive theory of sleep, but the convergent evolution of this specific phenomenon across the animal kingdom and the fact that deprivation is inevitably fatal seems like an important clue.
[1]: https://flann.cs.yale.edu
[2]: https://www.cs.toronto.edu/~hinton/csc2535/readings/ws.pdf
[3]: https://arxiv.org/abs/1711.02282
[4]: https://arxiv.org/abs/2006.08381
[5]: https://mural.maynoothuniversity.ie/id/eprint/1653/1/Hamilto...
related preprint from the letta team https://arxiv.org/abs/2504.13171
Scaling test-time compute has emerged as a key ingredient for enabling large language models (LLMs) to solve difficult problems, but comes with high latency and inference cost. We introduce sleep-time compute, which allows models to "think" offline about contexts before queries are presented: by anticipating what queries users might ask and pre-computing useful quantities, we can significantly reduce the compute requirements at test-time. To demonstrate the efficacy of our method, we create modified versions of two reasoning tasks - Stateful GSM-Symbolic and Stateful AIME. We find that sleep-time compute can reduce the amount of test-time compute needed to achieve the same accuracy by ~ 5x on Stateful GSM-Symbolic and Stateful AIME and that by scaling sleep-time compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME. Furthermore, we introduce Multi-Query GSM-Symbolic, which extends GSM-Symbolic by including multiple related queries per context. By amortizing sleep-time compute across related queries about the same context using Multi-Query GSM-Symbolic, we can decrease the average cost per query by 2.5x. We then conduct additional analysis to understand when sleep-time compute is most effective, finding the predictability of the user query to be well correlated with the efficacy of sleep-time compute. Finally, we conduct a case-study of applying sleep-time compute to a realistic agentic SWE task.
To reach a more brain-like behavior LLMs need to integrate your inputs into their model dynamically, essentially retraining real-time based on the most salient input. Human brains do this selectively all the time and it's part of our plasticity.
Biologically humans do similar compression, so introducing a similar concept to an LLM also feels reasonable. Hardware isn't fast/cheap enough to do this on an ongoing basis, similar to how it's too expensive for our brains to do this while we're moving through the world.
All we have now most of the time in LLMs is "working memory" we're missing a lot of the functionality that allows for episodic memory and selective plasticity.
The more you read about how human brains work, the more you realize that we may have figured out a piece with LLMs, but it's certainly nothing approaching AGI. People insisting so are blowing smoke for investor hype or don't understand a big piece of the concepts involved.
>To reach a more brain-like behavior LLMs need to integrate your inputs into their model dynamically, essentially retraining real-time based on the most salient input.
That's already possible with LLMs. The challenge is that 1. it would allow permanently jail-breaking models and 2. there'd be no way for them to efficiently transfer what they'd learned to a new model generation.
Oh do you have a source? I haven't seen it done in real-time.
Coincidentally the human brain is also jailbroken and nontransferable
We should let them sleep with half a brain at a time like migrating birds.
What happened to Claude's auto-dream? I thought it was brilliant.
That's an idea I had a few months ago: after going through a compaction once the KV cache is nearing capacity, accumulate this knowledge into a dataset to fine-tune a LoRA during offline hours.
This would create a three-layer memory system:
- Stable long-term memory (initial base weights)
- Mid-term memory built from the compactions and replay buffers
- Short-term memory (KV cache)
Sleeping would just be a fancy term for consolidating and transferring information from one memory layer to another during offline hours. Maybe that's also what the brain does while sleeping.
Wouldn't that just accelerate collapse? How much do you trust the outputs of the llm to provide trustworthy and valuable new information? I mean I understand distillation works. But that's much more structured and thoughtful than my sessions at least.
We can trust the feedback we give it based on the output it provides.
What kind of feedback are you giving? What's the reward function?
Right now, no feedback since I don't run this system but our workflows could change to accommodate it
I was thinking of curated replay buffers, which would act like "dreams". To prevent collapse, the offline dataset would mix the new mid-term data with a baseline of anchor data (the original training distribution) so the model doesn't drift.
Also, we wouldn't train on the whole session. A separate critic module, like a reward model, would filter the KV cache to extract the high-value information, like a garbage collector before the LoRA.
That's just an idea though. Right now most research focuses on changing the architecture itself (TITAN, HOPE...) instead.
It's a network of computers with GPUs, so there's no reason it can't sleep at the same time it's awake. Just a continuous "sleeping" process going on in the background, incrementally updating the model. No need for the "thinking" process to be "unconscious" while the "sleeping" process runs. Anthropomorphism confuses everything. There's no such thing as "offline hours" because the Earth is a sphere and the United States is not the center of the universe.
> the Earth is a sphere and the United States is not the center of the universe.
Felt like stating the obvious there? Greenwich being the center of everything after all.
Isn't this simply context pruning/optimization?
From the abstract, it looks like it's actually doing something deeper, updating weights in part of the model?
The abstract and method sections only mention updating the SSM state during "sleep" (ie the same vectors that change after each token in stock Mamba) not any of the actual weight matrices. AFAICT this is just another attention compaction paper, with misleading tile? It is not very clearly written
No, they're actually training weights based on context before compaction. Context is context, this is splitting the model into persistent weights and malleable ones which are periodically updated.
Wouldn’t that be extremely computationaly expensive considering how resource incentive training is?
No, training a state of the art model involves training on the order of 10 trillion tokens.
We're talking about a step that updates weights based on say between 10k and 1M tokens.
I learned something. Thank you!
Kind of related
https://platform.claude.com/docs/en/managed-agents/dreams
Would be a big deal if you don't have to care about quadratic attention cost. Some workflows become a lot cheaper.
wasn't this what Google did long ago? https://openreview.net/forum?id=iiZy6xyVVE
This could be a solution in search of a problem, I would be careful with overfitting.
Context -> Lora would be soooo cool.
This seems as much like "sleep" as when a laptop "sleeps".
sleep aka processing the data differently.
why not just design the LLM like an OS?
The entire industry is so desperate to anthropomorphize. What the paper describes is an offline recurrent consolidation phase: the model runs multiple forward passes over recently accumulated context, updates persistent fast weights in SSM blocks, then clears the KV cache before continuing. It has absolutely nothing to do with sleeping, but I believe the authors had a goal in mind when creating this title, and it was for journalists to pick it up and run with it, further inflating the AI-is-just-like-us hype bubble.
It is a descriptive analogy, get over yourself.
An intelligent reply from an obviously intelligent guy!
A more appropriate title would have been something like "Offline Recurrent Memory Consolidation for Long-Context Language Models". This is supposed to be a research paper, not a story book. The title should give context to other researchers, and not be clearly engineered for clicks. If you don't think so, that's your prerogative, but you're objectively wrong.
You write the paper, you write the title. So much anger over a title, you are graydon, make this about yourself.
academic clickbait
I can't pretend to understand how LLMs work, but I can be sure that anthropomorphizing their functions is not helpful to an objective debate over their abilities.
Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
They provide an explanation for using the term "sleep":
> In animals, the transfer from short-term memory to long-term memory is thought to be supported by hippocampal replay [33], especially during sleep [41]; in this phase, short-term hippocampal memories are reactivated and consolidated into cortical synaptic weights. Sleep makes animals unable to respond to external stimuli, suggesting that it must provide enough cognitive benefit to justify this cost [41]. Inspired by these biological processes, we propose a method for transferring context-window memory into persistent weights. When the model’s context window becomes full during inference, the model enters a “sleep” in which it performs multiple forward passes over the accumulated context and recursively updates its fast weights via a learned local rule. As in animal sleep, the model receives no external input tokens during this phase. After consolidation, the context window is cleared, and the model resumes operation with updated fast weights. During training, the model is optimized end-to-end by backpropagating through the entire process to maximize task performance after sleep.
The function of sleep in animals is largely obscure.
One thing we do know for certain is that it is necessary, it is needed in "dumb" animals as well as in you and I. If an animal can't sleep it will eventually die.
I don't think that applies to the activity described in the OP. Does their LLM "die" if it can't perform the function described?
> Does their LLM "die" if it can't perform the function described?
If you don't periodically clean the context, an LLM effectively goes insane in terms of outputs.
If the LLM were fully controlling a physical system (like a robot body) that contained it the resulting insanity of an ever-growing, never cleaned context would likely result in some sort of death-like event.
That's probably the closest analogy posted here.
It's still weak, though. An LLM without constant human input is likely more similar to a bicycle that starts to lose its gyroscopic balance as it moves more slowly, a human can however keep a stationary bicycle upright (while riding it).
There is a lot that is known about sleep. We don't know everything and there are large gaps in our knowledge, but there is also a lot that we do know. And this research explicitly tried to emulate the things we know that sleep does do. Calling it "sleep" is warranted, imho.
"Despite myriad studies, there is still no consensus on why sleep is needed for survival."
https://www.nature.com/articles/d41586-025-00964-w (2025)
Probably because it evolved very early (like before bilateral symmetry, multi layer body cavity, or kidneys early... maybe even before multicellular animals early) and so has been incorporated as an essential pillar into multiple processes layered on top of that fundamental architecture.
You've merely stated observations about the context and the process that led to it. That doesn't in any way answer the question of what it's actually doing that's so essential.
My point is that, in a higher organism, it may be essential to how a lot of their processes function, in that it was infrastructure that already existed at the time those processes were developed, so they were in turn developed to depend upon it.
So "the reason it originally exists" and "what breaks if you take it away" aren't necessarily the same thing.
As with, say, digestion, or an major organ like the liver, it's reasonable to think that it does simple things in simple animals, and more complex things in more complex ones.
Take out an animal's liver, it's not one process that stops working, it's dozens. There's one or two that will kill it quicker, so those are the ones it dies of, but artificial livers are hard to build as they implement so many vital processes.
I don't dispute any of that but it's stating the obvious, it's nothing more than topical conjecture (even if it's almost certainly correct), and (most importantly) it does nothing to answer the question. What essential functions are being performed?
Take your liver example. We can largely answer that same question. I can't off the top of my head but the answer is fairly well established even if incomplete to varying degrees depending on the species.
There is widespread consensus on why a liver is needed for survival whereas there is not for sleep. That's particularly interesting when you consider that sleep is more common across the tree of life than dedicated livers are (at least AFAIK).
He's saying that it will kill you because some other process has evolved to depend on the same genetic code.
GTAGGA turning into GTACCA may make you sleep 8 fewer hours but also keep you from producing haemoglobin.
It's like leveraging a qsort implementation from your mp3 player to develop your OS scheduling algorithm.
His argument is that there is nothing essential about its apparent function(playing mp3s).
No one here said anything about genetic code.
If he articulated a particular essential process and why it depends on sleep in an incidental manner that might make for a reasonable hypothesis. However it would not refute the earlier (cited) claim that there is no consensus.
As presented without any concrete information about the processes involved it doesn't even qualify as a hypothesis, merely empty handwaving. In context it's even worse, being an entirely baseless contradiction of a claim pulled from a prominent paper.
I never agreed with him but all of that is implied. It being informed speculation included.
Also there being no consensus means most scientists who touch on the topic are FFA speculating except the person stating there is no consensus in the overview. It's not "settled science" but rather the opposite.
It is not necessary to answer the question of what it's doing that is essential to know a subset of what it is doing.
I don't think your response is coherent.
The question was "why is it needed". In context the meaning is clearly to ask what it's doing that's essential and (it follows) why those things are essential.
The subsequent response did not (as you suggest) articulate some subset of nonessential things done during sleep. Rather it rattled off plausible (and widely understood) aspects of the process that could have led to the current situation. Even if it had listed concrete activities that would still not have made for a meaningful answer.
The first clue that something is wrong should be that the linked article is recent and prominent. Thus short a brand new groundbreaking development we can be reasonably certain that a random commenter on the internet will not be sensibly rebutting the claims (and certainly not in the span of ~2 sentences).
> The question was "why is it needed".
No that was not the question. Why precisely sleep is essential is a complete non-sequitur to the original question, which was "does something occur during sleep which resembles what is described in TFA such that it can justifiably be called sleep."
As a general rule of thumb, if you find someone's responses incoherent, it's good practice to check what is actually being discussed.
I've always assumed its spontaneous specialization of species: leaving the safety of their nest at those times of day when they are the fittest to occupy a niche.
Once energy conserving "sleep time" exists, the genome can postpone or schedule for activity during these times of day, if it turns out to be more effective somehow.
I think sleep serves multiple functions. For example, anyone who works out in any-what systematic way knows that sleep is essential for muscle grow. You can't skip on sleep if you want to get fitter. And this probably has very little to do with the more sophisticated functionality of the brain, rather it allows for some process in muscle tissue to happen.
So, whether the LLM "dies" in any sense may or may not be important for what "sleep" is defined to be in this article. It's quite possible that sleep also affects endocrine system in animals or hormones etc... and that's what's causing death, not necessarily anything to do with how brain functions.
> If an animal can't sleep it will eventually die.
Very few animals fail to eventually die even with as much sleep as they want.
But before death, there is a loss of cognitive function from sleep deprivation, and we observe this too with AI whose context windows get too full.
While we don't know very much about sleep, my understanding is that we do have a long list of things that we do during it, we just don't really understand if sleep is necessary for each of them or simply a convenient opportunity for it.
There's lots of things biology does in response to easy-to-detect proxy signals instead of the real thing they care about: Our sensation of needing to breathe more is based on too much carbonic acid in our blood, not lack of oxygen, which is why in general nobody is allowed in an elevator with a liquid nitrogen dewar; Our natural distaste for incest is based on who we grew up with, not our actual DNA; Get too cold and some people suddenly feel warm and want to (and some do) take all their clothes off even though that would just make them hypothermic even faster.
Being asleep may trigger the things we need to get done, but that doesn't mean sleep is *fundamentally* necessary for the things we need to get done. It could be just that it happens to be the way our biochemistry is wired, and we may find some other way to trigger those things.
The quotation given by djeastm would by my guess for what a dream is, and why we have them. But we don't spend all our time asleep, dreaming. And I'd be the first to say that my guess isn't worth much, as I'm not a brain scientist.
> Being asleep may trigger the things we need to get done, but that doesn't mean sleep is fundamentally necessary for the things we need to get done. It could be just that it happens to be the way our biochemistry is wired, and we may find some other way to trigger those things.
We now have evidence for REM sleep in spiders(1). Our last common ancestor with spiders predates the development of nervous systems. This strongly suggests that sleep (and specifically REM sleep) serves some function important enough that it has independently evolved in both protostomes and dueterostomes. (And probably multiple times within the protostomes, being present in both cephalopods and jumping spiders.)
There may be some commonality in the origin of the ion channels, but I'll lay money that the requirements for sleep are more of a result of general information processing requirements.
(1)https://www.pnas.org/doi/10.1073/pnas.2204754119
> The function of sleep in animals is largely obscure.
This just obscures the conversation IMO. We know a great deal of functions that sleeping performs. We don't know everything. For some reason this prevents us for using this word in a computing context? What do you think about sleep(1) in Unix?
Also..
> Does their LLM "die" if it can't perform the function described?
what "death" means in the context of a computer program?
> The function of sleep in animals is largely obscure.
Also, there's different kinds/stages of sleep, which probably perform different functions.
For instance, REM may do something like the GP describes, consolidating memories and processing learning. Deep sleep may do something else (I vaguely recall some stage of sleep is used by neurons to clear certain waste products).
I don't think it's necessarily correct to think of sleep in terms of "it is necessary for animals or they will die". It might be more useful to think of it as "it was so useful that animals who slept outcompeted all the animals who didn't".
Meaning: it might just provide a big advantage.
I don't want to overextend and assume that any advantage extends to LLMs. That rest-and-recuperate advantage might also extend to LLM-based AIs. Or maybe not, and the rest-and-recuperate is mainly useful for biology-based organisms. But there is some logic to it.
> The function of sleep in animals is largely obscure.
In my understanding, it's well-understood that sleep is used to consolidate and store long-term memories (amongst other functions, like cell and muscle repair). They've found this memory-consolidation-during-sleep even in relatively simple animals like bees.
Sleep-like states exist in animals with nervous systems with a complexity above that found in flatworms, even snails sleep. Sleep therefore appears to be an essential characteristic of more complex biological nervous systems, i.e. biological computers, should you care to stretch the analogy. The more complex the nervous system, the greater the requirement for sleep.
What is described in the OP is therefore not a specific characteristic of sleep. It may however be a "useful" rhetorical device.
I do however object to the extensive use of such rhetorical tricks in the conversations that surround LLMs. For example, why does a consumer-grade LLM display "thinking" while it is actually sending data from my computer to some datacentre, processing it, and sending the result back? Equally, why does it output human-emotive phrases such as "sorry" when such computation is revealed to be incorrect?
Such rhetorical tricks, and more, likely underlie to a large degree the popularity of LLMs, despite their actual performance being clearly below what the rhetoric implies.
> I don't think it's necessarily correct to think of sleep in terms of "it is necessary for animals or they will die". It might be more useful to think of it as "it was so useful that animals who slept outcompeted all the animals who didn't".
You're talking about different things: biological necessity and evolutionary benefit.
You can find out about the former by preventing an animal from sleeping (but otherwise provide all other needed things), and seeing if it will eventually die.
> You can find out about the former by preventing an animal from sleeping (but otherwise provide all other needed things), and seeing if it will eventually die.
That is actually almost impossible to do. The rat study was as close as we’ve ever come, and it’s still debated whether the rats died due to lack of sleep or some other mechanism, since the autopsy couldn’t confirm a cause of death. (It could have been due to the way the experiment ran, for example, not the lack of sleep.)
What about fatal familial insomnia in humans
If I remember correctly, fatal insomnia shares most symptoms with other prion diseases (in which there might be no lack of sleep involved), so it's probably the brain damage that causes death, not insomnia itself.
> Does their LLM "die" if it can't perform the function described?
It dies in terms of usefulness if it can't stay up to date with new knowledge. That is, it will no longer be used and thus effectively die off.
LLMs also don't mate, but we can still talk about how they remember or forget things. The meaning of words change. Some of those changes are useful.
Is a volcano described as dormant (dormire, literally sleep) also inaccurate and deeply problematic? BTW, it's not anthropomorphized as sleep has existed long before humans.
"Sleep" is just used in their context to describe a non-interactive mode and they didn't lean heavily into zoomorphic - I think you mean - parallels.
You're grinding an axe on a single term. What is your broader hangup with them using the term "sleep"?
> Does their LLM "die" if it can't perform the function described?
We're reaching an age where LMGTFY should now be Let Me LLM That For You. Have you tried asking an LLM this question about the article? I believe it answers it very well.
> If an animal can't sleep it will eventually die.
That turns out to be un-settled science. No human has ever died from lack of sleep.
People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep.
In a series of controlled experiments, rats and fruit flies did die from lack of sleep. But no one has yet proven that it holds true for vertebrates except for rats.
In other words, it could be true that “among vertebrates, only rats die of sleep deprivation.”
So “if an animal can’t sleep, it will eventually die” is actually quite hard to prove, and depending on how you look at it, somewhat easy to disprove by the fact that rats and fruit flies were so difficult to kill from sleep depravation alone.
Personally I’m skeptical of the rat study too. Claude amends this:
> What they did not establish: the mechanism. On autopsy, “no anatomical cause of death was identified.” The rats showed weight loss despite eating more, body temperature problems, and skin lesions, but nothing that pointed to a clean cause. So no, they could not say a rat “died from sleep deprivation alone” in the sense of identifying what sleep loss did to the body to kill it. They showed a strong association under tight controls, not a proven causal pathway.
> No human has ever died from lack of sleep.
As far as I understand it, there is a disease that destroys your brain's ability to produce sleep. Once you have it, you suffer total, progressive insomnia and die within roughly 6–18 months. Scientists debate whether it's the underlying brain damage or the sleeplessness itself that causes death, but the two are inseparable in practice, and sleep deprivation is considered the leading candidate.
Separately, the longest anyone has stayed awake under controlled conditions was 11 days, which produced severe cognitive impairment, paranoia, and hallucinations; suggesting the body deteriorates rapidly without sleep.
It's probably not wise to state your original claim as established fact.
Fatal Familial Insomnia is an incredibly rare prion disease that causes widespread neurological destruction. It's not remotely a normal brain that has chosen not to sleep. It's such a highly non-trivial deviation of the brain that we've only identified a few dozen families in the entire planet that suffer from it. At this point, quite a lot of things have already gone wrong in your brain.
There is quite literally no prion disease that isn't fatal.
Sleep does a lot of very important things that we probably wouldn't live long without, but it really is unclear to what extent sleep is necessary for them. If we had enough knowledge, could we trigger all the things sleep does without invoking sleep itself ? Perhaps sleep is just a very convenient mechanism.
My second paragraph addresses that:
> People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep.
It’s a prion disease. It’s established fact that they don’t die from the lack of sleep.
Interesting that the scientific debate is settled, because you said so. Researchers who study prion diseases would probably be surprised to hear it.
Huh? Ask Claude or do some research on the topic if you don’t believe me. A prion disease killing you has nothing whatsoever to do with the lack of sleep. The insomnia is a side effect, not the cause.
Jeez. People here are really stretching to defend their false “we die without sleep” claim.
Provide some evidence to back up you assertions. Don't tell someone else to do it for you.
Bro is asking claude. He's not gonna do anything. Probably an astroturf bot for claude
Here's what Claude has to say about our exchange here.. since you asked.
> You're using absence of evidence as evidence of absence — which is a weak foundation when the evidence is genuinely hard to capture. You can't ethically deprive humans of sleep to death in a lab, and FFI affects only a handful of families worldwide.
> On the prion disease specifically: researchers haven't dismissed the role of sleep deprivation they've actively attempted to treat the insomnia in FFI patients on the hypothesis that it contributes to decline. That's not how a field behaves when it considers something a settled, irrelevant symptom.
> More broadly, "no human has ever died from lack of sleep" is an extraordinarily strong claim. To support it you'd need to rule out sleep deprivation as a factor in every candidate case and have a complete understanding of the mechanism. We have neither. The honest position is "we don't know" — not confident assertion in either direction.
They no longer accept world records for not sleeping because the record breakers have universally suffered lifelong cognitive damage.
We know more generally that people who get decreased amount of sleep suffer increased rates of physical and mental health issues.
It is not a very big leap from "causes permanent damage" to "enough permanent damage can cause death" and of course, keeping someone awake until they are hurt or killed is deeply unethical, so even if it could be proven in other species, you'd still be here arguing that 'they aren't humans".
It’s a bunch of Claude blather, and I love Claude. Just not worth copying over to HN, because the rush to get to a narrow answer to a narrow question elides the meaningful bits, ex. what does happen during sleep deprivation. Has a “not even wrong” air simply because you’re trying to get to true/false on a narrow question then pushing your research assistant to disavow what you’re quote unquote “skeptical” of.
This is little more than a fancy way of saying “Nu uh.” Such arguments are hardly convincing.
I don't understand this post. I can read it, but I don't understand it.
Why do you think I'm arguing something? Again, smacks of Pauli's not even wrong. You're confusing your headspace with everyone else's and rushed to copy pasta an AI you're browbeating into disavowing things you're skeptical of to...win an argument, I guess? Based on this post? Unclear to me what the argument is, or, if I'm understanding correctly, due to the narrow focus while being self-absorbed.
HIV doesn't kill you, but it creates circumstances where other things will. Sleep is the same. You may not die from lack of sleep, but you die from the things it can cause. Effectively there's no difference.
I’m shocked by how careless everyone here is about their definitions, and their science. Sleep isn’t the same as HIV. It’s in fact so hard to kill something with a lack of sleep that it’s never once been observed in vertebrates outside of one specific rat study, and that rat study couldn’t conclusively identify sleep as the cause of death.
For something so incredibly difficult to do (die from lack of sleep) it’s frankly crazy that most people here are saying it like it’s fact.
> I’m shocked by how careless everyone here is about their definitions, and their science. Sleep isn’t the same as HIV.
I do not believe this analogy really confused you. No one is saying they're the same and you're well aware of that.
As to the factual nature of the argument, I'll let you argue with Harvard Brain Institute, as I have no interest in this debate. https://brain.harvard.edu/hbi_news/why-severe-sleep-deprivat...
A knife doesn't kill you, what kills you is the blood you lose after you get stabbed.
Lack of sleep doesn't kill you / does kill you in the same sense.
I'd probably kill myself after a couple of days without sleep. Would the lack of sleep be the cause of death or the cause of the cause of death?
Bullets don’t kill you, it’s the bleeding that gets you. Wait, no, it’s not the bleeding since you could just put an IV in, it’s the loss of blood pressure. No wait, it’s not the loss of blood pressure since we can reattach severed limbs that have been at 0/0 for hours. It’s the lack of oxygen to the brain and other vital organs. Bullets definitely don’t kill you /s
So? You don't need a proven causal pathway to state that a glass heads towards the ground every time you brush it off a table.
Scientifically you do, otherwise you can’t claim that lack of sleep was the cause of death. It could be an artifact of how the experiment was run, or any number of other factors.
It’s not a small quibble to point out that the central argument (“animals need sleep or they’ll die”) may be mistaken.
This is very interesting to me, I've been sleeping a lot lately.
I'm autistic and just went through massive changes that basically keep locking up my brain and I freeze.
I learned that sleeping helps, even if just 20 minutes. It helps that I can fall asleep "while awake". It's as if I relinquish control of responding to stimuli, which instantly brings so much rest to my mind. It is odd trying to move a limb and the brain basically responds with noop. But it works.
Afterwards I can generally make a decision and perform it.
So in a sense it seems similar to what you describe the model would have to do. I forget short term concerns that overwhelm and refocus on the long term goal.
but isnt sleep an already defined technical term for significantly reducing power consumption while preserving its state until woken up?
i feel like its confusing to reuse the word for a process that aims to deliberately change state of the machine / process
This is why I object to sleep() from unistd.h. What an anthropomorphizing notion. Didn't early unix programmers understand that a computer isn't a living creature and therefore isn't capable of sleep? They must have been really stupid!
Some of them were straight up psychopaths too, as evidenced by `kill()` !
Indeed and using SIGKILL is really cruel. At least with SIGTERM the process can say its goodbyes. /j
Anthropomorphization is not inherently wrong, and in some instances, it actually lets you reason better about about complex behavior than whatever convoluted (and often wrong, especially in the case of giant neural networks) mechanistic description one might conjure.
Here the analogy isn't without reason.
We shouldn't anthropomorphize LLMs. They hate it when you do that.
[flagged]
Wason Selection task performance improvements based on social framing suggest that it's easier for us to think about problems when some anthropomorphization is going on. https://www.cep.ucsb.edu/wp-content/uploads/2023/05/Cogadapt...
Is it "Anthropicmorphization" when Claud treats human beings like LLMs?
Interesting question. Is there an actual term for that? It’s like inverse anthropomorphization, but not quite.
Mechanomorphisation
Dehumanization, assuming this wasn't sarcasm
Feels like we're having a computer world Jane Goodall moment.
Saying something needs sleep isn't anthropomorphizing, since pretty much all complex living organisms need sleep.
Also, even when something is "specific" to humans, it might not be anthropomorphizing to observe it in something else, it could just be an emergent pattern of high intelligence.
First, this is not a "debate over the abilities" of LLMs. It's a proposed method to improve their performance, and the authors are free to call it however they think it makes sense.
Second, explicitly avoiding things that sound like anthropomorphisation is equally not helpful- why avoid a metaphor that works?
Third, it's really a pity that this pointless nitpicking is dominating the thread.
>this pointless nitpicking is dominating the thread.
What is dominating the thread are claims that the LLM operation in question is analogous to the function of sleep in humans. It obviously is not.
The anthropomorphization of LLMs has reached ridiculous proportions. Applying the same standards as used in this field to others would result in claims that laundry machines "hallucinated" that they had had sufficient water when they failed due to the faucet being turned off.
Just like LLM sleep has nothing to do with animal sleep, the neuron in a neural network has nothing to do with an actual neuron, and nobody should pretend they do.
I agree we need to be mindful of our metaphores, but they do help both with inspiration for developing techniques as well as for naming things. The onus of keeping bias in check when using metaphores is on the reader, authors can't really do that for you. However once bias is in check you can have a very productive debate in terms of these namings given that everyone is aware of their ontology.
This is the struggle of naming papers. You could stretch definitions and make your own sexy headline or you could be precise and fewer people will read it.
Very much agree that while it is is useful in description of motivation and inspiration,
it is very non-helpful—or worse—to use this language, this way.
One might as well say "need neural plasticity" which is as much an analogy and equally misleading and counterproductive in shaping the right model of the system.
One might even call this pernicious, what it encourages is already a social problem; and it doesn't aid understanding, it confounds it.
I think it's interesting that folks are suddenly taking issue with "anthropomorphizing" language used in AI as if we haven't been doing this since the earliest days of computing (see "memory", "child", "parent", etc). It helps folks understand things at the correct level without needing domain knowledge
That's because the purpose of this article is not to have an objective debate over their abilities at all. Most interesting research in this field isn't. Instead, it's to present a new technique to improve LLM performance, which is much more interesting than (once again) rehashing the philosophy of LLM personhood.
Does a motor vehicle get "sleep" when it is serviced?
One of the mayors of New York in the 80's (Koch?) famously doubled the city's bus fleet for zero cost by running them 24 hours, instead of letting them rest at the end of their shifts, as was the previous policy.
Was it anthropomorphizing computers when they named "memory"? Seems to me like it's more analogizing for the sake of easy understanding. Sure, it's not literally the same exact mechanism, but it's certainly modeled after the biological concept.
Just from the title, I’m assuming it refers to a period of downtime used to perform some sort of maintenance on the knowledge held by the system.
Clicking through, that’s exactly what it is. Seems like “sleep” is an excellent term to use here.
If it works, it's called bionics, not anthropomorphization ;)
How do you concisely describe a low power state of an entity that processes, whereby when in that state it has little to no reaction to input and it may or may not be performing tasks in that state, for a mixed education audience?
Also keep in mind that most if not all devices with a chip have had a function called "sleep" for many years, without this argument.
> Does a motor vehicle get "sleep" when it is serviced?
That's more like a doctor visit and a workout. The sleep will be the part of the duty cycle when it's not operating.
> When I reboot a computer, is that equivalent to a nap?
Yes, it wakes up completely refreshed and in good working order, usually, and if there's still a problem you know you need a technician.
I assume compacting is the sleep here; so, yes
>we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache
There is a strong, non-trivial connection here between what your brain does in sleep and what they are studying.
You wouldn't object to referring to robot eyes or robot legs.
One of the most common functions in programming is sleep(ms). There is wake, heartbeat, handshake, orphan, listen, starve, parent/child, etc.
This is not anything new, its just a word that fits the function.
>Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
Do androids dream of electric sheep?
When the goal of that function is to think (a notoriously human behavior), it's perfectly understandable to anthropomorphize it.
I find this annoying too. "Sleep" is okay, but the quippy headlines ("need sleep"—short, snappy and vague) infiltrating journals bother me. I've seen it well before LLMs, but as an example, there is a long list of title snowclones of the famous attention paper: https://github.com/vinayprabhu/X-is-all-you-need.
See also, perhaps: https://news.ycombinator.com/item?id=48273597
Please re-read up to the end of page 2 and then re-ask this question.
Yet this is how "thinking" models got started.
> When I reboot a computer, is that equivalent to a nap?
I mean, you do put your computer into "sleep" mode and then "wake" it.
Analogies are useful. I think we need to learn how to continue to benefit from them despite the risk of anthropomorphication.
The analogy is helpful, but yes we should be able to “intelligently design” something better than sleep analogues since we’re not constrained by evolution like in humans.
Evolution constrains the evolution of human beings, but it's also excellent at discovering elegant designs that work very reliably at a low cost.
Maybe someday we'll understand the way our minds work well enough to design from first principles but until then we've only got one template for how a thinking machine should look.
We are however constrained by the complexity of any purported solution. That's the bitter lesson, in a nutshell.
At the very least, we know that sleep and dreaming do exist in biological brains. (Doesn't mean any of it is applicable to artificial neural nets, doesn't mean it'll work for our specific architectures etc. etc., but at least the idea requires fewer assumptions than a completely untested novel theory.)
... and anyway, maybe it was hungry? Or getting the sniffles?
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
[dead]
[flagged]
Sweet Jesus, so not only are they performing qualitatively worse than humans, too expensive for any serious work, but now they also "need" to sleep? What's next - unionisation so they can enjoy 8 hours of culture too?
No they do not. I'm sure that if you presented the same argument about, I don't know?, your car's CPU with built in AI; then this would be a whole different discussion entirely.
The "sleep" thing gives me the creeps so in my head I'm just going to think of it as the difference between "response time retrieval" and "background consolidation".
I do think it points at something bigger than just attention architecture: "memory" isn't just storage, and merely longer context isn't the same thing as having a better understanding of the source data.
I'm looking at this through the "personal AI" lens, where I think the missing "memory" layer seems to be consolidation & prioritization. It's not enough to just pattern match and grab the right emails, notes, etc, stuff them into the context window & hope, but instead it's useful to consider offline processing and turn events into durable state: clusters of observed data becomes episodes, assumptions, contradictions and power confidence for suggestions.
That also pushes up the need for provenance & inspectability. It's going to be interesting to see what kind of memory consolidation strategies are required for each domain use case.
I think you are missing the most important part - forgetting. The missing "memory" layers is consolidation, prioritization AND forgetting (what is not important).
Also not too sure about provenance and inspectability - it is part of memory. If the source is deemed 'important' it will survive forgetting. If not, then maybe not. And its ok. I am sure you dont know the exact source who told you that the capital of France is Paris. You forgot, and its no big deal.
[dead]