So then you are talking about weight updates, not just memorization, which makes it much harder, and right away really messes with the business model of cloud-based AI where it's the same model being served to everyone.
Even if we had an algorithm to incrementally update weights without catastrophic forgetting (I don't believe we do - but doesn't seem like such a tough problem), then this implies that everyone has their own personalized model, else if you combine all these updates there is no data privacy. Cloud serving also really depends on everyone using the same model so that you can batch requests and not reload weights for each user.
The much more achievable goal, without needing to upend the whole serving model, would be just to implement continual "episodic" memorization (text-only maybe), but even with compaction/consolidation the next question would be how do you retrieve these external memories and get the necessary chunks into context when needed. Some sort of vector store, perhaps? How do you avoid vendor lock-in - perhaps have agents store/retrieve vector-store context chunks from a vendor-agnostic cloud store?
So, even the most basic form of continual-learning-like enhancement becomes tricky. What you'll first see is presumably just enhancements of the vendor-specific memory mechanisms that are already available.
What I would consider as true continual learning, close to what an animal or human does, would require much more extensive architectural and deployment/business model changes - since then we're really talking about more than just an update/recall problem, but rather the whole autonomous agentic loop of predicting/acting/failing/learning/etc with innate traits like curiosity (prediction failure) and boredom to make sure the agent is exposed to learning situations in the first place. At this point you are building an artificial brain, not just an LLM. Some companies such as Google/DeepMind have a more ambitious definition of AGI (more than just an LLM) that is perhaps a step in this direction.
Even with this sort of animal-like continual exploration/learning, you still wouldn't have something that is human level, but at least much closer in terms of ability to learn.
The reason i am asking is about research mathematics. In this application what is really important is continual intuition and theory building, this takes time and trial and error. However this is very fruitful in solving problem. Currently the LLM can do mathematics that is somehow already in their weight very well. See the recent things. But i suppose true research would require continual learning that goes beyond the context window. Thats why i am wondering if this solvable. My intuition is that it is easily solvable, hence we could make a true mathematical continuously building theory to prove things.
For that sort of application then maybe a chunked vector store memory would work if 1M context is too small to hold the entire history? In Terrance Tao's ChatGPT discussion it seemed that the LLM was making good use of the in-context information.
I think for true, especially open-ended, research you really also need things like curiosity, but perhaps a lot can still be done with the same sort of prompting that got the Jacobian result - just tell the model to keep on going, investigate anything that seems interesting/unexplained, etc?
Yes maybe you could scheme it, like think of 10 things you could try, spawn subagents, try them, perhaps one would succseed and yes that think that worked could be placed in a memory system. But there is also value in things that dont work ect. So true continual learning would be really good. I am currently reasearching in mathematics and some things it cant get, i really have to think of them myself. But other things where it is easier to "build" from already existing knowledge works really well. I kinda wanne know how this will continue. It can do things that are "in distribution", but can it build its own new distribution and continue building. Mabye ill try this using a normal RAG system. I would concider it a succsess if it create a new "theory" like 100 pages genuinely new stuff, and then applies it to a problem. But my intuition is this will not work lol
The issue remains the context window. I haven't seem any example of a "memories" system that the agent can update over time without it devolving into slop. The magic of the human brain is that we seem to have decently sophisticated heuristics parsing/saving memories as well as for letting memories that aren't accessed decay.
Yes i agree, but somehow knowledge got put into the LLMs head during train time, but why does it not work during inference time? Too little examples? Do llms know stuff with exactly one occurence in the training data?
What gives LLMs the ability to reason so well about math specifically is the math-specific RL training they are given where they are rewarded for chaining together reasoning steps etc in a way that results in success (e.g. a known correct result).
The difference with something completely new at inference time, that was not in the training data, or not used by RL training, is that while it will be able to use it to some extent, it has not been taught via RL how to best use in a reasoning chain.
The problem is that LLMs don't really have the generic ability to reason, so they instead need to fake it by either:
1) Fine tuning on reasoning data (very specific)
2) RL training (more generalizable, especially for math/coding)
3) Prompting that encourages "keep on going" "tree of thoughts" exploration where they may discover a reasoning chain largely by luck
Demis Hassabis has talked about combining LLMs with search (cf systems like AlphaGo) which sounds like it might be useful for math research.
RL involves weight updates, but the model is frozen after training - no inference time weight update! An RL-trained model does tend to have generic "reward maximizing" long-term goal behavior at inference time, but it's ability to correctly/fruitfully chain together reasoning steps is much dependent on RL training.
> also i think that it is reasoning exactly like us humans do. Indistinguishable
Yes, it is copying human reasoning so it will appear the same, but the difference is when you don't know what to do/try next - when you are trying to solve a problem that you have never solved before and don't know from experience what to try next. This is when having real/generic ability to reason, not just "reason from memory" matters. This is when things like human curiosity are useful : "I wonder what happens if I try this ..."
It's like that all that knowledge exists as a vague memory. I've found if I ask them specifics about any book I've read (w/o allowing tool calling), when they get it wrong their hallucinating things that are often correct or wrong in subtle ways. Are they're just completely out of pocket.
It depends on what you mean by continual learning. Do you just mean some form of memorization, or something more/different?
I mean it gains the same level of expertise as it does on stuff in training data.
So then you are talking about weight updates, not just memorization, which makes it much harder, and right away really messes with the business model of cloud-based AI where it's the same model being served to everyone.
Even if we had an algorithm to incrementally update weights without catastrophic forgetting (I don't believe we do - but doesn't seem like such a tough problem), then this implies that everyone has their own personalized model, else if you combine all these updates there is no data privacy. Cloud serving also really depends on everyone using the same model so that you can batch requests and not reload weights for each user.
The much more achievable goal, without needing to upend the whole serving model, would be just to implement continual "episodic" memorization (text-only maybe), but even with compaction/consolidation the next question would be how do you retrieve these external memories and get the necessary chunks into context when needed. Some sort of vector store, perhaps? How do you avoid vendor lock-in - perhaps have agents store/retrieve vector-store context chunks from a vendor-agnostic cloud store?
So, even the most basic form of continual-learning-like enhancement becomes tricky. What you'll first see is presumably just enhancements of the vendor-specific memory mechanisms that are already available.
What I would consider as true continual learning, close to what an animal or human does, would require much more extensive architectural and deployment/business model changes - since then we're really talking about more than just an update/recall problem, but rather the whole autonomous agentic loop of predicting/acting/failing/learning/etc with innate traits like curiosity (prediction failure) and boredom to make sure the agent is exposed to learning situations in the first place. At this point you are building an artificial brain, not just an LLM. Some companies such as Google/DeepMind have a more ambitious definition of AGI (more than just an LLM) that is perhaps a step in this direction.
Even with this sort of animal-like continual exploration/learning, you still wouldn't have something that is human level, but at least much closer in terms of ability to learn.
The reason i am asking is about research mathematics. In this application what is really important is continual intuition and theory building, this takes time and trial and error. However this is very fruitful in solving problem. Currently the LLM can do mathematics that is somehow already in their weight very well. See the recent things. But i suppose true research would require continual learning that goes beyond the context window. Thats why i am wondering if this solvable. My intuition is that it is easily solvable, hence we could make a true mathematical continuously building theory to prove things.
For that sort of application then maybe a chunked vector store memory would work if 1M context is too small to hold the entire history? In Terrance Tao's ChatGPT discussion it seemed that the LLM was making good use of the in-context information.
I think for true, especially open-ended, research you really also need things like curiosity, but perhaps a lot can still be done with the same sort of prompting that got the Jacobian result - just tell the model to keep on going, investigate anything that seems interesting/unexplained, etc?
Yes maybe you could scheme it, like think of 10 things you could try, spawn subagents, try them, perhaps one would succseed and yes that think that worked could be placed in a memory system. But there is also value in things that dont work ect. So true continual learning would be really good. I am currently reasearching in mathematics and some things it cant get, i really have to think of them myself. But other things where it is easier to "build" from already existing knowledge works really well. I kinda wanne know how this will continue. It can do things that are "in distribution", but can it build its own new distribution and continue building. Mabye ill try this using a normal RAG system. I would concider it a succsess if it create a new "theory" like 100 pages genuinely new stuff, and then applies it to a problem. But my intuition is this will not work lol
The issue remains the context window. I haven't seem any example of a "memories" system that the agent can update over time without it devolving into slop. The magic of the human brain is that we seem to have decently sophisticated heuristics parsing/saving memories as well as for letting memories that aren't accessed decay.
Yes i agree, but somehow knowledge got put into the LLMs head during train time, but why does it not work during inference time? Too little examples? Do llms know stuff with exactly one occurence in the training data?
What gives LLMs the ability to reason so well about math specifically is the math-specific RL training they are given where they are rewarded for chaining together reasoning steps etc in a way that results in success (e.g. a known correct result).
The difference with something completely new at inference time, that was not in the training data, or not used by RL training, is that while it will be able to use it to some extent, it has not been taught via RL how to best use in a reasoning chain.
The problem is that LLMs don't really have the generic ability to reason, so they instead need to fake it by either:
1) Fine tuning on reasoning data (very specific)
2) RL training (more generalizable, especially for math/coding)
3) Prompting that encourages "keep on going" "tree of thoughts" exploration where they may discover a reasoning chain largely by luck
Demis Hassabis has talked about combining LLMs with search (cf systems like AlphaGo) which sounds like it might be useful for math research.
ok but why no inference time RL? also i think that it is reasoning exactly like us humans do. Indistinguishable
RL involves weight updates, but the model is frozen after training - no inference time weight update! An RL-trained model does tend to have generic "reward maximizing" long-term goal behavior at inference time, but it's ability to correctly/fruitfully chain together reasoning steps is much dependent on RL training.
> also i think that it is reasoning exactly like us humans do. Indistinguishable
Yes, it is copying human reasoning so it will appear the same, but the difference is when you don't know what to do/try next - when you are trying to solve a problem that you have never solved before and don't know from experience what to try next. This is when having real/generic ability to reason, not just "reason from memory" matters. This is when things like human curiosity are useful : "I wonder what happens if I try this ..."
It's like that all that knowledge exists as a vague memory. I've found if I ask them specifics about any book I've read (w/o allowing tool calling), when they get it wrong their hallucinating things that are often correct or wrong in subtle ways. Are they're just completely out of pocket.
my initials = LLM :(