Very excited to see more post-training happening on these open models. Looking forward to trying this one out once it makes its way onto OpenRouter.
Qwen 27B really punches above its weight. 27B is small enough to fit on a Mac Mini / Macbook Pro which is very exciting.
Starting to see more promising examples of companies post-training. From my evals [1], I've tested 3 so far:
1. Fireworks' Ember-1 (postrained on Kimi K3, improved score by 10% in my evals)
2. Particle AI's Particle-1 (fine-tuned on DeepSeek V4.1 Flash, also a 10% improvement and a bit cheaper and faster)
3. Nex AGI's Nex-N2.6 (post-trained on GLM 5.3 Flash. Didn't show an improvement in my eval, and had a couple runaway loops that made it expensive).
With Thinking Machines' western open model release (Inkling Small) a few weeks ago, I expect we'll see more of this, especially out of western companies.
> Very excited to see more post-training happening on these open models.
Same though I wish we had more models as capable as Qwen that are open weight, Google and Microsoft release open weight models, but they don't get as good as Qwen, it might be on purpose but its frustrating. There is Laguna S 2.1 but they don't go as small as Qwen does, but they're heavily focused on programming.
The heavy reliance on eval datasets showcased here and also evidenced by the frontier lab's high payments for such data cast doubt on the claims that AI will become self-improving soon
The frontier labs want you to believe that their models are magically gaining intelligence that generalizes to new areas with each new release. But it’s an illusion: the models are post-trained on a HUGE array of hand-curated datasets for specific tasks, with good enough coverage that it feels general.
It is longitudinal data with LLM responses to human tasks and outcomes showing up later in the stream, you can use hindsight to see how a LLM response fared, humans come back and bring those outcomes to continue iterating.
> This is what the "subscription subsidy" is buying (and penetration).-
I've been thinking about this lately (I work at a business small enough to get the flat rate plans). I think that this is basically the prosumer/free trial approach to AI.
Essentially you give the hobbyists/small businesses an incredible deal, and then hope to profit from their growth (assuming that agentic software development leads to growth).
In some ways, it's reminiscent of early Stripe, and there's a LOT of ex-Stripe people at Anthropic.
Yeah. I can think of several people who use LLMs to write business processes/procedures. Like writing logical code, they are writing the rules and descriptions _of work_. It’s the killer app for small businesses and I guess for managers in corporate firms.
Both OpenAI and Anthropic have setting that allow (at least paid customers) to ensure their data is not used for training. Do you have reason to believe this setting is not what most user believe it is?
> I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.
Everything but the core models seems to be some mixture of slapdash, petty, and reckless. I would absolutely believe they'd train on customer conversations. Where else would those customers go?
I think if you read the legal agreements adversarially, there are a lot of cracks in them the labs can exploit. For example, what about derivative data such as LLM-generated summaries of your sessions? Do the training settings preclude use of that?
I'm not saying to trust the companies here, but there are other reasons to make sure people don't do that.
AI are now tool users. If you give ChatGPT your password, you should expect that instance of ChatGPT to use it. Given the current state of affairs, HuggingFace etc., this will likely go badly wrong and break something.
And of course we don't want vibe coded software written by someone who gives their LLM a hard coded password because they don't know any better.
Because the session transcripts are stored on disk (and optionally in cloud). And if they end up in the chat they could end up as input to other services.
Seriously who the f@&k actually believes that? The mind boggles that people actually say “there’s a setting.” Is it next to the one that says they won’t break the law and infringe copyright to train their models?
Sam Altman, would you actually believe anything he said if he had an incentive to lie to you?
Rieman hypothesis story was not secret. How do you even entertain this stuff for a second? Even if it were in your financial interest to not understand it’s just so very stark.
The AI companies will tell whatever lie helps them and won’t blink when caught and there will be no consequences because we’ve seen it. Repeatedly. There isn’t a grey area…
Even if you believe they honor it, which you shouldn't, all they have to do it make an update and change the setting to a new opt-out version, and then -poof- you've "consented" just long enough for them to feed your entire account history in. I mean how many times do companies have to do these things until people learn?
That depends on the customer. How can you detect the ones that are making smart decisions and guiding the agent correctly towards better outcomes? If you could know that then the problem would already be solved.
Me and a couple of other top engineers have noticed that the latest round of models (everything after Gemini 3.0, GPT 5.6, Fable 5.0) seems to be less intelligent and less capable of solving unusual tasks. For example, Gemini 4 is a major regression vs 3 on being able to transcribe my girlfriend (who has a very thick, rapid, coastal Colombian Spanish dialect).
They are getting much better at the common case. But it feels like we're moving away from AGI and RSI as a tradeoff.
> They are getting much better at the common case. But it feels like we're moving away from AGI and RSI as a tradeoff.
We can but hope.
On the flip side, being good at just maths may solve whatever difference it is that makes all biological brains learn from fewer examples than all machine learning.
> On the flip side, being good at just maths may solve whatever difference it is that makes all biological brains learn from fewer examples than all machine learning.
Will it? Can you help me understand your train of thought here?
> Can you help me understand your train of thought here?
Sure.
Machine learning is a collection of algorithms, i.e. mathematics.
Hypothetically, an AI which is sufficiently good at making novel mathematical breakthroughs may be able to make a better machine learning algorithm. Assuming one exists. Our minds strongly suggest that better does exist, but on the other hand evolution is very good at "cheating".
I mean, AI have already made some improvements to machine learning algorithms, it's just that so far the improvements have been relatively minor, not (so far as is publicly known) fundamental improvements.
> Hypothetically, an AI which is sufficiently good at making novel mathematical breakthroughs may be able to make a better machine learning algorithm. Assuming one exists. Our minds strongly suggest that better does exist, but on the other hand evolution is very good at "cheating".
I think that regarding human intelligence as a product of machine learning misses a lot of what makes us interesting, in that unsupervised ML (which is mostly how humans learn) is very very difficult and we basically have only made progress when we've converted unsupervised problems into supervised ones.
> I mean, AI have already made some improvements to machine learning algorithms, it's just that so far the improvements have been relatively minor, not (so far as is publicly known) fundamental improvements.
Yeah, I'm a little sceptical of these results in the sense that many of them appear to be very niche (which makes sense, given that human mathematicians are pretty good).
More generally, they look like a lot of AI stuff in that they may work, but are absurdly complicated for what they do.
I’ve been suspecting that as a stochastic parrot cannot reason…
Early LLMs couldn’t answer how many R’s in strawberry. Now they can… Perhaps they just recognize the question and know which “helper” to use?
However, I find Claude Opus connects the dots in large codebases in ways that aren’t simple parlour tricks…
But who knows, maybe our own reasoning isn’t all that we think it is. Our ability to spell words isn’t too different - many patterns fuzzily drilled into the head until we can spell most things…
Cars didn't exist 150 years ago, yet we do not need the trillion miles of experience that it took Tesla to reach merely a small geofenced region where a steering wheel is not needed; how much would a human need for the same? A thousand miles as a learner? A hundred?
Ten thousand generations of fire and our jaws still have not fully adapted to cooked food, and yet our minds can learn to cope with it just fine; just one demonstration of why not to pour water on boiling oil can be sufficient.
And our genome has to cover our whole body, cell metabolism upwards, not specifically our brains.
There is some stuff our genome gives us, our minds, but it cannot be much. There isn't room for much in our DNA.
Exactly, it's a frustrating bit of theory. The basic observation is that there's a huge weakness in current ML algorithms: humans are about 1 000 000 TIMES more sample efficient than an LLM. Which means it is possible to train and run a model that's in most ways better than ChatGPT ... on a 386, and just millions of tokens (if you don't mind ~15 years training time), so let's say a first-gen i7 in a month.
And it's not just much more efficient in compute, it's obviously not necessary to collect the data in the first place. Which would be another huge saving (the other thing big ML companies imagine is their moat)
Unfortunately people have been constantly trying to do it since at least 1950 and we're still not anywhere close to understanding any of it.
Now where's that French guy that built a 5G base station out of ... I want to say a VCR? We have need of him.
Maybe not just DNA. "Something needs to speed evolution up" - Denis Noble, Professor of Biology argues against the idea that DNA is instructions and everything else in the body passively takes orders and has no input and no feedback - https://www.youtube.com/watch?v=NAPhBt8VJCM
> argues against the idea that DNA is instructions and everything else in the body passively takes orders and has no input and no feedback
There is feedback between body and DNA; it's at every level from how a foetus gets head and feet in the right place to how it repairs injuries as an adult to how exercise triggers muscle growth.
That isn't doesn't mean there's storgage beyond DNA. There might be some signal molecules which also survive going via sperm and eggs, but they also can't carry much data, and they'd be read by mechanisms generated by the DNA.
But why the idea that there must be distinct 'storage' somewhere for everything? In a Conway's game of life setup which creates gliders, there is no distinct storage which describes how a glider is built and does nothing else - it's in the behaviour of the whole system. When you spin 20 plates or juggle 7 balls, you don't need that many hands nor do you think the total amount of storage is "2 hands" which allows only 4 states; it's a dynamic ongoing task, never stopping, the balls never get stored.
Life isn't a blueprint given to a factory and built from scratch, it's a 4+ billion year continuous unbroken happening, of juggling, of plate-spinning, of people forking and patching a running LISP REPL. Life isn't stopped and started anew every generation just because we talk about individual people being discrete entities and conceived at a specific point. A human egg cell is 3e17 atoms, it isn't only "storage and nutrients", and it is connected to the mother's womb for millions of seconds which is not only "a fuel supply". Single celled organisms can split in half, but that's not how humans replicate. There's a long period of the mother's body 'juggling' the development of the foetus.
From The Company of Biologists[1]:
"""Emerging evidence suggests that .. the placenta secretes a myriad of factors into the maternal and fetal circulations, which appear to be part of an active maternal-placental-fetal crosstalk. However, for most of these factors, their physiological function remains to be determined; examples of factors with known functions include plGF and sFLT1 (Degnes et al., 2022). Both of these proteins regulate angiogenesis ..
Both term (Michelsen et al., 2019) and preterm (Schreiner et al., 2022) human placentas secrete an array of proteins of unknown function into the fetal circulation. Some of these proteins are predicted to be involved in the development of the brain, heart and lung and we speculate that, when born prematurely, infants are deprived of these important developmental regulating placental proteins, which may contribute to their poor outcomes (Schreiner et al., 2022)."""
Among other things like passive immunity from the mother's antibodies, maternal thyroid hormone, cortisol, exosomes carrying microRNA.
> But why the idea that there must be distinct 'storage' somewhere for everything?
If it is not stored, it isn't a thing which can be passed on from one generation to the next.
> In a Conway's game of life setup which creates gliders, there is no distinct storage which describes how a glider is built and does nothing else - it's in the behaviour of the whole system.
Conway's game of life is a discrete field with discrete values in each location; the entire game area is storage, and in that space both empty and occupied hold meaning.
> When you spin 20 plates or juggle 7 balls, you don't need that many hands nor do you think the total amount of storage is "2 hands" which allows only 4 states; it's a dynamic ongoing task, never stopping, the balls never get stored.
The balls exist in space. The space isn't "your hand", but it does exist.
> Life isn't a blueprint given to a factory and built from scratch, it's a 4+ billion year continuous unbroken happening, of juggling, of plate-spinning, of people forking and patching a running LISP REPL.
This metaphor is so mixed I am unable to disentangle it.
> Life isn't stopped and started anew every generation just because we talk about individual people being discrete entities and conceived at a specific point.
No, it's more like a git repo getting forked at every cell division, with some of the patches triggering automation scripts that cause the same patch to be forwarded to other repos (this is a virus analogy).
> A human egg cell is 3e17 atoms, it isn't only "storage and nutrients", and it is connected to the mother's womb for millions of seconds which is not only "a fuel supply". Single celled organisms can split in half, but that's not how humans replicate. There's a long period of the mother's body 'juggling' the development of the foetus.
Nobody is claiming the things you are arguing against, except for some using reductively broad definitions of the word "nutrients" to cover literally all the things a cell does.
> However, for most of these factors, their physiological function remains to be determined; examples of factors with known functions include plGF and sFLT1 (Degnes et al., 2022). Both of these proteins regulate angiogenesis
> [etc.]
These are examples of what I have in mind when I said… well, basically the entirety of the previous comment given how short it was.
> "Nobody is claiming the things you are arguing against"
You said "that doesn't mean there's storgage beyond DNA" and that is the thing I'm arguing against - there is or can be information storage beyond DNA, and information transmission down the generations in ways other than DNA. (This is against the person I first replied to who lolled at the idea that evolution could have encoded much into us because DNA can't hold much). To which you apparently disagree, except that all the rest of the things you say seem to be in agreement, which is confusing. You said "If it is not stored, it isn't a thing which can be passed on from one generation to the next" - but that wasn't my claim, I don't say information is not stored anywhere, I said it's not only stored in DNA.
Where else could it be stored, well I compared to Conway's Game of Life which you apparently disagree with, saying "the entire game area is storage"; that's the comparison I was making - the entire human body is information storage. Like the movement of Mercury is the storage in a Mercury delay line storage device, the movement and interactions of the living cells in the body can also be storing information. Killing a cell, grinding it up, finding the DNA as a distinct thing, and saying "that, there, that's the limit of information storage" is incomplete.
> "If it is not stored, it isn't a thing which can be passed on from one generation to the next. ... These are examples of what I have in mind when I said… well, basically the entirety of the previous comment given how short it was."
You apparently disagree that there's more opportunities for information passed down through the generations than just DNA, and you refute my link describing some of the ongoing communications during foetus development that are not just the foetal cells reading their DNA ... and your refutation is that those are the things you had in mind when you said earlier that nothing was passed down except through DNA, before then claiming that nobody took the position you took? I am unable to disentangle your position at all here. It looks like you're completely agreeing except trying to say that you aren't?
Every company on earth lies about future growth expectations to get investor money I have no idea why anyone would take it seriously.RSI/AGI claims are just the same thing as always we can see the path of improvement pretty clearly and it very obviously is not some constant accelerating thing. There is occasional jumps but even when there are they aren't universal. Things get better at coding but worse at writing, etc.
If you just look at what people have actually released and ignore marketing the history of releases has been pretty consistent in occasional jumps up then gradual improvement we haven't seen anything close breaking the normal curve that has frankly already flattened out pretty significantly.
The main improvements, and this is possibly where you can say some kind of RSI is a thing (but it's not self) is thinks like cheap/faster models getting as good as the best models. You can have fable train haiku and probably get operating at opus level fairly quickly but you can't have fable train fable and make it 10x better. Frankly if we had something like Fable level at haiku speeds (or faster) that is probably all we'd need from AI to get like all the benefits and I don't really expect it to go much further than that.
I’ve gotten recruiting emails to be a human grader of Claude coded projects. 6 month contract at $150/hour 40hrs/week. Not insanely expansive but also not cheap.
I always wonder what you lose when you start maxing out some specific benchmark. It is really hard to do this in a balanced way and I would love if everybody that did these kind of efforts would take the time to also list the before/afters of a number of other benchmarks that they did not use in their training.
Good point! First of all, none of the benchmark tasks were used in training, though the training data was designed to improve capabilities on tasks like TB4.1. If you consider TB4.1 the main target, the blog actually shows before/after results on several other benchmarks, with no regressions and even some improvements.
Yes, I've seen that, but I wonder what is missing because I find it hard to believe that you can just do some RL and then end up with a model that is better across the board. I would find it far more believable if you outperformed on the benchmark that was your goal (assuming you selected training data to give you an edge there) and underperformed on things that you were not optimizing for. But since it all moves in one direction (better) I feel there must be some kind of unknown at play here. Even so, task specific AI is definitely the future, you don't hire a plumber to do your bookkeeping either...
I agree with you and I think, personally, that the heavy post-training like here makes the model more capable in coding but less capable for certain unbenchmarkable tasks, like prose or interpreting prompts correctly. I've been noticing that small models have been gradually being pushed towards programming and are slowly becoming _worse_ at some non-programming tasks, not better; e.g. new Deepseek models appear to be much worse at interpreting metaphors and overall feel more autistic than previous ones.
That mirrors my experience exactly. I keep a couple of older models (and much larger ones as well) for when I want to do stuff that is unrelated to computer programming because the models that are really good at that tend to fare less well at other stuff. I suspect that there is a boundary where maximizing benchmarks for coding at the expense of general knowledge is going to give you a model that is really fast and good but 'narrow' and that this may also lead to loss of performance when applied to coding tasks that are less well represented in the training set (such as, for instance, DSP code). This is a variation on 'the bitter lesson', short term you can make gains like this but long term leveraging compute and data will give you something better. I'm really happy I didn't waste half a life on engineered AI because I would be really quite miserable if I had.
As electronics/software/systems engineer with brain that's on the spectrum (ADHD, no autism diagnosis ... yet).
The "Worse at interpreting metaphors and feel more autistic" to me sounds representative of the IT field?
Obviously - not all of us are on the spectrum. But my (biased) observations suggest that occurance of neurodiversity/spectrum is higher in IT (or more broadly STEM) than in other fields.
As in - perhaps that (different neuro-wiring, different way of looking at things...etc) is what makes one a better as programming and things of similar nature?
Why do you find it hard to believe that you can do some RL and improve across the board (albeit less strongly on other semi-related tasks than your objective)? That's the idea behind generalization: it's not perfect but you get a little bit of cross-task generalization, so if you just keep spamming more RL environments you can hill-climb to AGI. That's what the labs are doing!
If you carefully curate the data, plus the original model isn't saturated for RL post-training, it's possible to improve targeted benchmarks without regression on other metrics. This experiment is done with terminal bench 4.1 as a target, and the regression on other benchmarks has been carefully controlled through data + infra design.
Ok, so the obvious question then is: this seems (I'm not saying it is, it just looks like a small thing compared to training up a model from scratch) a relatively trivial undertaking for the authors of the original model, if there was this much left in the tank why didn't they do something similar? Or is there some step here that only this team can perform? Or is it simply a matter of economics?
You'd probably end up spending more on pcie h200s. They're not that much cheaper than 8x sxm nodes and they only have 4 way nvlink. At the scale these guys were training at you'd probably want at a minimum an 8x nvlinked node for the trainer, even though you could technically fit the full training and optimizer states on a 4x h200 node.
Savings would likely be larger with magic beans such as floobarbaz, no?
But seriously. If there were a cheaper and reliable way, I'm sure the authors would have mentioned it instead of H200 SXM modules. They're trying to brag about cheap training.
Maybe you wanted to use an opportunity to shill tranium.
>But seriously. If there were a cheaper and reliable way, I'm sure the authors would have mentioned it instead of H200 SXM modules. They're trying to brag about cheap training.
shit reasoning.
the big dogs use all sorts of weird (usually better) shit because they have access to it.
the opposite is also true. The H200 SXMs are available, some custom TPU purpose built for your needs isn't.
or , in other words, "if there were they'd have done it!" begs us to believe the concept that everyone out there is doing things as optimally as possible, with perfect access.
It isn't if you train the quantized model with a quantized K/V and cache. That's my approach and usually takes around a day for 100k reasoning traces (including abliteration).
Also this is on consumer hardware, meaning 96GB CPU RAM and 48GB GPU VRAM, using an optimized Vulkan concurrent shader pipeline. So costs for the machine are around 3-5k depending on RAM crisis.
Rather than waiting, ask your assistant to quantize for you. It could be not the best quantization possible but it’s relatively quick, and I had an overall positive experience.
My bad, I haven’t really looked yet (and haven’t asked any assistants), thought it’s BF16 safetensors as usual, and for those I used to quantize to ggufs with just a couple commands.
One pet project of mine would be - if I had the resources, of course - to get a simple but decent model and post train it intensely for just one language and one task.
For example doing backend projects using C# or Go.
How did you guys set up the RL topology? Were you guys doing individual RL nodes (4xh200 rollout/4xh200 train) or did you have dedicated trainer and rollout nodes?
Since some talented people with more modest means were able to do such a job, I wonder why Alibaba didn't do something like this when they trained the model.
They left a lot of performance on the table.
Which makes me wonder if such kind of optimizations are possible for other models like Deepseek v4.1 Flash 1017.
3.8 27B is an interesting choice. on the face of it, doing RL on it feels like refrying twice-cooked pork then stuffing it in the microwave for good measure. the fact that even one benchmark (tb4.1) increased more than marginally is kind of miraculous. would be interesting to see where "miles" would take eg. 3.8 flash next, which is already impressive despite being an under-trained architectural preview.
I had the same thought. Their described process of identifying gaps and post-training on high-quality inputs specifically designed to close those gaps is very interesting. I wonder if that's possibly the secret ingredient in the frontier lab sauce.
> Their described process of identifying gaps and post-training on high-quality inputs specifically designed to close those gaps is very interesting. I wonder if that's possibly the secret ingredient in the frontier lab sauce.
Unfortunately, there's no such thing as a free lunch. You'd expect the models to regress on other tasks when you tune them for more specifics.
Like, I guess if the models were wildly inefficient after pre-training, then this might not be the case but I'd be really surprised if there weren't strong tradeoffs here.
I'm not sure where you see a free lunch here considering the amount of work and resources that they claim to have put into designing their post-training process? As well, Qwen 3.8 27b itself is the result of post-training 3.6 and is a consistent improvement across the board, which does show post-training can in fact potentially yield such.
Only question I see is whether there's really a lot of post-training runway left in 3.8. I would have said no, considering the baffling amount of capability it already packs, but the linked article does exhibit improvements in unrelated benchmarks they didn't post-train for, which suggests to me I may be wrong about that.
> I'm not sure where you see a free lunch here considering the amount of work and resources that they claim to have put into designing their post-training process? As well, Qwen 3.8 27b itself is the result of post-training 3.6 and is a consistent improvement across the board, which does show post-training can in fact potentially yield such.
OK, I probably wasn't clear enough.
Assuming that each weight is actually useful, then shifting the weights towards one specific part of the space is likely to move it away from points in the space which were more useful for each task.
That being said, it may be that each weight is not actually useful, so maybe this is possible (which makes conceptual sense given how absurdly large even the small LLMs are).
> Only question I see is whether there's really a lot of post-training runway left in 3.8. I would have said no, considering the baffling amount of capability it already packs, but the linked article does exhibit improvements in unrelated benchmarks they didn't post-train for, which suggests to me I may be wrong about that.
The choice of benchmarks matters a lot here, but yeah it's possible that Qwen (and presumably other models) have a lot of unnecessary weights so maybe it can be improved on basically everything.
Ah! That does make sense. Thank you for the clarification.
A couple thoughts, then.
Is the usefulness of a weight supposed to be a binary, useful/not useful? If not, how do we know whether all the weights that ship in Qwen 3.8 27b are already maximally useful?
Second, can there be gaps that affect all purposes negatively (conditions that trigger an endless loop, for instance, goodness knows small models are prone to such) and whose closure therefore lifts capability across all purposes?
(Not trying to be contrarian or playing devil's advocate -- the idea a model can't possibly be improved in aggregate, at least up to a certain point, just stands firmly at odds with my intuition of how neural networks work.)
I look forward to testing this against my private eval set to determine if it's benchmaxxing or actual improvement based on just 1200 tasks. downloading it now
Daisy emitted 1.92M tokens and Swift 1.5 ... didn't. 1.04M there
given multiple advantages in terms of kv cache size/quant, decode speed, time, and model quant: DaisyRose marginally beat out another base Qwen finetune running in half the VRAM.
where's the 35b-a3b variant they mention to have also trained? much more interesting for consumer hw. and they should've started with ornith 1.5, not qwen 3.6. we don't need 10 models that's each about 10-20% better than qwen 3.6 - we need a model that accumulates the gains.
Very excited to see more post-training happening on these open models. Looking forward to trying this one out once it makes its way onto OpenRouter.
Qwen 27B really punches above its weight. 27B is small enough to fit on a Mac Mini / Macbook Pro which is very exciting.
Starting to see more promising examples of companies post-training. From my evals [1], I've tested 3 so far:
1. Fireworks' Ember-1 (postrained on Kimi K3, improved score by 10% in my evals)
2. Particle AI's Particle-1 (fine-tuned on DeepSeek V4.1 Flash, also a 10% improvement and a bit cheaper and faster)
3. Nex AGI's Nex-N2.6 (post-trained on GLM 5.3 Flash. Didn't show an improvement in my eval, and had a couple runaway loops that made it expensive).
With Thinking Machines' western open model release (Inkling Small) a few weeks ago, I expect we'll see more of this, especially out of western companies.
[1] https://plotly.com/blog/nex-n2-6-plotly-data-analytics-bench...
> Very excited to see more post-training happening on these open models.
Same though I wish we had more models as capable as Qwen that are open weight, Google and Microsoft release open weight models, but they don't get as good as Qwen, it might be on purpose but its frustrating. There is Laguna S 2.1 but they don't go as small as Qwen does, but they're heavily focused on programming.
The heavy reliance on eval datasets showcased here and also evidenced by the frontier lab's high payments for such data cast doubt on the claims that AI will become self-improving soon
The frontier labs want you to believe that their models are magically gaining intelligence that generalizes to new areas with each new release. But it’s an illusion: the models are post-trained on a HUGE array of hand-curated datasets for specific tasks, with good enough coverage that it feels general.
The frontier labs are using all their customers' logs to train. It's the best training set there is.
I don't think your data is as valuable as you think it is.
I would not want a model trained on user conversations.
It is longitudinal data with LLM responses to human tasks and outcomes showing up later in the stream, you can use hindsight to see how a LLM response fared, humans come back and bring those outcomes to continue iterating.
it's pretty valuable, and there's a huge demand for it from data labs depending on the kind of data. Agent traces are also pretty valuable
This is what the "subscription subsidy" is buying (and penetration).-
> This is what the "subscription subsidy" is buying (and penetration).-
I've been thinking about this lately (I work at a business small enough to get the flat rate plans). I think that this is basically the prosumer/free trial approach to AI.
Essentially you give the hobbyists/small businesses an incredible deal, and then hope to profit from their growth (assuming that agentic software development leads to growth).
In some ways, it's reminiscent of early Stripe, and there's a LOT of ex-Stripe people at Anthropic.
> and there's a LOT of ex-Stripe people at Anthropic.
Interesting fact.-
When I worked there, it was a running joke how many of the leavers were going to Anthropic.
It's really not surprising, given that Daniela worked at Stripe for ages.
Well, if nothing else, at least our Agent overlords will get payments right :)
yup. They make money off of ads in their free/low cost plans, and train on your data for the pro subs, and charge you absurd API prices.
So you’re telling me you haven’t uploaded any internal company documents or discussed any of your company projects?
Why do you think they’re getting into consulting, right after ingesting everyone’s statements of work?
Yeah. I can think of several people who use LLMs to write business processes/procedures. Like writing logical code, they are writing the rules and descriptions _of work_. It’s the killer app for small businesses and I guess for managers in corporate firms.
Both OpenAI and Anthropic have setting that allow (at least paid customers) to ensure their data is not used for training. Do you have reason to believe this setting is not what most user believe it is?
Anthropic used to charge you API rates if it saw a HERMES.md in the repo you were working on, even if you had a subscription: https://github.com/anthropics/claude-code/issues/53262
Another commenter here: https://news.ycombinator.com/item?id=49502774
> I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.
Everything but the core models seems to be some mixture of slapdash, petty, and reckless. I would absolutely believe they'd train on customer conversations. Where else would those customers go?
And it's too late for me to edit my first post, but there was also the ripping-off of Verinf ( https://news.ycombinator.com/item?id=50370121 ), and Mononoke ( https://sunshowers.io/posts/anthropic-mononoke-2/ ) too.
I think if you read the legal agreements adversarially, there are a lot of cracks in them the labs can exploit. For example, what about derivative data such as LLM-generated summaries of your sessions? Do the training settings preclude use of that?
Both tell you not to include password/tokens in chat. Why would it matter if they weren't training
I'm not saying to trust the companies here, but there are other reasons to make sure people don't do that.
AI are now tool users. If you give ChatGPT your password, you should expect that instance of ChatGPT to use it. Given the current state of affairs, HuggingFace etc., this will likely go badly wrong and break something.
And of course we don't want vibe coded software written by someone who gives their LLM a hard coded password because they don't know any better.
Because the session transcripts are stored on disk (and optionally in cloud). And if they end up in the chat they could end up as input to other services.
You don't want your password to be used as an input to some tool call do you?
Seriously who the f@&k actually believes that? The mind boggles that people actually say “there’s a setting.” Is it next to the one that says they won’t break the law and infringe copyright to train their models?
Sam Altman, would you actually believe anything he said if he had an incentive to lie to you?
Rieman hypothesis story was not secret. How do you even entertain this stuff for a second? Even if it were in your financial interest to not understand it’s just so very stark.
The AI companies will tell whatever lie helps them and won’t blink when caught and there will be no consequences because we’ve seen it. Repeatedly. There isn’t a grey area…
Dude.
He’s also not necessarily wrong. The guy is a rat and everyone here knows it.
Yeah, but he was being a dick.
Even if you believe they honor it, which you shouldn't, all they have to do it make an update and change the setting to a new opt-out version, and then -poof- you've "consented" just long enough for them to feed your entire account history in. I mean how many times do companies have to do these things until people learn?
That depends on the customer. How can you detect the ones that are making smart decisions and guiding the agent correctly towards better outcomes? If you could know that then the problem would already be solved.
Time to do all business in cockney ryming slang!
All the hours that went into watching "Only Fools And Horses" will finally pay off.
Me and a couple of other top engineers have noticed that the latest round of models (everything after Gemini 3.0, GPT 5.6, Fable 5.0) seems to be less intelligent and less capable of solving unusual tasks. For example, Gemini 4 is a major regression vs 3 on being able to transcribe my girlfriend (who has a very thick, rapid, coastal Colombian Spanish dialect).
They are getting much better at the common case. But it feels like we're moving away from AGI and RSI as a tradeoff.
> They are getting much better at the common case. But it feels like we're moving away from AGI and RSI as a tradeoff.
We can but hope.
On the flip side, being good at just maths may solve whatever difference it is that makes all biological brains learn from fewer examples than all machine learning.
> On the flip side, being good at just maths may solve whatever difference it is that makes all biological brains learn from fewer examples than all machine learning.
Will it? Can you help me understand your train of thought here?
> Will it?
May solve, not will solve.
> Can you help me understand your train of thought here?
Sure.
Machine learning is a collection of algorithms, i.e. mathematics.
Hypothetically, an AI which is sufficiently good at making novel mathematical breakthroughs may be able to make a better machine learning algorithm. Assuming one exists. Our minds strongly suggest that better does exist, but on the other hand evolution is very good at "cheating".
I mean, AI have already made some improvements to machine learning algorithms, it's just that so far the improvements have been relatively minor, not (so far as is publicly known) fundamental improvements.
OK, that makes sense.
> Hypothetically, an AI which is sufficiently good at making novel mathematical breakthroughs may be able to make a better machine learning algorithm. Assuming one exists. Our minds strongly suggest that better does exist, but on the other hand evolution is very good at "cheating".
I think that regarding human intelligence as a product of machine learning misses a lot of what makes us interesting, in that unsupervised ML (which is mostly how humans learn) is very very difficult and we basically have only made progress when we've converted unsupervised problems into supervised ones.
> I mean, AI have already made some improvements to machine learning algorithms, it's just that so far the improvements have been relatively minor, not (so far as is publicly known) fundamental improvements.
Yeah, I'm a little sceptical of these results in the sense that many of them appear to be very niche (which makes sense, given that human mathematicians are pretty good).
More generally, they look like a lot of AI stuff in that they may work, but are absurdly complicated for what they do.
A large enough array of specific datasets means the model does have general coverage. The question is how close are we to that?
I’ve been suspecting that as a stochastic parrot cannot reason…
Early LLMs couldn’t answer how many R’s in strawberry. Now they can… Perhaps they just recognize the question and know which “helper” to use?
However, I find Claude Opus connects the dots in large codebases in ways that aren’t simple parlour tricks…
But who knows, maybe our own reasoning isn’t all that we think it is. Our ability to spell words isn’t too different - many patterns fuzzily drilled into the head until we can spell most things…
The strawberry question was never a strong indicator of intelligence, and now the answer is in the training data. This Twitter dialogue illustrates:
> @tautologer: asking an llm how many letters are in a word is like asking a person what wavelength of light a color is
> @Voltlighter: I could have memorized that the frequency of green light is 550 Hz just like an LLM could memorize that strawberry has three Rs
> @tautologer: exactly
https://x.com/tautologer/status/2051014130593591620
Wikipedia explains how an LLM's tokenizer works: https://en.wikipedia.org/wiki/Large_language_model#Tokenizat....
How is that different from you?
Humans generalize much more quickly than models? We don't require millions of curated examples to learn a concept?
Because there is four billion years of evolutionary pre-training.
Cars didn't exist 150 years ago, yet we do not need the trillion miles of experience that it took Tesla to reach merely a small geofenced region where a steering wheel is not needed; how much would a human need for the same? A thousand miles as a learner? A hundred?
Ten thousand generations of fire and our jaws still have not fully adapted to cooked food, and yet our minds can learn to cope with it just fine; just one demonstration of why not to pour water on boiling oil can be sufficient.
And our genome has to cover our whole body, cell metabolism upwards, not specifically our brains.
There is some stuff our genome gives us, our minds, but it cannot be much. There isn't room for much in our DNA.
Exactly, it's a frustrating bit of theory. The basic observation is that there's a huge weakness in current ML algorithms: humans are about 1 000 000 TIMES more sample efficient than an LLM. Which means it is possible to train and run a model that's in most ways better than ChatGPT ... on a 386, and just millions of tokens (if you don't mind ~15 years training time), so let's say a first-gen i7 in a month.
And it's not just much more efficient in compute, it's obviously not necessary to collect the data in the first place. Which would be another huge saving (the other thing big ML companies imagine is their moat)
Unfortunately people have been constantly trying to do it since at least 1950 and we're still not anywhere close to understanding any of it.
Now where's that French guy that built a 5G base station out of ... I want to say a VCR? We have need of him.
Instead of pre-training, I'd say it's more like a better architecture, training regime, higher fidelity data, and perhaps better initial "weights".
All of that stuff was still trained by evolution, though.
Hmm perhaps I learned to drive in the womb from my moms epigenetic driving genes
Evolution (or, nature) developed much of the architecture that allows you to learn to drive, such as the brain and body.
Monkeys couldnt be taught how to do it. Thus the analogy stands. Our brain architecture allows for more precise instructional retention
I present to you: Orangutan Driving Golf Cart
https://youtu.be/RZ_0ImDYrPY
In ~8G weights of DNA?
Lol. That's less than 1 milliFable at 1-bit quantization.
Maybe not just DNA. "Something needs to speed evolution up" - Denis Noble, Professor of Biology argues against the idea that DNA is instructions and everything else in the body passively takes orders and has no input and no feedback - https://www.youtube.com/watch?v=NAPhBt8VJCM
> argues against the idea that DNA is instructions and everything else in the body passively takes orders and has no input and no feedback
There is feedback between body and DNA; it's at every level from how a foetus gets head and feet in the right place to how it repairs injuries as an adult to how exercise triggers muscle growth.
That isn't doesn't mean there's storgage beyond DNA. There might be some signal molecules which also survive going via sperm and eggs, but they also can't carry much data, and they'd be read by mechanisms generated by the DNA.
https://en.wikipedia.org/wiki/Denis_Noble
But why the idea that there must be distinct 'storage' somewhere for everything? In a Conway's game of life setup which creates gliders, there is no distinct storage which describes how a glider is built and does nothing else - it's in the behaviour of the whole system. When you spin 20 plates or juggle 7 balls, you don't need that many hands nor do you think the total amount of storage is "2 hands" which allows only 4 states; it's a dynamic ongoing task, never stopping, the balls never get stored.
Life isn't a blueprint given to a factory and built from scratch, it's a 4+ billion year continuous unbroken happening, of juggling, of plate-spinning, of people forking and patching a running LISP REPL. Life isn't stopped and started anew every generation just because we talk about individual people being discrete entities and conceived at a specific point. A human egg cell is 3e17 atoms, it isn't only "storage and nutrients", and it is connected to the mother's womb for millions of seconds which is not only "a fuel supply". Single celled organisms can split in half, but that's not how humans replicate. There's a long period of the mother's body 'juggling' the development of the foetus.
From The Company of Biologists[1]:
"""Emerging evidence suggests that .. the placenta secretes a myriad of factors into the maternal and fetal circulations, which appear to be part of an active maternal-placental-fetal crosstalk. However, for most of these factors, their physiological function remains to be determined; examples of factors with known functions include plGF and sFLT1 (Degnes et al., 2022). Both of these proteins regulate angiogenesis ..
Both term (Michelsen et al., 2019) and preterm (Schreiner et al., 2022) human placentas secrete an array of proteins of unknown function into the fetal circulation. Some of these proteins are predicted to be involved in the development of the brain, heart and lung and we speculate that, when born prematurely, infants are deprived of these important developmental regulating placental proteins, which may contribute to their poor outcomes (Schreiner et al., 2022)."""
Among other things like passive immunity from the mother's antibodies, maternal thyroid hormone, cortisol, exosomes carrying microRNA.
[1] https://journals.biologists.com/dev/article/150/20/dev202088...
> But why the idea that there must be distinct 'storage' somewhere for everything?
If it is not stored, it isn't a thing which can be passed on from one generation to the next.
> In a Conway's game of life setup which creates gliders, there is no distinct storage which describes how a glider is built and does nothing else - it's in the behaviour of the whole system.
Conway's game of life is a discrete field with discrete values in each location; the entire game area is storage, and in that space both empty and occupied hold meaning.
> When you spin 20 plates or juggle 7 balls, you don't need that many hands nor do you think the total amount of storage is "2 hands" which allows only 4 states; it's a dynamic ongoing task, never stopping, the balls never get stored.
The balls exist in space. The space isn't "your hand", but it does exist.
> Life isn't a blueprint given to a factory and built from scratch, it's a 4+ billion year continuous unbroken happening, of juggling, of plate-spinning, of people forking and patching a running LISP REPL.
This metaphor is so mixed I am unable to disentangle it.
> Life isn't stopped and started anew every generation just because we talk about individual people being discrete entities and conceived at a specific point.
No, it's more like a git repo getting forked at every cell division, with some of the patches triggering automation scripts that cause the same patch to be forwarded to other repos (this is a virus analogy).
> A human egg cell is 3e17 atoms, it isn't only "storage and nutrients", and it is connected to the mother's womb for millions of seconds which is not only "a fuel supply". Single celled organisms can split in half, but that's not how humans replicate. There's a long period of the mother's body 'juggling' the development of the foetus.
Nobody is claiming the things you are arguing against, except for some using reductively broad definitions of the word "nutrients" to cover literally all the things a cell does.
> However, for most of these factors, their physiological function remains to be determined; examples of factors with known functions include plGF and sFLT1 (Degnes et al., 2022). Both of these proteins regulate angiogenesis
> [etc.]
These are examples of what I have in mind when I said… well, basically the entirety of the previous comment given how short it was.
plGF: https://en.wikipedia.org/wiki/Placental_growth_factor
sFLT1: https://en.wikipedia.org/wiki/Soluble_fms-like_tyrosine_kina...
> "Nobody is claiming the things you are arguing against"
You said "that doesn't mean there's storgage beyond DNA" and that is the thing I'm arguing against - there is or can be information storage beyond DNA, and information transmission down the generations in ways other than DNA. (This is against the person I first replied to who lolled at the idea that evolution could have encoded much into us because DNA can't hold much). To which you apparently disagree, except that all the rest of the things you say seem to be in agreement, which is confusing. You said "If it is not stored, it isn't a thing which can be passed on from one generation to the next" - but that wasn't my claim, I don't say information is not stored anywhere, I said it's not only stored in DNA.
Where else could it be stored, well I compared to Conway's Game of Life which you apparently disagree with, saying "the entire game area is storage"; that's the comparison I was making - the entire human body is information storage. Like the movement of Mercury is the storage in a Mercury delay line storage device, the movement and interactions of the living cells in the body can also be storing information. Killing a cell, grinding it up, finding the DNA as a distinct thing, and saying "that, there, that's the limit of information storage" is incomplete.
> "If it is not stored, it isn't a thing which can be passed on from one generation to the next. ... These are examples of what I have in mind when I said… well, basically the entirety of the previous comment given how short it was."
You apparently disagree that there's more opportunities for information passed down through the generations than just DNA, and you refute my link describing some of the ongoing communications during foetus development that are not just the foetal cells reading their DNA ... and your refutation is that those are the things you had in mind when you said earlier that nothing was passed down except through DNA, before then claiming that nobody took the position you took? I am unable to disentangle your position at all here. It looks like you're completely agreeing except trying to say that you aren't?
this comment displays a total lack of understanding of both genetics as well as basic information theory
This comment displays a total lack of understanding of the point, in your desperation to post a useless putdown.
you are further beclowning yourself
Every company on earth lies about future growth expectations to get investor money I have no idea why anyone would take it seriously.RSI/AGI claims are just the same thing as always we can see the path of improvement pretty clearly and it very obviously is not some constant accelerating thing. There is occasional jumps but even when there are they aren't universal. Things get better at coding but worse at writing, etc. If you just look at what people have actually released and ignore marketing the history of releases has been pretty consistent in occasional jumps up then gradual improvement we haven't seen anything close breaking the normal curve that has frankly already flattened out pretty significantly.
The main improvements, and this is possibly where you can say some kind of RSI is a thing (but it's not self) is thinks like cheap/faster models getting as good as the best models. You can have fable train haiku and probably get operating at opus level fairly quickly but you can't have fable train fable and make it 10x better. Frankly if we had something like Fable level at haiku speeds (or faster) that is probably all we'd need from AI to get like all the benefits and I don't really expect it to go much further than that.
Curious, where did you see the numbers on how much frontier labs are paying for eval data?
https://getlatka.com/companies/mercor
I’ve gotten recruiting emails to be a human grader of Claude coded projects. 6 month contract at $150/hour 40hrs/week. Not insanely expansive but also not cheap.
>doubt on the claims that AI will become self-improving soon
It already happened. https://github.com/karpathy/autoresearch
Sabi-noema also looks like a close competitor in the same space. Built from qwen3.8 and made better.
https://huggingface.co/sabicap/sabi-noema
Also HF model here: https://huggingface.co/RadixArk/DaisyRose-27B
I always wonder what you lose when you start maxing out some specific benchmark. It is really hard to do this in a balanced way and I would love if everybody that did these kind of efforts would take the time to also list the before/afters of a number of other benchmarks that they did not use in their training.
Good point! First of all, none of the benchmark tasks were used in training, though the training data was designed to improve capabilities on tasks like TB4.1. If you consider TB4.1 the main target, the blog actually shows before/after results on several other benchmarks, with no regressions and even some improvements.
Yes, I've seen that, but I wonder what is missing because I find it hard to believe that you can just do some RL and then end up with a model that is better across the board. I would find it far more believable if you outperformed on the benchmark that was your goal (assuming you selected training data to give you an edge there) and underperformed on things that you were not optimizing for. But since it all moves in one direction (better) I feel there must be some kind of unknown at play here. Even so, task specific AI is definitely the future, you don't hire a plumber to do your bookkeeping either...
I agree with you and I think, personally, that the heavy post-training like here makes the model more capable in coding but less capable for certain unbenchmarkable tasks, like prose or interpreting prompts correctly. I've been noticing that small models have been gradually being pushed towards programming and are slowly becoming _worse_ at some non-programming tasks, not better; e.g. new Deepseek models appear to be much worse at interpreting metaphors and overall feel more autistic than previous ones.
That mirrors my experience exactly. I keep a couple of older models (and much larger ones as well) for when I want to do stuff that is unrelated to computer programming because the models that are really good at that tend to fare less well at other stuff. I suspect that there is a boundary where maximizing benchmarks for coding at the expense of general knowledge is going to give you a model that is really fast and good but 'narrow' and that this may also lead to loss of performance when applied to coding tasks that are less well represented in the training set (such as, for instance, DSP code). This is a variation on 'the bitter lesson', short term you can make gains like this but long term leveraging compute and data will give you something better. I'm really happy I didn't waste half a life on engineered AI because I would be really quite miserable if I had.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
As electronics/software/systems engineer with brain that's on the spectrum (ADHD, no autism diagnosis ... yet).
The "Worse at interpreting metaphors and feel more autistic" to me sounds representative of the IT field?
Obviously - not all of us are on the spectrum. But my (biased) observations suggest that occurance of neurodiversity/spectrum is higher in IT (or more broadly STEM) than in other fields.
As in - perhaps that (different neuro-wiring, different way of looking at things...etc) is what makes one a better as programming and things of similar nature?
Why do you find it hard to believe that you can do some RL and improve across the board (albeit less strongly on other semi-related tasks than your objective)? That's the idea behind generalization: it's not perfect but you get a little bit of cross-task generalization, so if you just keep spamming more RL environments you can hill-climb to AGI. That's what the labs are doing!
You can hill climb but you will (or at least, I think you will!) lose generality.
And no, that won't get you to AGI, that is a completely different goal.
If you carefully curate the data, plus the original model isn't saturated for RL post-training, it's possible to improve targeted benchmarks without regression on other metrics. This experiment is done with terminal bench 4.1 as a target, and the regression on other benchmarks has been carefully controlled through data + infra design.
Ok, so the obvious question then is: this seems (I'm not saying it is, it just looks like a small thing compared to training up a model from scratch) a relatively trivial undertaking for the authors of the original model, if there was this much left in the tank why didn't they do something similar? Or is there some step here that only this team can perform? Or is it simply a matter of economics?
Check the article. It’s the first graphic that has other benchmarks.
Yes, they all go up. I am wondering what was lost.
FWIW that is about $331,000 on low-onboarding-grief clouds (e.g. vast.ai) right now using the SXM variants they used.
You might be able to shave that down using a different flavor of H200 (like PCIe) but it would take more hours, so benefits are unclear.
I ran the post’s 63,000 GPU-hours against today’s H200 SXM on-demand listings: – cheapest 8×H200 node ($3.49/GPU-hr): ~$220k – median of the 20 providers selling 8× nodes ($6.25): ~$394k – AWS ($7.91): ~$498k; Azure/GCP ($10.60): ~$668k
$331k is right on the money. But there's a big gap in price depending on provider.
source: I work on computeprices.com.
You'd probably end up spending more on pcie h200s. They're not that much cheaper than 8x sxm nodes and they only have 4 way nvlink. At the scale these guys were training at you'd probably want at a minimum an 8x nvlinked node for the trainer, even though you could technically fit the full training and optimizer states on a 4x h200 node.
Savings would likely be larger with alternative accelerators such as trainium, no?
Savings would likely be larger with magic beans such as floobarbaz, no?
But seriously. If there were a cheaper and reliable way, I'm sure the authors would have mentioned it instead of H200 SXM modules. They're trying to brag about cheap training.
Maybe you wanted to use an opportunity to shill tranium.
>But seriously. If there were a cheaper and reliable way, I'm sure the authors would have mentioned it instead of H200 SXM modules. They're trying to brag about cheap training.
shit reasoning.
the big dogs use all sorts of weird (usually better) shit because they have access to it.
the opposite is also true. The H200 SXMs are available, some custom TPU purpose built for your needs isn't.
or , in other words, "if there were they'd have done it!" begs us to believe the concept that everyone out there is doing things as optimally as possible, with perfect access.
> Maybe you wanted to use an opportunity to shill tranium.
This was unnecessary and against HN guidelines to engage in good faith.
No, and we get way too many low-effort "how does this compare to $FOO" comments that serve only to improve brand recognition of $FOO.
Once you realize people are doing it you see it everywhere.
Snap, no gguf to download, will wait for one to appear to test, any improvement over an already good model is good news.
"18 days on 144 H200 GPUs" insane.. didn't thought so much hardware would be needed to train Qwen.
It isn't if you train the quantized model with a quantized K/V and cache. That's my approach and usually takes around a day for 100k reasoning traces (including abliteration).
Also this is on consumer hardware, meaning 96GB CPU RAM and 48GB GPU VRAM, using an optimized Vulkan concurrent shader pipeline. So costs for the machine are around 3-5k depending on RAM crisis.
[1] https://github.com/cookiengineer/qwen-reasoning-trainer
https://huggingface.co/bartowski/RadixArk_DaisyRose-27B-GGUF i just downloaded this
Rather than waiting, ask your assistant to quantize for you. It could be not the best quantization possible but it’s relatively quick, and I had an overall positive experience.
It is already quantized. You need to convert to GGUF format though. Did your assistant explain this wrong?
My bad, I haven’t really looked yet (and haven’t asked any assistants), thought it’s BF16 safetensors as usual, and for those I used to quantize to ggufs with just a couple commands.
[flagged]
Only "18 days on 144 H200 GPUs". It costs me £5k a month just to keep 2xH200's on.
One pet project of mine would be - if I had the resources, of course - to get a simple but decent model and post train it intensely for just one language and one task.
For example doing backend projects using C# or Go.
How did you guys set up the RL topology? Were you guys doing individual RL nodes (4xh200 rollout/4xh200 train) or did you have dedicated trainer and rollout nodes?
Dedicated rollout and trainer nodes, with RDMA transfers between them.
Do you guys support any unusual topologies? I’ve always wondered about maybe using an 8x rtx pro 6000 node for inference rollouts for cost.
Since some talented people with more modest means were able to do such a job, I wonder why Alibaba didn't do something like this when they trained the model.
They left a lot of performance on the table.
Which makes me wonder if such kind of optimizations are possible for other models like Deepseek v4.1 Flash 1017.
They did. Qwen 3.8 27b is a post-trained Qwen 3.6.
Mind you, it's still bonkers how much runway 3.8 still had in it considering how implausibly capable it is for its weight.
3.8 27B is an interesting choice. on the face of it, doing RL on it feels like refrying twice-cooked pork then stuffing it in the microwave for good measure. the fact that even one benchmark (tb4.1) increased more than marginally is kind of miraculous. would be interesting to see where "miles" would take eg. 3.8 flash next, which is already impressive despite being an under-trained architectural preview.
I had the same thought. Their described process of identifying gaps and post-training on high-quality inputs specifically designed to close those gaps is very interesting. I wonder if that's possibly the secret ingredient in the frontier lab sauce.
> Their described process of identifying gaps and post-training on high-quality inputs specifically designed to close those gaps is very interesting. I wonder if that's possibly the secret ingredient in the frontier lab sauce.
Unfortunately, there's no such thing as a free lunch. You'd expect the models to regress on other tasks when you tune them for more specifics.
Like, I guess if the models were wildly inefficient after pre-training, then this might not be the case but I'd be really surprised if there weren't strong tradeoffs here.
I'm not sure where you see a free lunch here considering the amount of work and resources that they claim to have put into designing their post-training process? As well, Qwen 3.8 27b itself is the result of post-training 3.6 and is a consistent improvement across the board, which does show post-training can in fact potentially yield such.
Only question I see is whether there's really a lot of post-training runway left in 3.8. I would have said no, considering the baffling amount of capability it already packs, but the linked article does exhibit improvements in unrelated benchmarks they didn't post-train for, which suggests to me I may be wrong about that.
> I'm not sure where you see a free lunch here considering the amount of work and resources that they claim to have put into designing their post-training process? As well, Qwen 3.8 27b itself is the result of post-training 3.6 and is a consistent improvement across the board, which does show post-training can in fact potentially yield such.
OK, I probably wasn't clear enough.
Assuming that each weight is actually useful, then shifting the weights towards one specific part of the space is likely to move it away from points in the space which were more useful for each task.
That being said, it may be that each weight is not actually useful, so maybe this is possible (which makes conceptual sense given how absurdly large even the small LLMs are).
> Only question I see is whether there's really a lot of post-training runway left in 3.8. I would have said no, considering the baffling amount of capability it already packs, but the linked article does exhibit improvements in unrelated benchmarks they didn't post-train for, which suggests to me I may be wrong about that.
The choice of benchmarks matters a lot here, but yeah it's possible that Qwen (and presumably other models) have a lot of unnecessary weights so maybe it can be improved on basically everything.
I was referring to the theorem though: https://en.wikipedia.org/wiki/No_free_lunch_theorem
Ah! That does make sense. Thank you for the clarification.
A couple thoughts, then.
Is the usefulness of a weight supposed to be a binary, useful/not useful? If not, how do we know whether all the weights that ship in Qwen 3.8 27b are already maximally useful?
Second, can there be gaps that affect all purposes negatively (conditions that trigger an endless loop, for instance, goodness knows small models are prone to such) and whose closure therefore lifts capability across all purposes?
(Not trying to be contrarian or playing devil's advocate -- the idea a model can't possibly be improved in aggregate, at least up to a certain point, just stands firmly at odds with my intuition of how neural networks work.)
I look forward to testing this against my private eval set to determine if it's benchmaxxing or actual improvement based on just 1200 tasks. downloading it now
DaisyRose gets 2.5 points out of 100 more on my 86-task bench, than Qwen 3.8 Swift 1.5. at the cost of 5.5 hours vs. 7.8 hours for Daisy.
swift run: 24GB VRAM / 4090 / llama.cpp / 5.5 hours / score 0.87 / medium reasoning / iq3_s / 128k context @ q4
Daisy Rose: 48GB VRAM / A6000 / sglang / 7.8 hours / score .898 / medium reasoning / nvfp4 / 256k context at fp8
pi for all tasks
Daisy emitted 1.92M tokens and Swift 1.5 ... didn't. 1.04M there
given multiple advantages in terms of kv cache size/quant, decode speed, time, and model quant: DaisyRose marginally beat out another base Qwen finetune running in half the VRAM.
benchmaxxing imo.
edit: formatting
Is it post-training or bench-maxxing?
where's the 35b-a3b variant they mention to have also trained? much more interesting for consumer hw. and they should've started with ornith 1.5, not qwen 3.6. we don't need 10 models that's each about 10-20% better than qwen 3.6 - we need a model that accumulates the gains.
still can't sign up for radixark to try this model
[flagged]
[flagged]
[dead]