> Most AI runtimes can only execute one inference call to a model at a time.
I don't know if the LMAX article makes much sense in this space. The latency of calling the models dominates any sort of local effects and the volume of the requests tends to be very low relative to any kind of meaningful system limits.
More broadly, parallel agent systems are of questionable utility compared to ones that operate over the problem serially. Parallelism bringing any uplift implies that the effects of different parts of the problem don't interact much. If things do depend on other things over time, then you have something that can only be solved with one serial narrative.
The case of high frequency trading works really well because the shared resource (global sequence) is extremely contentious and local. You actually get meaningful benefits out of these principles. OpenClaw is not the same kind of problem. Running an actual inference model on the CPU is maybe a different story.
I looked for a term like "mechanical sympathy" for years. In fact, I think I even came up with that independently when I was thinking about it. I remember asking on some internet forum if anyone else feels a sense of sympathy towards machines and most people made fun of me. A few admitted they didn't like crunching the gears in their car gearbox.
But I was talking about all machines. I don't like it when a washing machine is off balance and vibrating violently. I don't like to use my tools at, or anywhere near, there mechanical limits. When I observe normies they don't seem to care. They'll force and abuse things all the time. I did wonder if it was part of my predisposition towards engineering.
> When I observe normies they don't seem to care. They'll force and abuse things all the time. I did wonder if it was part of my predisposition towards engineering.
Caring implies doing the right thing, which you can't afford lest you now be bogged down with the burden of doing it correctly, which requires effort.
Some people use dull knives because the thought of maintaining and sharpening them is worse than almost losing a finger every time they need to cut a tomato in half.
I suspect that a lot of people find proactive and sustained effort to be so draining that they'd rather continually have second rate experiences and find peace in that. Then there's people who just generally don't care.
I'm hoping (for my own sanity) that this is a bias on the observer as it's easy to look at someone from the outside and not see the areas where they do care and do the right more times than not. I know I've been unable to live up to my own standards plenty of times throughout the years for factors outside of caring.
The idea that a blunt knife is more dangerous than a sharp one is a total fallacy.
Every time I've cut myself on a knife, it's been because it was too sharp, not because it was too blunt.
In the limit, a blunt knife is a sphere and a sharp knife is a sharp knife. Very obviously sharp knives are more dangerous than blunt ones because sharp knives cut better and blunt knives cut worse.
> The idea that a blunt knife is more dangerous than a sharp one is a total fallacy.
I don't think so. I personally find dull knives more dangerous because I need to apply more pressure and when it starts to cut, the knife becomes uncontrollable.
When the blade is sharp, and you know it's sharp, you respect the blade and give actual thought to what you're doing.
My wife didn't used to sharpen her knives. When I started to sharpen them, she had a couple of minor accidents, but now the accident rate is at 0.0. She even wants me to sharpen the knives when they become dull.
This is exactly what "having a feeling for the machine is". You know and respect it for what it is. It bites back when you don't respect it. Let it be a knife or a space shuttle, it doesn't matter.
If it's going to make you feel better, I'm pretty same with you. I want to be able to see and feel that everything I use, let it be software or hardware (from keyboard to washing machine and to car) is working smoothly and happily with lots of design tolerance left.
While I'm also very engineering inclined, I was like that since very young age. Maybe it's how my brain is wired, or a natural extension of my curiosity, I don't know.
From what I observe, many people doesn't know what's normal and what's not. This is two pronged. Most of the people doesn't know that they are running to the limit unless the tool misbehaves or breaks down. A minority just don't care. They either don't value, or think that machines are meant to be used that way.
Either way, it's more not being knowledgeable than being malicious against devices, at least from my experience.
Historically I think the people that first ran into these issues and started talking about this was gamedevelopers even if it flew under the radar for many others since it was an era of less third party engines and code sharing.
The motivation was partially to solve processing on PS3's, the 7 usable Cell units have small memories of 256kb each so processing had to be moved to a compact/streamable memory format but developers were also concurrently fighting larger cache latencies caused by Entity Component hierarchies that had arisen as a solution to problems with inheritance using dislocated object hierarchies (ie separated linked objects).
Out of this grew the Data Oriented Design paradigm (that's intrinsically built on mechanical sympathy) that suited both Cell like streaming as well as minimizing cache effects for main-memory code where Entity Component Systems were reorganized to use cache efficient array processing instead of linked objects.
Was the outside world oblivious? I'd say so to a large part, CPU's had gotten faster, more cores and memory latencies were masked to a certain degree for other kinds of code!
The Java people even pushed out their biggest mistake just as these effects started to become visible, the erased objects for generics that kind of locked in a design that requires generic list types to have disaggregated storage due to separate objects. In hindsight, this is IMHO the single biggest benefit that C# got over Java as their paths diverged as generic struct List<> object's have radically more efficent storage in C# compared to a Java ArrayList<>.
Afaik Project Valhalla still hasn't become mainstream even after almost _12_ years at this point, had they gone with proper generics at day one it would've been a trivial compiler upgrade.
I abused this term a bit to help data scientists & data engineers understand why they should take interest in each others skillsets. I used to liken it to a formula 1 driver (scientist) and the car / pit crew (engineers).
Sure, you can maybe be a great driver without caring about the car or the crew, but it is definitely going to have its limits. Likewise, at the end of the day the crew is there to make the driver shine, and need to be invested in understanding how they operate.
Creates a much better sense of culture and collaboration, and overall better products, when everyone can see the part they play and how important the relationship is to their peers.
> If a single writer handles batches of writes (or reads!), build each batch greedily: Start the batch as soon as data is available, and finish when the queue of data is empty or the batch is full.
> ..prioritize observability before optimization. You can't improve what you can't measure. Before applying any of these principles, define your SLIs, SLOs, and SLAs so you know where to focus and when to stop.
These principles apply not only to individual applications, but also to all systems as a whole. The single-writer principle improves performance both when writing/reading large databases and when reading/writing to RAM. Where input/output is intensive, performance improves even further.
It's high time that programming under resource constraints was rediscovered and valued. It's been so easy to throw hardware (or someone else's hardware, ie cloud) at problems that efficiency and cost per compute output has been forgotten. Maybe a few years of serious memory shortage will focus minds. Perhaps this is one of those things that needs to be recast in terms a CFO can understand before it gets attention: "you're paying XXX per compute node because your team writes flabby software. You could save $$$ if they fit into Y instead"
It’s funny that even cloud hosting and such didn’t drive more responsible resource usage.
We could easily have 10x-100x more utilization of cloud servers if the software was leaner.
Imagine a single server hosting 10000 VDI instances concurrently with high performance. Sounds insane but Windows, Word, Excel were usable on a 16MHz 386 with 2MB RAM and 20-40MB storage.
Systems today are literally 10000X as powerful (without even getting into the CPU architecture and cache improvements).
A single fundamental principle underlies the implementation of “mechanical sympathy” in software: make the topology of the software match the topology of the hardware as closely as possible. All of the various tricks and heuristics are just narrow cases of this single principle. Hardware architecture is immutable but software architecture is not.
It was how I learned to design code for supercomputers and is remarkably effective at generating extreme efficiency and scalability in diverse contexts. Importantly, it generalizes to systems of every shape and size. While it is difficult to get good results on a supercomputer if you don’t understand this, it works just as well for a random web app.
This clicks for message parsing too. Had field lookups in a std::map, fine until throughput climbed. Flat sorted array fixed it. Turns out cache prefetch actually kicks in when you give it sequential scans.
> Most AI runtimes can only execute one inference call to a model at a time.
I don't know if the LMAX article makes much sense in this space. The latency of calling the models dominates any sort of local effects and the volume of the requests tends to be very low relative to any kind of meaningful system limits.
More broadly, parallel agent systems are of questionable utility compared to ones that operate over the problem serially. Parallelism bringing any uplift implies that the effects of different parts of the problem don't interact much. If things do depend on other things over time, then you have something that can only be solved with one serial narrative.
The case of high frequency trading works really well because the shared resource (global sequence) is extremely contentious and local. You actually get meaningful benefits out of these principles. OpenClaw is not the same kind of problem. Running an actual inference model on the CPU is maybe a different story.
I looked for a term like "mechanical sympathy" for years. In fact, I think I even came up with that independently when I was thinking about it. I remember asking on some internet forum if anyone else feels a sense of sympathy towards machines and most people made fun of me. A few admitted they didn't like crunching the gears in their car gearbox.
But I was talking about all machines. I don't like it when a washing machine is off balance and vibrating violently. I don't like to use my tools at, or anywhere near, there mechanical limits. When I observe normies they don't seem to care. They'll force and abuse things all the time. I did wonder if it was part of my predisposition towards engineering.
> When I observe normies they don't seem to care. They'll force and abuse things all the time. I did wonder if it was part of my predisposition towards engineering.
Caring implies doing the right thing, which you can't afford lest you now be bogged down with the burden of doing it correctly, which requires effort.
Some people use dull knives because the thought of maintaining and sharpening them is worse than almost losing a finger every time they need to cut a tomato in half.
I suspect that a lot of people find proactive and sustained effort to be so draining that they'd rather continually have second rate experiences and find peace in that. Then there's people who just generally don't care.
I'm hoping (for my own sanity) that this is a bias on the observer as it's easy to look at someone from the outside and not see the areas where they do care and do the right more times than not. I know I've been unable to live up to my own standards plenty of times throughout the years for factors outside of caring.
The idea that a blunt knife is more dangerous than a sharp one is a total fallacy.
Every time I've cut myself on a knife, it's been because it was too sharp, not because it was too blunt.
In the limit, a blunt knife is a sphere and a sharp knife is a sharp knife. Very obviously sharp knives are more dangerous than blunt ones because sharp knives cut better and blunt knives cut worse.
> The idea that a blunt knife is more dangerous than a sharp one is a total fallacy.
I don't think so. I personally find dull knives more dangerous because I need to apply more pressure and when it starts to cut, the knife becomes uncontrollable.
When the blade is sharp, and you know it's sharp, you respect the blade and give actual thought to what you're doing.
My wife didn't used to sharpen her knives. When I started to sharpen them, she had a couple of minor accidents, but now the accident rate is at 0.0. She even wants me to sharpen the knives when they become dull.
This is exactly what "having a feeling for the machine is". You know and respect it for what it is. It bites back when you don't respect it. Let it be a knife or a space shuttle, it doesn't matter.
> you respect the blade and give actual thought to what you're doing.
Is this perhaps because you know it to be dangerous?
I'm not opposed to having sharp knives btw. Sharp knives are better because they cut more easily. But for that reason they are also more dangerous.
Would you rather hand a toddler a butter knife or a chef's knife to play with?
Depends on the toddler's character and understanding level.
I was using Swiss army knives as early as 5-6 years old. I probably cut myself thrice at most.
However, if the toddler is the reckless kind, they'll start with a plastic butter knife.
I have cut myself once, and only once with a sharp knife. I was about 7 years old, and my grandfather gave me a new (sharp!) Swiss army knife.
Then you've never injured yourself with a blunt blade. Good for you!
If it's going to make you feel better, I'm pretty same with you. I want to be able to see and feel that everything I use, let it be software or hardware (from keyboard to washing machine and to car) is working smoothly and happily with lots of design tolerance left.
While I'm also very engineering inclined, I was like that since very young age. Maybe it's how my brain is wired, or a natural extension of my curiosity, I don't know.
From what I observe, many people doesn't know what's normal and what's not. This is two pronged. Most of the people doesn't know that they are running to the limit unless the tool misbehaves or breaks down. A minority just don't care. They either don't value, or think that machines are meant to be used that way.
Either way, it's more not being knowledgeable than being malicious against devices, at least from my experience.
You are not alone. https://en.wikipedia.org/wiki/Wu_wei
Fascinating. I have been reading the Tao Te Ching but had not yet made this connection.
Historically I think the people that first ran into these issues and started talking about this was gamedevelopers even if it flew under the radar for many others since it was an era of less third party engines and code sharing.
The motivation was partially to solve processing on PS3's, the 7 usable Cell units have small memories of 256kb each so processing had to be moved to a compact/streamable memory format but developers were also concurrently fighting larger cache latencies caused by Entity Component hierarchies that had arisen as a solution to problems with inheritance using dislocated object hierarchies (ie separated linked objects).
Out of this grew the Data Oriented Design paradigm (that's intrinsically built on mechanical sympathy) that suited both Cell like streaming as well as minimizing cache effects for main-memory code where Entity Component Systems were reorganized to use cache efficient array processing instead of linked objects.
https://gamesfromwithin.com/data-oriented-design
Was the outside world oblivious? I'd say so to a large part, CPU's had gotten faster, more cores and memory latencies were masked to a certain degree for other kinds of code!
The Java people even pushed out their biggest mistake just as these effects started to become visible, the erased objects for generics that kind of locked in a design that requires generic list types to have disaggregated storage due to separate objects. In hindsight, this is IMHO the single biggest benefit that C# got over Java as their paths diverged as generic struct List<> object's have radically more efficent storage in C# compared to a Java ArrayList<>.
Afaik Project Valhalla still hasn't become mainstream even after almost _12_ years at this point, had they gone with proper generics at day one it would've been a trivial compiler upgrade.
I abused this term a bit to help data scientists & data engineers understand why they should take interest in each others skillsets. I used to liken it to a formula 1 driver (scientist) and the car / pit crew (engineers).
Sure, you can maybe be a great driver without caring about the car or the crew, but it is definitely going to have its limits. Likewise, at the end of the day the crew is there to make the driver shine, and need to be invested in understanding how they operate.
Creates a much better sense of culture and collaboration, and overall better products, when everyone can see the part they play and how important the relationship is to their peers.
> If a single writer handles batches of writes (or reads!), build each batch greedily: Start the batch as soon as data is available, and finish when the queue of data is empty or the batch is full.
> ..prioritize observability before optimization. You can't improve what you can't measure. Before applying any of these principles, define your SLIs, SLOs, and SLAs so you know where to focus and when to stop.
These principles apply not only to individual applications, but also to all systems as a whole. The single-writer principle improves performance both when writing/reading large databases and when reading/writing to RAM. Where input/output is intensive, performance improves even further.
It's high time that programming under resource constraints was rediscovered and valued. It's been so easy to throw hardware (or someone else's hardware, ie cloud) at problems that efficiency and cost per compute output has been forgotten. Maybe a few years of serious memory shortage will focus minds. Perhaps this is one of those things that needs to be recast in terms a CFO can understand before it gets attention: "you're paying XXX per compute node because your team writes flabby software. You could save $$$ if they fit into Y instead"
It’s funny that even cloud hosting and such didn’t drive more responsible resource usage.
We could easily have 10x-100x more utilization of cloud servers if the software was leaner.
Imagine a single server hosting 10000 VDI instances concurrently with high performance. Sounds insane but Windows, Word, Excel were usable on a 16MHz 386 with 2MB RAM and 20-40MB storage.
Systems today are literally 10000X as powerful (without even getting into the CPU architecture and cache improvements).
A single fundamental principle underlies the implementation of “mechanical sympathy” in software: make the topology of the software match the topology of the hardware as closely as possible. All of the various tricks and heuristics are just narrow cases of this single principle. Hardware architecture is immutable but software architecture is not.
It was how I learned to design code for supercomputers and is remarkably effective at generating extreme efficiency and scalability in diverse contexts. Importantly, it generalizes to systems of every shape and size. While it is difficult to get good results on a supercomputer if you don’t understand this, it works just as well for a random web app.
This clicks for message parsing too. Had field lookups in a std::map, fine until throughput climbed. Flat sorted array fixed it. Turns out cache prefetch actually kicks in when you give it sequential scans.