We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower compared to enqueuing a job, but in our case the job usually takes long enough to make it irrelevant.
This might be of interest to others: Last night I stumbled across Hornbeam, a library in a similar vein from the author of Gunicorn that handles WSGI / ASGI apps as well as a specific wrapper for ML inference
We also had a similar use case, so I built Snex[0] - specifically for Elixir-Python interop. Elixir-side spawns interpreters with Ports managed by GenServers, Python-side has a thin asyncio runtime to run arbitrary user code. Declarative environments (uv), optimized serde with language-specific objects (like `%MapSet{}` <-> `set`), etc. Interpreters are meant to be long lived, so you pay for initialization once.
It's a very different approach than ex_cmd, as it's not really focused on the "streaming data" use case. Mine is a very command/reply oriented approach, though the commands can flow both ways (calling BEAM modules from Python). The assumption is that big data is passed around out of band; I may have to revisit that.
No problem. It's an SLM, I have a dedicated on-prem GPU server that I deploy behind tailscale for inference. For training, I reach out to lambdalabs and just get a beefy GPU for a few hours for the cost of a Starbucks coffee.
Similar use case as well. I use erl ports to spawn a python process as well. Error handling is a mess, but using python as a short scripting language and elixir for all the database/application/architecture has been very ideal
Honestly you saved yourself major possible headaches down the line with this approach.
At my work we run a fairly large webshop and have a ridiculous number of jobs running at all times. At this point most are running in Sidekiq, but a sizeable portion remain in Resque simply because it does just that, start a process.
Resque workers start by creating a fork, and that becomes the actual worker.
So when you allocate half your available RAM for the job, its all discarded and returned to the OS, which is FANTASTIC.
Sidekiq, and most job queues uses threads which is great, but all RAM allocated to the process stays allocated, and generally unused. Especially if you're using malloc it's especially bad. We used jemalloc for a while which helped since it allocates memory better for multithreaded applications, but easiest is to just create a process.
I don't know how memory intensive ML is, what generally screwed us over was image processing (ImageMagick and its many memory leaks) and... large CSV files. Yeah come to think of it, you made an excellent architectural choice.
I use a similar strategy for python calls from elixir. This is in a web server, usually they're part of a process pool. So we start up N workers and they hang out and answer requests when needed. I just have an rpc abstraction that handles all the fiddly bits. The two sides pass erlang terms back and forth. Pretty simple.
Very nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker (https://github.com/agoodway/pgflow) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs that Elixir/Python Oban can pick up. Also, I wonder if another approach vs the python workers picking things up is to have elixir workers call/run python/lua, etc code or is that too limiting?
This is a similar concept to Faktory, which uses a built in Redis server to manage shared job state.
You then implement workers in your language of choice and subscribe to queues.
Very interesting though, the article mentioned a few things I hadn't considered before like shared access to one database from multiple (different) apps.
I wonder how database schema state is handled in a case like that. And CI/CD.
When rust was still a fairly new language i remember using capn'n proto to communicate between some rust code and python as a way to experiment with handing off performance critical tasks to a compiled language.
I wonder how well a similar approach would work with elixir + python. Elixir obviously has very easy process isolation, but I think you'd be stuck using a NIF approach for Elixir, which probably removes any reason to try capn'n proto over just protobufs?
So if your app is 99% elixir but 1% is easier in Python because a lib you should rewrite the whole app? Makes no sense. Do you think Python devs rewrite everything in C if they have a small part that needs to use C instead of Python?
Reading that made me think how much that might be related to Elixir being very similar in syntax to Ruby. Do LLMs really differentiate between the two?
Specific studies, as the one quoted, are a long way from original real world problems.
LLMs absolutely understand and write good Elixir. I've done complex OTP and distributed work in tandem with Sonnet/Opus and they understand it well and happily keep up. All the Elixir constructs distinct from ruby are well applied: pipes, multiple function clauses, pattern matching, etc.
I can say that anecdotally, CC/Codex are significantly more accurate and faster working with our 250K lines of Elixir than our 25K lines of JS (though not typescript).
I suspect this is partly due to the quality of documentation for Elixir, Erlang, and BEAM. The OTP documentation has been around for a long time and has been excellently written. Erlang/Elixer doc gen outputs function signatures, arity, and both Elixir and Erlang handle concepts like function overloading in very explicit, well-defined ways.
* Largely stable and unchanged language through out its whole existance
* Authorship is largely senior engineers so the code you train on is high quality
* Relatively low number of abstractions in comparisson to other languages. Meaning there's less ways to do one thing.
* Functional Programming style pushes down hidden state, which lowers the complexity when understanding how a slice of a system works, and the likelyhood you introduce a bug
I suspect the biggest advantage Elixir has is the relative quality of the publicly available code. Approximately no one has Elixir as their first programming language, which keeps a lot of the absolute trash-tier code that we all make when first learning to program out of the training set. If you look at languages that are often people's first (Python, JavaScript, Java), only Java has an above average score. Of those three, Java's significantly more likely to be taught in a structured learning environment, compared to kids winging it with the other two.
(And Elixir's relationship to Ruby is pretty overstated, IMO. There's definitely inspiration, but the OO-FP jump is a makes the differences pretty extreme)
Agree with the quality level but there are other languages where that is also the case: Erlang for example is probably one of those languages.
> Elixir's relationship to Ruby is pretty overstated
Perhaps I am actually am over thinking this. Elixir has probably diverged enough from Ruby (e.g. defmodule, pipe operators, :atom syntax) for LLMs to notice the difference between the two. But it does open the question, though, how does an LLM actually recognise the difference in code blocks in its training data.
There are probably many more programming languages where similarities exist.
Having written a lot of both languages, I'd be surprised if LLMs don't get tripped up on some of Ruby's semantics and weird stuff people do with monkey patching. I also find Ruby library documentation to be on average pretty poor.
> I also find Ruby library documentation to be on average pretty poor.
That surprises me :)
From my time doing Ruby (admittedly a few years back), I found libraries were very well documented and tested. But put into context of then (not now), documentation and testing weren't that popular amongst other programming languages. Ruby was definitely one of the drivers for the general adaption of TDD principles, for example.
I think they're often very well tested, but the documentation piece has always been lacking compared to Elixir.
I used to frequently find myself reading the source code of popular libraries or prying into them at runtime. There's also no central place or format for documentation in ruby. Yes rubydoc.info exists, but it's sort of an afterthought. Sidekiq uses a github wiki, Nokogiri has a dedicated site, Rails has a dedicated site, Ruby itself has yet another site. Some use RDoc, some don't. Or look at Devise https://rubydoc.info/github/heartcombo/devise/main/frames, there's simply nothing documented for most of the classes, and good luck finding in the docs where `before_action :authenticate_user!` comes from.
I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture.
This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your system should be organized.
It’s almost as though choosing a single-threaded, GIL-encumbered interpreted scripting language as the primary interface to an ecosystem of extremely parallelized and concurrent high-performance hardware-dependent operations wasn’t quite the right move for our industry.
Ha. The question now is whether the ML industry will change directions or if the momentum of Python is a runaway train.
I can't guess. Perl was once the "800-pound gorilla" of web development, but that chapter has long been closed. Python on the other hand has only gained traction since that time.
Strange opinion. Plenty of apps have more than one language. I might end up using this.
Why? Because my app is built in Elixir and right now I’m also using a python app that is open source but I really just need a small part of the python app. I don’t wanna rewrite everything in Elixir because while it’s small I expect it to change over time (basically fetching a lot of data sources) and it will be pain to keep rewriting it when data collections needs to change (over a 100 different sources). Right now I run the python app as an api but it’s just so overkill and harder to manage vs just handling everything except the actually data collection in Elixir where I am already using Oban.
Sometimes the "right tool for the job" philosophy leads to breaking down a larger problem into two small problems, each which has a different "right tool".
Choosing a single tool that tries to solve every single problem can lead to its own problems.
I disagree, using python for a web-server and something like celery for background work is a pretty common pattern.
My reading of this is it more or less allows you to use Postgres (which you're likely already using as your DB) for the task orchestration backend. And it comes with a cool UI.
Wait until you find out about some people not writing pure python apps but also have some code in JavaScript. Crazy to mix more than one language in one machine.
I don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set.
I don't see the point of TypeScript either, I can make the LLM output JavaScript and the tokens saved not having to add types can be used to write additional tests...
The aesthetics or safety features of the languages no longer matter IMO. Succinctness, functionality and popularity of the language are now much more important factors.
Furthermore, it's actually kind of annoying that the LLMs are not better than us, and still benefit from having code properly typed, well-architected, and split into modules/files. I was lamenting this fact the other day; the only reason we moved away from Assembly and BASIC, using GOTOs in a single huge file was because us humans needed the organization to help us maintain context. Turns out, because of how they're trained, so do the LLMs.
So TypeScript types and tests actually do help a lot, simply because they're deterministic guardrails that the LLM can use to check its work and be steered to producing code that actually works.
I don't think LLMs benefit from having code properly typed (at the call definition). It's costly to have to check a possibly remote file to check. The LLM should be able to intuit what the types are at the callsite and elixir has ~strong conventions that LLMs probably take advantage of
Think about fitts law: the fastest place to click under a cursor is the location of the cursor. For an LLM the least context-expensive feedback is no feedback at all.
I think codebases that are strongly typed sometimes have bad habits that "you can get away with" because of the typing and feedback loops, the LLM has learned this.
This is well put. If the LLM gets the type wrong, then we're already discussing a failure scenario with a feedback loop involving back-and-forth changes.
LLMs are not really good at this. The idea that LLMs benefit from TypeScript is a case of people anthropomorphizing AI.
The kinds of mistakes AI makes are very different. It's WAY better than humans at copying stuff verbatim accurately and nailing the 'form' of the logic. What it struggles with is 'substance' because it doesn't have a complete worldview so it doesn't fully understand what we mean or what we want.
LLMs struggle more with requirements engineering and architecture since architecture ties into anticipating requirements changes.
> The kinds of mistakes AI makes are very different.
I think that's a bit extreme. If a programming language has good ergonomics for a short attention span human, it will likely be better for an LLM too.
However, to make good predictions about what an LLM will or will not be good at you should have a good "theory of mind" for the LLMs that will in some ways be different from a human.
> I don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set.
I can't say if it works better with other languages, but I can definitely say both Opus and Codex work really well with Elixir. I work on a fairly large application and they consistently produce well structured working code, and are able to review existing code to find issues that are very easy to miss.
The LLM needs guidance around general patterns, e.g. "Let's use a state machine to implement this functionality" but it writes code that uses language idioms, leverages immutability and concurrency, and generally speaking it's much better than any first pass that I would manually do.
I have my ethical concerns, but it would be foolish of me to state that it works poorly - if anything it makes me question my own abilities and focus in comparison (which is a whole different topic).
LLMs work great with Elixir. Running tsc in a loop while generating code still catches type errors introduced by an LLM and it’s faster than generating additional tests. Elixir is also succinct and highly functional. If you can’t find a specific library it’s easier than ever to build out the barebones functionality you need yourself or use NIFs, ports, etc.
> Succinctness, functionality and popularity of the language are now much more important factors.
Not my experience at all. The most important factor is simplicity and clarity. If an LLM can find the pattern, it can replicate that pattern.
Language matters to the extent it encourages/forces clear patterns. Language with more examples, shorter tokens, popularity, etc - doesn't matter at all if the codebase is a mess.
Functional languages like Elixir make it very easy to build highly structured applications. Each fn takes in a thing and returns another. Side effects? What side effects? LLMs can follow this function composition pattern all day long. There's less complexity, objectively.
But take languages that are less disciplined. Throw in arbitrary side effects and hidden control flow and mutable state ... the LLM will fail to find an obviously correct pattern and guess wildly. In practice, this makes logical bugs much more likely. Millions of examples don't help if your codebase is a swamp. And languages without said discipline often end up in a swamp.
Your code doesn’t run anywhere? Running on the BEAM is extremely helpful for a lot of things. Also, I review my LLM output, I want that experience to be enjoyable.
> Succinctness, functionality and popularity of the language are now much more important factors.
No. I would argue that popularity per se is irrelevant: if there are a billion examples of crap code, the LLMs learn crap code. conversely know only 250 documents can poison an LLM independent if model size. [Cite anthropic paper here].
The most important thing is conserve context. Succinctness is not really what you want because most context is burned on thinking and tool calls (I think) and not codegen.
Here is what I think is not important: strong typing, it requires a tool call anyways to fetch the type.
Here is what I think is important:
- fewer footguns
- great testing (and great testing examples)
- strong language conventions (local indicators for types, argument order conventions, etc)
- no weird shit like __init__.py that could do literally anything invisible to the standard code flow
I'm starting to see a new genre of post here in the AI bubble, where people go to topics that aren't about AI at all, and comment something like, "this doesn't matter because it's not AI". This is the third I've seen in a week.
We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower compared to enqueuing a job, but in our case the job usually takes long enough to make it irrelevant.
This might be of interest to others: Last night I stumbled across Hornbeam, a library in a similar vein from the author of Gunicorn that handles WSGI / ASGI apps as well as a specific wrapper for ML inference
https://erlangforums.com/t/hornbeam-wsgi-asgi-server-for-run... https://github.com/benoitc/hornbeam
We also had a similar use case, so I built Snex[0] - specifically for Elixir-Python interop. Elixir-side spawns interpreters with Ports managed by GenServers, Python-side has a thin asyncio runtime to run arbitrary user code. Declarative environments (uv), optimized serde with language-specific objects (like `%MapSet{}` <-> `set`), etc. Interpreters are meant to be long lived, so you pay for initialization once.
It's a very different approach than ex_cmd, as it's not really focused on the "streaming data" use case. Mine is a very command/reply oriented approach, though the commands can flow both ways (calling BEAM modules from Python). The assumption is that big data is passed around out of band; I may have to revisit that.
[0]: https://github.com/kzemek/snex
I have one vibecoded ml pipeline now and I'm strongly considering just clauding it into Nx so I can ditch the python
I did exactly this in early 2025 with a small keyword tagging pipeline.
You may run into some issues with Docker and native deps once you get to production. Don’t forget to cache the bumblebee files.
No problem. It's an SLM, I have a dedicated on-prem GPU server that I deploy behind tailscale for inference. For training, I reach out to lambdalabs and just get a beefy GPU for a few hours for the cost of a Starbucks coffee.
Similar use case as well. I use erl ports to spawn a python process as well. Error handling is a mess, but using python as a short scripting language and elixir for all the database/application/architecture has been very ideal
Honestly you saved yourself major possible headaches down the line with this approach.
At my work we run a fairly large webshop and have a ridiculous number of jobs running at all times. At this point most are running in Sidekiq, but a sizeable portion remain in Resque simply because it does just that, start a process.
Resque workers start by creating a fork, and that becomes the actual worker.
So when you allocate half your available RAM for the job, its all discarded and returned to the OS, which is FANTASTIC.
Sidekiq, and most job queues uses threads which is great, but all RAM allocated to the process stays allocated, and generally unused. Especially if you're using malloc it's especially bad. We used jemalloc for a while which helped since it allocates memory better for multithreaded applications, but easiest is to just create a process.
I don't know how memory intensive ML is, what generally screwed us over was image processing (ImageMagick and its many memory leaks) and... large CSV files. Yeah come to think of it, you made an excellent architectural choice.
Is this part of a web server or some other system where you could end up spawning N python processes instead of 1 at a time?
I use a similar strategy for python calls from elixir. This is in a web server, usually they're part of a process pool. So we start up N workers and they hang out and answer requests when needed. I just have an rpc abstraction that handles all the fiddly bits. The two sides pass erlang terms back and forth. Pretty simple.
No, it's a background job. We can easily control the Python process count by controlling the job queue concurrency on the Elixir side.
Very nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker (https://github.com/agoodway/pgflow) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs that Elixir/Python Oban can pick up. Also, I wonder if another approach vs the python workers picking things up is to have elixir workers call/run python/lua, etc code or is that too limiting?
that's easy with https://hexdocs.pm/pythonx/Pythonx.html and https://hexdocs.pm/lua/Lua.html and works well too
btw, a lot of postgres envs are not going to have pgmq, so just use Oban and don't reinvent the wheel like I did ;)
This is a similar concept to Faktory, which uses a built in Redis server to manage shared job state.
You then implement workers in your language of choice and subscribe to queues.
Very interesting though, the article mentioned a few things I hadn't considered before like shared access to one database from multiple (different) apps.
I wonder how database schema state is handled in a case like that. And CI/CD.
When rust was still a fairly new language i remember using capn'n proto to communicate between some rust code and python as a way to experiment with handing off performance critical tasks to a compiled language.
I wonder how well a similar approach would work with elixir + python. Elixir obviously has very easy process isolation, but I think you'd be stuck using a NIF approach for Elixir, which probably removes any reason to try capn'n proto over just protobufs?
Oban is great!
[dead]
I absolutely love Elixir, but if this is the bridge you need to cross, just write it in Python in the first place.
So if your app is 99% elixir but 1% is easier in Python because a lib you should rewrite the whole app? Makes no sense. Do you think Python devs rewrite everything in C if they have a small part that needs to use C instead of Python?
It's 2026 and the LLMs score high on elixir, just write it in python and patch it over to elixir gradually
Or patch it over to python, I assume LLMs are even better at python.
Don't assume. Empirically, they are not. (This post Feb 2026 may change in future yadda yadda)
See: autocodebench
https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/ma...
Reading that made me think how much that might be related to Elixir being very similar in syntax to Ruby. Do LLMs really differentiate between the two?
Specific studies, as the one quoted, are a long way from original real world problems.
Here are some thoughts on it from José Valim: https://dashbit.co/blog/why-elixir-best-language-for-ai
LLMs absolutely understand and write good Elixir. I've done complex OTP and distributed work in tandem with Sonnet/Opus and they understand it well and happily keep up. All the Elixir constructs distinct from ruby are well applied: pipes, multiple function clauses, pattern matching, etc.
I can say that anecdotally, CC/Codex are significantly more accurate and faster working with our 250K lines of Elixir than our 25K lines of JS (though not typescript).
I suspect this is partly due to the quality of documentation for Elixir, Erlang, and BEAM. The OTP documentation has been around for a long time and has been excellently written. Erlang/Elixer doc gen outputs function signatures, arity, and both Elixir and Erlang handle concepts like function overloading in very explicit, well-defined ways.
Thats a large reason for sure!
I'd layer in a few more
* Largely stable and unchanged language through out its whole existance
* Authorship is largely senior engineers so the code you train on is high quality
* Relatively low number of abstractions in comparisson to other languages. Meaning there's less ways to do one thing.
* Functional Programming style pushes down hidden state, which lowers the complexity when understanding how a slice of a system works, and the likelyhood you introduce a bug
I suspect the biggest advantage Elixir has is the relative quality of the publicly available code. Approximately no one has Elixir as their first programming language, which keeps a lot of the absolute trash-tier code that we all make when first learning to program out of the training set. If you look at languages that are often people's first (Python, JavaScript, Java), only Java has an above average score. Of those three, Java's significantly more likely to be taught in a structured learning environment, compared to kids winging it with the other two.
(And Elixir's relationship to Ruby is pretty overstated, IMO. There's definitely inspiration, but the OO-FP jump is a makes the differences pretty extreme)
Agree with the quality level but there are other languages where that is also the case: Erlang for example is probably one of those languages.
> Elixir's relationship to Ruby is pretty overstated
Perhaps I am actually am over thinking this. Elixir has probably diverged enough from Ruby (e.g. defmodule, pipe operators, :atom syntax) for LLMs to notice the difference between the two. But it does open the question, though, how does an LLM actually recognise the difference in code blocks in its training data.
There are probably many more programming languages where similarities exist.
Here are my thoughts:
https://m.youtube.com/watch?v=YZa5GqrZeao
Having written a lot of both languages, I'd be surprised if LLMs don't get tripped up on some of Ruby's semantics and weird stuff people do with monkey patching. I also find Ruby library documentation to be on average pretty poor.
> I also find Ruby library documentation to be on average pretty poor.
That surprises me :)
From my time doing Ruby (admittedly a few years back), I found libraries were very well documented and tested. But put into context of then (not now), documentation and testing weren't that popular amongst other programming languages. Ruby was definitely one of the drivers for the general adaption of TDD principles, for example.
I think they're often very well tested, but the documentation piece has always been lacking compared to Elixir.
I used to frequently find myself reading the source code of popular libraries or prying into them at runtime. There's also no central place or format for documentation in ruby. Yes rubydoc.info exists, but it's sort of an afterthought. Sidekiq uses a github wiki, Nokogiri has a dedicated site, Rails has a dedicated site, Ruby itself has yet another site. Some use RDoc, some don't. Or look at Devise https://rubydoc.info/github/heartcombo/devise/main/frames, there's simply nothing documented for most of the classes, and good luck finding in the docs where `before_action :authenticate_user!` comes from.
I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture.
This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your system should be organized.
It’s almost as though choosing a single-threaded, GIL-encumbered interpreted scripting language as the primary interface to an ecosystem of extremely parallelized and concurrent high-performance hardware-dependent operations wasn’t quite the right move for our industry.
Ha. The question now is whether the ML industry will change directions or if the momentum of Python is a runaway train.
I can't guess. Perl was once the "800-pound gorilla" of web development, but that chapter has long been closed. Python on the other hand has only gained traction since that time.
Strange opinion. Plenty of apps have more than one language. I might end up using this.
Why? Because my app is built in Elixir and right now I’m also using a python app that is open source but I really just need a small part of the python app. I don’t wanna rewrite everything in Elixir because while it’s small I expect it to change over time (basically fetching a lot of data sources) and it will be pain to keep rewriting it when data collections needs to change (over a 100 different sources). Right now I run the python app as an api but it’s just so overkill and harder to manage vs just handling everything except the actually data collection in Elixir where I am already using Oban.
Sometimes the "right tool for the job" philosophy leads to breaking down a larger problem into two small problems, each which has a different "right tool".
Choosing a single tool that tries to solve every single problem can lead to its own problems.
I disagree, using python for a web-server and something like celery for background work is a pretty common pattern.
My reading of this is it more or less allows you to use Postgres (which you're likely already using as your DB) for the task orchestration backend. And it comes with a cool UI.
That's not the sort of architecture I'm referring to. I'm specifically talking about splitting your application layer between Elixir and Python.
Wait until you find out about some people not writing pure python apps but also have some code in JavaScript. Crazy to mix more than one language in one machine.
What leads you to this conclusion
I don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set.
I don't see the point of TypeScript either, I can make the LLM output JavaScript and the tokens saved not having to add types can be used to write additional tests...
The aesthetics or safety features of the languages no longer matter IMO. Succinctness, functionality and popularity of the language are now much more important factors.
So I know these are just benchmarks, but apparently Elixir is one of the best languages to use with AI, despite having a smaller training dataset: https://www.youtube.com/watch?v=iV1EcfZSdCM and https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/ma...
Furthermore, it's actually kind of annoying that the LLMs are not better than us, and still benefit from having code properly typed, well-architected, and split into modules/files. I was lamenting this fact the other day; the only reason we moved away from Assembly and BASIC, using GOTOs in a single huge file was because us humans needed the organization to help us maintain context. Turns out, because of how they're trained, so do the LLMs.
So TypeScript types and tests actually do help a lot, simply because they're deterministic guardrails that the LLM can use to check its work and be steered to producing code that actually works.
I don't think LLMs benefit from having code properly typed (at the call definition). It's costly to have to check a possibly remote file to check. The LLM should be able to intuit what the types are at the callsite and elixir has ~strong conventions that LLMs probably take advantage of
llms benefit greatly from feedback and typing/type errors are one of the fastest and easiest methods of feedback to give to an llm.
Think about fitts law: the fastest place to click under a cursor is the location of the cursor. For an LLM the least context-expensive feedback is no feedback at all.
I think codebases that are strongly typed sometimes have bad habits that "you can get away with" because of the typing and feedback loops, the LLM has learned this.
https://x.com/neogoose_btw/status/2023902379440304452?s=61
This is well put. If the LLM gets the type wrong, then we're already discussing a failure scenario with a feedback loop involving back-and-forth changes.
LLMs are not really good at this. The idea that LLMs benefit from TypeScript is a case of people anthropomorphizing AI.
The kinds of mistakes AI makes are very different. It's WAY better than humans at copying stuff verbatim accurately and nailing the 'form' of the logic. What it struggles with is 'substance' because it doesn't have a complete worldview so it doesn't fully understand what we mean or what we want.
LLMs struggle more with requirements engineering and architecture since architecture ties into anticipating requirements changes.
> The kinds of mistakes AI makes are very different.
I think that's a bit extreme. If a programming language has good ergonomics for a short attention span human, it will likely be better for an LLM too.
However, to make good predictions about what an LLM will or will not be good at you should have a good "theory of mind" for the LLMs that will in some ways be different from a human.
> I don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set.
I can't say if it works better with other languages, but I can definitely say both Opus and Codex work really well with Elixir. I work on a fairly large application and they consistently produce well structured working code, and are able to review existing code to find issues that are very easy to miss.
The LLM needs guidance around general patterns, e.g. "Let's use a state machine to implement this functionality" but it writes code that uses language idioms, leverages immutability and concurrency, and generally speaking it's much better than any first pass that I would manually do.
I have my ethical concerns, but it would be foolish of me to state that it works poorly - if anything it makes me question my own abilities and focus in comparison (which is a whole different topic).
LLMs work great with Elixir. Running tsc in a loop while generating code still catches type errors introduced by an LLM and it’s faster than generating additional tests. Elixir is also succinct and highly functional. If you can’t find a specific library it’s easier than ever to build out the barebones functionality you need yourself or use NIFs, ports, etc.
https://dashbit.co/blog/why-elixir-best-language-for-ai
> Succinctness, functionality and popularity of the language are now much more important factors.
Not my experience at all. The most important factor is simplicity and clarity. If an LLM can find the pattern, it can replicate that pattern.
Language matters to the extent it encourages/forces clear patterns. Language with more examples, shorter tokens, popularity, etc - doesn't matter at all if the codebase is a mess.
Functional languages like Elixir make it very easy to build highly structured applications. Each fn takes in a thing and returns another. Side effects? What side effects? LLMs can follow this function composition pattern all day long. There's less complexity, objectively.
But take languages that are less disciplined. Throw in arbitrary side effects and hidden control flow and mutable state ... the LLM will fail to find an obviously correct pattern and guess wildly. In practice, this makes logical bugs much more likely. Millions of examples don't help if your codebase is a swamp. And languages without said discipline often end up in a swamp.
Your code doesn’t run anywhere? Running on the BEAM is extremely helpful for a lot of things. Also, I review my LLM output, I want that experience to be enjoyable.
> Succinctness, functionality and popularity of the language are now much more important factors.
No. I would argue that popularity per se is irrelevant: if there are a billion examples of crap code, the LLMs learn crap code. conversely know only 250 documents can poison an LLM independent if model size. [Cite anthropic paper here].
The most important thing is conserve context. Succinctness is not really what you want because most context is burned on thinking and tool calls (I think) and not codegen.
Here is what I think is not important: strong typing, it requires a tool call anyways to fetch the type.
Here is what I think is important:
- fewer footguns - great testing (and great testing examples) - strong language conventions (local indicators for types, argument order conventions, etc) - no weird shit like __init__.py that could do literally anything invisible to the standard code flow
I'm starting to see a new genre of post here in the AI bubble, where people go to topics that aren't about AI at all, and comment something like, "this doesn't matter because it's not AI". This is the third I've seen in a week.