It’s true. But static proof based checks can happen across system boundaries it’s just for historical and stupid reasons we misapply it here.
First put all your services in a monorepo have it all build as one under CI. That’s a static check across the entire system. When you do a polyrepo you are opening up a mathematical hole where two repos can be mismatched.
Second don’t use databases. Databases don’t type check and aren’t compatible with your functional code written on servers.
The first one is stupid… almost everyone just makes this mistake and most of the time unless you were there in the beginning to enforce monorepo you have to live with someone deploying a breaking change that effects a repo somehwere else. I’ve been in places where they simply didn’t understand the hole with polyrepos and they went with it despite the issue. Polyrepos exist for people who like organizing things with names and don’t understand static safety.
The second one is also stupid but you had to be there in the very very beginning to have stopped it. The inception of the concept of databases should have been created such that any query you run on it needs to be compiled to run on the database and is thus statically checked. This is something that can easily be done but wasn’t done and now we all just make do with testing.
That’s like half the problems with distributed systems completely solved. And these problems just exist because of raw stupidity. Stuff like race conditions and dead locks are solvable too but much harder. But these issues are obvious. What’s not obvious are the aforementioned problems that almost nobody talks about even though they are completely solvable.
This article, despite clocking in at nearly 10,000 words, appears to have been written in only two days. In fact the author has written five engineering articles in the last week, seeming to average around 5000+ words per article. What a spurt of productivity! Coincidentally, the article happens to be filled with pithy headers and constant sensationally dramatic contrasts that are out of place in long-form technical writing, but which LLMs are known to spam ad nauseum because they are effective for hooking attention in clickbait rags. Curious, that.
Normally the database and message queue is decoupled from the backend service. This decoupling makes managing simpler and make cross-language things easier.
But if the type system need to cover all these components, then they start coupling again.
Coupling is not necessarily a bad thing as long as it gives good developer UX. If the database is tightly coupled with programming language, then it looks like ORM but better. And it probably can also reduce CRUD biolerplate or N+1 issue etc.
Related, there is SpacetimeDB that make backend run within database, and the backend code is highly coupled with SpacetimeDB's own API
The challenges mentioned on the article are challenges that are known to every larger distributed system. They are not challenges solved or to be solved by functional programming.
But the solutions and tools functional programming provides help to have higher verifiability, fewer side effects and more compile-time checks of code in the deployment unit. That other challenges exists is fully correct. But without a functional approach, you have these challenges plus all others.
So while FP is not a one-size-fits-all solution to distributed systems challenges, it does help solve a subset of the challenges on a single system level.
An awesome article (if only we remove "functional programmers", feel like they are unfairly called out), hit close to home on operations and pull up a lot of (painful) memories.
Fundamentally - all your system after a certain scale becomes distributed systems, accuracy on a local level is usually not where systemic risks are coming from. A distributed system of fully accurate and correct local programs can implode or explode. Migration, compatibility and system stability are surprisingly dependent on engineer experience till this day.
I went into this expecting it to be a criticism in the context of low-level systems, and was a bit surprised when it ended up being about distributed. The mismatch the author is describing, I think, is not really about functional programmers so much as everyone not working directly on modern distributed or web software. For being upfront about all the ways distributed programming is different, this is actually one of the best intros I have seen for the non-distributed programmer on what distributed programming is really like.
I still want to believe that future programming languages will be capable of tackling these issues, but exploring the design space will require being cognizant of those issues in the first place.
> Static types, algebraic data types, making illegal states unrepresentable: the functional programming tradition has developed extraordinary tools for reasoning about programs.
But none of these things are functional programming? This is more the tradition of 'expressive static types' than it is of FP.
What about Lisp, Racket, Scheme, Clojure, Erlang / Elixir...
Agreed, but the article begins with the previous quote, and is entitled "what functional programmers get wrong", so I feel like there's some preliminary assumptions being made about FP that warrant examining.
In practice, much of the article seems to be about the problems introduced by rigid typing. Your quote, for instance, is used in the context of reading old logs using a typed schema if that schema changes. But that's a non-issue in the FP languages mentioned above since they tend towards the use of unstructured data (maps, lists) and lambdas. Conversely, reading state from old, schema-incompatible logs might be an issue in something like Java or C++, which certainly are not FP languages as the term is usually understood.
So overall, not really an FP issue at all, and yet the article is called "what functional programmers get wrong". The author's points might be very valid for his version of FP, but his version of FP seems to be 'FP in the ML tradition, with types so rigid you might want to consult a doctor after four hours'.
This is really, really good as an overview of system-level issues. I write only to encourage others to read it.
Ideas that helped me situate many of the issues we struggled with at various companies:
- distinguishing programs (the code as designed) from (distributed) systems (the whole mess, including maintenance and migration, as deployed)
- versioning code (reversibly) and data (irreversibly)
- walk-through of software solutions for distributed systems, and their blind spots wrt data versioning; temporal databases
- semantic drift: schema-less change
- Limitations of type systems for managing invariants
- co-evolving code and data; running old code on old data
The real benefit of systems thinking is recognizing whether an issue is unsolvable and/or manageable (with the current approach, or warrants a technology upgrade or directional shift). It's like O(n) evaluation of algorithms, not for space/time but for issue tractability.
This is an absolutely wonderful article. I helped build a system very similar to the one quoted below:
> A deploy pipeline that queries your orchestrator for running image tags, checks your migration history against the schema registry, diffs your GraphQL schema against collected client operations, and runs Buf’s compatibility checks: this is buildable today, with off-the-shelf parts.
that was largely successful at eliminating interservice compatibility errors, but it felt like we were scrambling around in a dark and dusty corner of software engineering that not many people really cared about. Good to see that there's a body of academic work that I was not entirely aware about that is looking into this.
The article successfully identifies that, yes, there are ways, with sufficient effort, to statically verify that a schema change is (mostly) safe or unsafe before deployment. But even with that determination, a lot of IDLs make it still very hard to evolve types! The compatibility checker will successfully identify that strengthening a type from `optional` to `required` isn't backwards compatible: _now what_. There isn't a safe pathway for schema evolution, and you need to reach for something like Cambria (https://www.inkandswitch.com/cambria/, cited in the article) to handle the evolution.
I've only seen one IDL try to do this properly by baking evolution semantics directly into its type model, typical (https://github.com/stepchowfun/typical) with it's `asymmetric` type label: it makes a field required in the constructor but not the deserialiser. I would like to see these "asymmetric" types find their ways into IDLs like Protocol Buffers such that their compatibility checkers, like Buf's `buf breaking` (https://buf.build/docs/reference/cli/buf/breaking/), can formally reason about them. I feel like there is a rich design space of asymmetric types that are poorly described (enum values that can only be compared against but never constructed, paired validations, fallback values sent over the wire, etc.)
Another one that I think makes a pretty good attempt is the config language CUE and its type system (https://cuelang.org/docs/concept/the-logic-of-cue/), which allows you to do first-class version compatibility modelling. But I have yet to investigate whether CUE is fit for purpose for modelling compatibility over the wire / between service deploys.
The version problem is solved well in FP by catching the old function and translating it cleanly. This ofc also works with data representation.
Im not sure what the point of the post is. Minimizing side effects in your code (keeping functions pure) is what gives you the flexibility he says FP misses?!
In this case, we don't have good data representations for code which produces or consumes structured data, where the structure of that data changes over time. We have some pieces figured out for certain domains, such as database schemas, but we do not have a "theory-of-everything" for enforcing types/structure across all domains of our systems. The domain addressed by GraphQL, HTTP APIs, is another instance of this same problem.
There are experimental systems like Unison that keep old versions of your code alive as data, which are fascinating solutions to the underlying problem.
Did you read the article? The issue isn’t code vs. data, it’s code vs. system state. The system can be in states not model-able by code, e.g. versions 1 and 2 of a piece of software running concurrently. Other examples I can think of are things like nodes going down, database consistency, etc.
Are there any other libaries / research / etc that sorta take a more functional approach to solving these sort of problems?
For db stuff - what if we flipped the way we write schemas around. A schema is its something that you derive at a current state, rather than starting by writing a schema and then generating migrations? So you can reason over all of it rather than a particular snapshot?
I thought MongoDB was the worst thing I’d ever heard of. Today, I learned there is a new champion. “Let us deliberately lean into EAV, and eschew types entirely.” What?!
I haven't used datomic specifically, but the datalog model it implements is honestly a pretty good way to think about things. The worst part about learning it has been realizing how many systems out there are just worse reimplementations of the same idea.
Gameplay tags for example [0], and quite a lot of SQL.
Datomic is not sql but it is not mongo either. Mongo is completely unprincipled. Sql is halfway principled. Datomic and datalog are actually completely principled
Thats actually how its done! You accept all schemas in your db and default value or afterfit around old interfaces to keep compatibility.
You never really write directly ofc so maybe the author is confused as to how FP handles this. For a category this is just another morphism be it a simple schema or a more complicated function, its just many in/out plugged together.
So when you call writeXY your caller has absolutely no need to know what actually happens. Catching and modifying old versions is just another morphism. You can even keep the layout and just accept version and payload as input.
I use a lot of elixir and rust. `mix compile` does not necessarily reason across migrations for the app with what limited type checking it does in 1.19.
You define an ecto schema and write your own migrations, but like the state of the app seems tied to that kinda snapshot of the schema. Its not like you have some lens into the overall chain of all the migrations together.
The value of a schema in a db like Postgres isn’t in the description of the storage format or indexing, but the invariants the database can enforce on the data coming in. That includes everything from uniqueness and foreign key constraints to complex functions that can pull in dozens of tables to decide whether a row is valid or should be rejected. How do you derive declarative rules from a turing complete language built into the database?
Eg some table Users -> you start with `add user_id` , `add org_id`, `remove org_id` for example, so then the current state is `Users{ user_id }`. But you trust the compiler to derive that, and then when you want to do something with Users you have to scope into it, or tell it how to handle different steps in that chain.
Im not saying this is not equivalent at the end of the day, just if anything surfaces it this way, or makes it more ergonomic.
The issue is our current databases were not designed for proper schema resolution.
The correct answer here is that a database ought to be modeled as a data type. Sql treats data type changes separate from the value transform. To say this is retarded is an understatement.
The actual answer is that the schema update is a typed function from schema1 to schema2. The type / schema of the db is carried in the types of the function. But the actual data is moved via the function computation.
Keeping multiple databases around is honestly a potential good use of homotopy type theory extended with support for partial / one-way equivalences
With all due respect to Duncan... The reason why this is an issue at all is because the people who architected the internet were not thinking functionally. If they were, they would have come up with something like Nix.
But yes, interfacing between formally proven code and the wild wild West is always an issue. This is similar to when a person from a country with a strong legal system suddenly find themselves subject to the rule of the jungle.
The better question to ask is why did we choose to build a chaotic jungle and how do we get out of it
It’s true. But static proof based checks can happen across system boundaries it’s just for historical and stupid reasons we misapply it here.
First put all your services in a monorepo have it all build as one under CI. That’s a static check across the entire system. When you do a polyrepo you are opening up a mathematical hole where two repos can be mismatched.
Second don’t use databases. Databases don’t type check and aren’t compatible with your functional code written on servers.
The first one is stupid… almost everyone just makes this mistake and most of the time unless you were there in the beginning to enforce monorepo you have to live with someone deploying a breaking change that effects a repo somehwere else. I’ve been in places where they simply didn’t understand the hole with polyrepos and they went with it despite the issue. Polyrepos exist for people who like organizing things with names and don’t understand static safety.
The second one is also stupid but you had to be there in the very very beginning to have stopped it. The inception of the concept of databases should have been created such that any query you run on it needs to be compiled to run on the database and is thus statically checked. This is something that can easily be done but wasn’t done and now we all just make do with testing.
That’s like half the problems with distributed systems completely solved. And these problems just exist because of raw stupidity. Stuff like race conditions and dead locks are solvable too but much harder. But these issues are obvious. What’s not obvious are the aforementioned problems that almost nobody talks about even though they are completely solvable.
This article, despite clocking in at nearly 10,000 words, appears to have been written in only two days. In fact the author has written five engineering articles in the last week, seeming to average around 5000+ words per article. What a spurt of productivity! Coincidentally, the article happens to be filled with pithy headers and constant sensationally dramatic contrasts that are out of place in long-form technical writing, but which LLMs are known to spam ad nauseum because they are effective for hooking attention in clickbait rags. Curious, that.
Idk I'm halfway through it and am really resonating with its points AND missing the telltale signs of ai.
So either his prompt is making the writing more palatable, or maybe he's just prolific.
In either case, I would count this as a w
What pity headers?
Even if it is, or isn't ai, the content is in fact correct.
There is no 'one' state of your application, unless you literally do maintenance window deploys and 0 queues and keep everything sync.
Normally the database and message queue is decoupled from the backend service. This decoupling makes managing simpler and make cross-language things easier.
But if the type system need to cover all these components, then they start coupling again.
Coupling is not necessarily a bad thing as long as it gives good developer UX. If the database is tightly coupled with programming language, then it looks like ORM but better. And it probably can also reduce CRUD biolerplate or N+1 issue etc.
Related, there is SpacetimeDB that make backend run within database, and the backend code is highly coupled with SpacetimeDB's own API
The challenges mentioned on the article are challenges that are known to every larger distributed system. They are not challenges solved or to be solved by functional programming.
But the solutions and tools functional programming provides help to have higher verifiability, fewer side effects and more compile-time checks of code in the deployment unit. That other challenges exists is fully correct. But without a functional approach, you have these challenges plus all others.
So while FP is not a one-size-fits-all solution to distributed systems challenges, it does help solve a subset of the challenges on a single system level.
An awesome article (if only we remove "functional programmers", feel like they are unfairly called out), hit close to home on operations and pull up a lot of (painful) memories.
Fundamentally - all your system after a certain scale becomes distributed systems, accuracy on a local level is usually not where systemic risks are coming from. A distributed system of fully accurate and correct local programs can implode or explode. Migration, compatibility and system stability are surprisingly dependent on engineer experience till this day.
I went into this expecting it to be a criticism in the context of low-level systems, and was a bit surprised when it ended up being about distributed. The mismatch the author is describing, I think, is not really about functional programmers so much as everyone not working directly on modern distributed or web software. For being upfront about all the ways distributed programming is different, this is actually one of the best intros I have seen for the non-distributed programmer on what distributed programming is really like.
I still want to believe that future programming languages will be capable of tackling these issues, but exploring the design space will require being cognizant of those issues in the first place.
It’s about a specific branch of functional programming that values an almost mathematical rigor, though (like Haskell).
I don’t think the criticism of the author applies LISP-style functional programming, which is much more accommodating to embracing chaos.
> Static types, algebraic data types, making illegal states unrepresentable: the functional programming tradition has developed extraordinary tools for reasoning about programs.
But none of these things are functional programming? This is more the tradition of 'expressive static types' than it is of FP.
What about Lisp, Racket, Scheme, Clojure, Erlang / Elixir...
I think the more relevant functional programming quote is
> The immutability of the log is the entire value proposition.
Agreed, but the article begins with the previous quote, and is entitled "what functional programmers get wrong", so I feel like there's some preliminary assumptions being made about FP that warrant examining.
In practice, much of the article seems to be about the problems introduced by rigid typing. Your quote, for instance, is used in the context of reading old logs using a typed schema if that schema changes. But that's a non-issue in the FP languages mentioned above since they tend towards the use of unstructured data (maps, lists) and lambdas. Conversely, reading state from old, schema-incompatible logs might be an issue in something like Java or C++, which certainly are not FP languages as the term is usually understood.
So overall, not really an FP issue at all, and yet the article is called "what functional programmers get wrong". The author's points might be very valid for his version of FP, but his version of FP seems to be 'FP in the ML tradition, with types so rigid you might want to consult a doctor after four hours'.
This is really, really good as an overview of system-level issues. I write only to encourage others to read it.
Ideas that helped me situate many of the issues we struggled with at various companies:
- distinguishing programs (the code as designed) from (distributed) systems (the whole mess, including maintenance and migration, as deployed)
- versioning code (reversibly) and data (irreversibly)
- walk-through of software solutions for distributed systems, and their blind spots wrt data versioning; temporal databases
- semantic drift: schema-less change
- Limitations of type systems for managing invariants
- co-evolving code and data; running old code on old data
The real benefit of systems thinking is recognizing whether an issue is unsolvable and/or manageable (with the current approach, or warrants a technology upgrade or directional shift). It's like O(n) evaluation of algorithms, not for space/time but for issue tractability.
This is an absolutely wonderful article. I helped build a system very similar to the one quoted below:
> A deploy pipeline that queries your orchestrator for running image tags, checks your migration history against the schema registry, diffs your GraphQL schema against collected client operations, and runs Buf’s compatibility checks: this is buildable today, with off-the-shelf parts.
that was largely successful at eliminating interservice compatibility errors, but it felt like we were scrambling around in a dark and dusty corner of software engineering that not many people really cared about. Good to see that there's a body of academic work that I was not entirely aware about that is looking into this.
The article successfully identifies that, yes, there are ways, with sufficient effort, to statically verify that a schema change is (mostly) safe or unsafe before deployment. But even with that determination, a lot of IDLs make it still very hard to evolve types! The compatibility checker will successfully identify that strengthening a type from `optional` to `required` isn't backwards compatible: _now what_. There isn't a safe pathway for schema evolution, and you need to reach for something like Cambria (https://www.inkandswitch.com/cambria/, cited in the article) to handle the evolution.
I've only seen one IDL try to do this properly by baking evolution semantics directly into its type model, typical (https://github.com/stepchowfun/typical) with it's `asymmetric` type label: it makes a field required in the constructor but not the deserialiser. I would like to see these "asymmetric" types find their ways into IDLs like Protocol Buffers such that their compatibility checkers, like Buf's `buf breaking` (https://buf.build/docs/reference/cli/buf/breaking/), can formally reason about them. I feel like there is a rich design space of asymmetric types that are poorly described (enum values that can only be compared against but never constructed, paired validations, fallback values sent over the wire, etc.)
Another one that I think makes a pretty good attempt is the config language CUE and its type system (https://cuelang.org/docs/concept/the-logic-of-cue/), which allows you to do first-class version compatibility modelling. But I have yet to investigate whether CUE is fit for purpose for modelling compatibility over the wire / between service deploys.
The version problem is solved well in FP by catching the old function and translating it cleanly. This ofc also works with data representation.
Im not sure what the point of the post is. Minimizing side effects in your code (keeping functions pure) is what gives you the flexibility he says FP misses?!
If your gut response is "here's a way to make the code better" then, yeah, you missed the entire point of the post.
I was hoping for discussion and got a zero content kneejerk response
The point of the article is that the system is not representable in code so all the tools FP provides are useless in systems programming.
You my friend need more milewski in your life and let go of the idea that code and data are different things.
In this case, we don't have good data representations for code which produces or consumes structured data, where the structure of that data changes over time. We have some pieces figured out for certain domains, such as database schemas, but we do not have a "theory-of-everything" for enforcing types/structure across all domains of our systems. The domain addressed by GraphQL, HTTP APIs, is another instance of this same problem.
There are experimental systems like Unison that keep old versions of your code alive as data, which are fascinating solutions to the underlying problem.
Did you just state that FP cannot solve the problem because some protocol wont allow you to specify types in the manner?
Did you read the article? The issue isn’t code vs. data, it’s code vs. system state. The system can be in states not model-able by code, e.g. versions 1 and 2 of a piece of software running concurrently. Other examples I can think of are things like nodes going down, database consistency, etc.
Very thoughtful and well written article.
OMG. Sweet relief. This is one of the greatest things I've read in ages.
An FP programmer admitting they live in ivory towers. I never thought I'd see the day.
I love it. HN needs 10 times more of this.
This person has grown up - the rest of you? Get a grip.
this website is incredibly unfriendly to scroll on firefox mobile. i could read it. it jumps every time i scroll down
Desktop too.
reader mode
Are there any other libaries / research / etc that sorta take a more functional approach to solving these sort of problems?
For db stuff - what if we flipped the way we write schemas around. A schema is its something that you derive at a current state, rather than starting by writing a schema and then generating migrations? So you can reason over all of it rather than a particular snapshot?
You might be interested in the datomic model [0], where you never remove data and all previous views of the database remain accessible.
[0] https://docs.datomic.com/datomic-overview.html#information-m...
I thought MongoDB was the worst thing I’d ever heard of. Today, I learned there is a new champion. “Let us deliberately lean into EAV, and eschew types entirely.” What?!
I haven't used datomic specifically, but the datalog model it implements is honestly a pretty good way to think about things. The worst part about learning it has been realizing how many systems out there are just worse reimplementations of the same idea.
Gameplay tags for example [0], and quite a lot of SQL.
[0] https://dev.epicgames.com/documentation/en-us/unreal-engine/...
Datomic is not sql but it is not mongo either. Mongo is completely unprincipled. Sql is halfway principled. Datomic and datalog are actually completely principled
What do you mean by “principled?”
The semantics are simple and complete. Sql is based off a complete system (relational algebra), but the null handling results in degenerate cases.
Thats actually how its done! You accept all schemas in your db and default value or afterfit around old interfaces to keep compatibility. You never really write directly ofc so maybe the author is confused as to how FP handles this. For a category this is just another morphism be it a simple schema or a more complicated function, its just many in/out plugged together.
So when you call writeXY your caller has absolutely no need to know what actually happens. Catching and modifying old versions is just another morphism. You can even keep the layout and just accept version and payload as input.
I use a lot of elixir and rust. `mix compile` does not necessarily reason across migrations for the app with what limited type checking it does in 1.19.
You define an ecto schema and write your own migrations, but like the state of the app seems tied to that kinda snapshot of the schema. Its not like you have some lens into the overall chain of all the migrations together.
How do you derive constraints without a schema?
The value of a schema in a db like Postgres isn’t in the description of the storage format or indexing, but the invariants the database can enforce on the data coming in. That includes everything from uniqueness and foreign key constraints to complex functions that can pull in dozens of tables to decide whether a row is valid or should be rejected. How do you derive declarative rules from a turing complete language built into the database?
The current state must satisfy all constraints?
Eg some table Users -> you start with `add user_id` , `add org_id`, `remove org_id` for example, so then the current state is `Users{ user_id }`. But you trust the compiler to derive that, and then when you want to do something with Users you have to scope into it, or tell it how to handle different steps in that chain.
Im not saying this is not equivalent at the end of the day, just if anything surfaces it this way, or makes it more ergonomic.
Ah I think I misunderstood. That sounds like event sourcing? As far as I know that has never been implemented at a language level.
The issue is our current databases were not designed for proper schema resolution.
The correct answer here is that a database ought to be modeled as a data type. Sql treats data type changes separate from the value transform. To say this is retarded is an understatement.
The actual answer is that the schema update is a typed function from schema1 to schema2. The type / schema of the db is carried in the types of the function. But the actual data is moved via the function computation.
Keeping multiple databases around is honestly a potential good use of homotopy type theory extended with support for partial / one-way equivalences
With all due respect to Duncan... The reason why this is an issue at all is because the people who architected the internet were not thinking functionally. If they were, they would have come up with something like Nix.
But yes, interfacing between formally proven code and the wild wild West is always an issue. This is similar to when a person from a country with a strong legal system suddenly find themselves subject to the rule of the jungle.
The better question to ask is why did we choose to build a chaotic jungle and how do we get out of it