I don't have much experience with GitHub's CI offering. But if this is an accurate description of the steps you need to take to use it securely ... then I don't think it can, in fact, ever be used securely.
Even if you trust Microsoft's cloud engineering on the backend, this is a system that does not appear to follow even the most basic principles of privilege and isolation? I'm not sure why you would even try to build "supply-chain security" on top of this.
I would agree with this. I recently tried to figure out how to properly secure agent-authored code in GitHub Actions. I believe I succeeded in doing this[1] but the secure configuration ended up being so delicate that I don’t have high hopes of this being a scalable path.
Now, as other commenter pointed out, maybe this is just inherent complexity in this space. But more secure defaults could go a long way making this more secure in practice.
Out of curiosity, is there a build setup you have seen in the past that you think could be a good replacement for this complex GitHub CI setup? Asking for a friend ;)
Update: now I've finished reading the article, my impression is that complexity is mostly inherent to this problem space. I'd be glad to be proven wrong, though!
I think any of the webhook-based providers are better, because you can isolate your secrets. PRs go to a PR webhook that runs in an environment that just doesn’t have access to any secrets.
Releases go to the release webhook, which should output nothing and ideally should be a separate machine/VM with firewall rules and DNS blocks that prevent traffic to anywhere not strictly required.
Things are a lot harder to secure with modern dynamic infrastructure, though. Makes me feel old, but things were simpler when you could say service X has IP Y and add firewall rules around it. Nowadays that service probably has 15 IP addresses that change once a week.
The complexity comes from how the whole system is designed.
There’s no single repository or curated packages as is typical in any distribution: instead actions pull other actions, and they’re basically very complex wrapper around scripts which downloads binaries from all over the place.
For lots of very simple actions, instead of installing a distribution package and running a single command, a whole “action” is used which creates and entire layer of abstraction over that command.
It’s all massive complexity on top of huge abstractions, none of which were designed with security in mind: it was just gradually bolted on top over the years.
Yes, this problem space has inherent complexity, but no, this inherent complexity does not require Github's insanely insecure defaults and incoherent security model.
As a practical step, one could try using webhooks to integrate their github repo with literally any other CI provider. This would at least give you a single, low-coupling primitive to build your workflows on. It would not, in any way, eliminate the domain's inherent complexity (secrets, 3rd party contributions, trusted publishing, etc.), but it starts out safe because by default it doesn't do anything - it's just an HTTP call that gets fired under certain conditions.
This is no different from any package registry getting some packages compromised.
Not many of them allow for immutable relases. And if they do, nothing blocks you from releasing a patch version that will most likely be automatically pulled in by many many projects during build.
The whole dependencies ecosystem is currently broken. Thats why its so easy (relatively) to attack via supply-chain.
Only way to be really secured is to have own registry of vetted dependencies pinned to exact version and maintain own upgrade pipeline.
NOONE (beside google) is going to do that. Its too costly, you need two big teams just to handle that one part.
> NOONE (beside google) is going to do that. Its too costly, you need two big teams just to handle that one part.
And yet my team and I at stagex are building a decentralized code review system to handle this anyway. Not waiting around with our fingers crossed for the corpos to solve supply chain security for us. Has to be a community led effort.
The only binaries of uv in the world you can get that were full source bootstrapped from signed package commits to signed reviews to multi-signed deterministic artifacts are the ones from my teammates and I at stagex.
All keys on geodistributed smartcards held by maintainers tied to a web of trust going back 25 years with over 5000 keys.
Though thankful for clients that let individual maintainers work on stagex part time once in a while, we have had one donation ever for $50 as a project. (thanks)
Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.
OpenClaw has been an outstanding success, it is providing people the ability to leak their keys, secrets, and personal data, and allowing people to be subject to an incredible number of supply chain attacks when its users have felt their attack surface was just too low.
Your efforts have been on increasing security and reducing supply chain attacks, when the market is strongly signaling to you that people want reduced security and more supply chain attacks!
> All keys on geodistributed smartcards held by maintainers tied to a web of trust going back 25 years with over 5000 keys.
Neither the age nor the cardinality of the key graph tells me anything if I don’t trust the maintainers themselves; given that you’re fundamentally providing third-party builds, what’s the threat model you’re addressing?
It’s worth nothing that all builds of uv come from a locked resolution and, as mentioned in TFA, you can get signed artifacts from us. So I’m very murky on the value of signed package commits that come from a different set of identities than the ones actually building the software.
StageX does reproducible builds, so they are signed independently and can also be verified locally. I don't think it applies to Astral, but it's useful for packages with a single maintainer or a vulnerable CI, where there is only one point of failure.
But I also think it'd be nice if projects provided a first-party StageX build, like many do with a Dockerfile or a Nix flake.
Once we have better support for multiarch in stagex, since StageX is distributed as OCI images, you could just replace your existing Dockerfile bases with stagex.
You definitely trust the same web of trust key graph already in every single layer of your current CI solution. Everything at Astral and by all indications also OpenAI is built with third party services, third party (blind) signing, using third party binaries signed by those 5000 keys directly or indirectly.
That web of trust is the trust foundation of the entire internet and likely every server that powers Github, Astral, and OpenAI including every CI system you described.
One node in that graph is also nowhere near good enough to stop supply chain attacks, which is why we use -multiple- points thanks to full source bootstrapped deterministic builds.
Let me flip it and ask why anyone should trust that an Astral/OpenAI employee that does not sign their commits and does not sign their reviews, has not been impersonated or had an account takeover due to the phishable 2FA that is allowed, and won't just make a commit to CI stack for uv (or uv itself!) under a pseudonym then merge their pseudonym's code.
One person can burn it all down in spite of the practices in this blog post. Letting machines blindly sign whatever non-deterministic outputs come out of an automated process does not actually buy you much in practice against many of the supply chain attack tactics actually used in the wild. Also of course the same applies to the third party build systems you trust. Github themselves also don't use any of these basic supply chain security practices either so many many points of failure here.
Astral/OpenAI are actually giving -thousands- of randos other than the authors the ability to backdoor the uv binaries you produce, and without a reproducible full source bootstrapped build process, no one would be able to quickly or easily prove it.
To package or change uv in stagex one maintainer must sign the commit, and another must sign the review/merge commit. Then -multiple- maintainers must compile 180 bytes of human readable machine code, build up to tinycc, then gcc, then llvm, and eventually to a rust compiler, that we then use to build uv, all deterministically.
So, we actually don't trust any third parties other than the actual authors of the source code to a limited extent in our process. That said we are working on a solution for decentralized review of upstream code as well right now because we largely don't trust upstreams to not let their identities get stolen because most teams for whatever reason refuse to sign their commits and reviews, so we will have to do that for them too. Regardless, we can prove we faithfully deliver honest compilations of whatever upstream code is published without any single points of failure.
We ask users downloading binaries to trust that a bunch of maintainers are putting their personal reputations and keys (which long predate AI and are hard to impersonate) on the line to sign their bit for bit identical builds of uv, and the entire toolchain underneath it, and provide faithful compilations of upstream source code.
It would make everyone a lot safer if upstreams, especially well funded ones, could meet or exceed the threat model we must support downstream.
> You definitely trust the same web of trust key graph already in every single layer of your current CI solution. Everything at Astral and by all indications also OpenAI is built with third party services, third party (blind) signing, using third party binaries signed by those 5000 keys directly or indirectly.
I don't think we do; there are places we trust distribution signers, but we don't do so in a "web" topology; we trust them because a small set of keys is pre-baked into VMs, Docker images, etc. The web of trust, as it existed 20 years ago, is dead[1].
Topologically this is a lot like a CA ecosystem, except worse in material ways: even distros (full of talented, motivated people!) struggle to operationalize PGP, so we end up with a bunch of de facto unexpirable and irrevocable keys[2] that nobody is really tracking. Consequently, nobody is really factoring these into their security story, whether or not they're a web.
You can call it dead, and yet, it is the only system signing internet infrastructure at scale right now because of Debian, Suse, Fedora, Arch, Gentoo, Ubuntu, Redhat, etc. It is very much alive for those use cases that need it most. If one of those keys was compromised or the people that hold them, most of the internet is backdoored. Except for things built on stagex, in which case you would have to compromise several. Still not good enough, but better.
PGP Web of Trust for all its faults and early design facepalms (of which there are many) is the only proof-of-human system where humans meet humans and sign each others keys that we ever built before AI. No one can reasonably expect any recently created keys were not created by made up LLM identities unless signed into the web of trust by well published existing keys held by well known and trusted humans.
But even if you don't want to look at the Web of Trust you can prove the key I sign stagex releases with is mine via all sorts of other ways thanks to keyoxide: https://keyoxide.org/E90A401336C8AAA9
Also PGP specs supports modern crypto now, attestation via dns, and even hackernews. You can attest my PGP key is tied to my HN profile right now. I would agree -gpg- is dead, with no real reason to use it anymore now that we have modern rust tooling with modern crypto.
But! If someone wants to generate an ssh key on a smartcard or something and sign with that instead, we would absolutely consider it. Not married to supporting only a single spec, but we absolutely need human beings to hold their own private keys on smartcards which are themselves attested by other human held private keys and the online services shared by the same identities.
No, I call it dead because it's dead. The SKS network is dead, the strong set is moribund, and the remaining real users of PGP are instead slinging key bundles around by baking them into pre-trusted artifacts (like ISOs). But that's not a "web of trust," it's just bespoke centralized key distribution with a certification format that every single serious cryptographer agrees is terrible.
(And this is before a more brute statistical argument: even at its greatest extent, the PGP ecosystem was minuscule[1].)
I am deeply aware of Moxies views on this, and we have talked about them at length, and he is wrong. Also SKS and GnuPG are not OpenPGP. GnuPG no longer conforms to modern OpenPGP standards and is the IE6 grade implementation that we should stop talking about and using and on that point at least moxie and I agree. I found a major CVE in gpg myself.
But regardless of tooling, it is about the keys and who holds them and who they endorse. It does not really matter how keys are distributed. It matters that keys signed other keys and that we have a way of downloading them and verifying that.
We cache a copy of all 5444 keys in the web of trust of stagex maintainers in our keys repo and you can draw a line from our keys to the keys that signed commits to the linux kernel today. These also sync and update from a dozen SKS keyservers that are still online for anyone that wants to build a key directory as we did.
Though SKS is being rapidly replaced with WKD where every domain hosts their own keys and they are automatically discovered.
Are you really going to say this has no trust or security value?
We should all just stop and let Github sign everything for us even though they don't full source bootstrap anything or sign commits or use deterministic builds?
What is the outcome you are actually arguing for here.
> It does not really matter how keys are distributed. It matters that keys signed other keys and that we have a way of downloading them and verifying that.
I think it matters if you want to call it a WoT. But also, I don't think any signatures originating from these keys are being verified usefully at any meaningful scale.
> Are you really going to say this has no trust or security value?
I think it has marginal security value, maybe net-negative if you balance it with the fact that cryptographers and cryptographic engineers have to waste time arguing against using PGP.
> What is the outcome you are actually arguing for here.
I like binary transparency. I also think identity-based signing is significantly more ergonomic, and has seen more adoption in the last 4 years than PGP has in the last 35. And I think this is actually a stunning indictment, because I'd say that identity-based signing schemes like Sigstore are still running behind my expectations.
> I think it matters if you want to call it a WoT. But also, I don't think any signatures originating from these keys are being verified usefully at any meaningful scale.
Web of trust is a web of mutually trusting keys, not a network of servers. That web can be verified on any computer as in the blog post by kron I linked earlier, and it is verified for every package install in our soon-to-be published sxctl tool we will be presenting at some conferences next month.
> I think it has marginal security value, maybe net-negative if you balance it with the fact that cryptographers and cryptographic engineers have to waste time arguing against using PGP.
So again, are you really saying all the maintainers of most services running the internet should stop using the only IETF standard built for human-identity-bound signing with keys held by those humans?
The alternative everyone seems to be suggesting with a straight face is login with github or google and let them sign for you with "keyless signing"? That is the only alternative that is gaining adoption, and it is a ridiculous downgrade. I consider it mostly security theater.
The whole point of humans holding their own signing keys locally is to be able to make it not matter if your centralized online accounts are taken over. Something that is usually easy to do because no one uses hardware 2FA or renews their personal email domains.
But, if they did use hardware 2FA, hey look they have a local signing key... why not just... sign the binaries with that hardware directly instead of using that to login and let someone else sign for you. And then if you are going to do that, you don't want to be impersonated, so why not publish those public keys, and have other maintainers sign them. And now we have re-invented the web of trust.
I don't think you are annoyed. You have done this to produce a reproducible linux distribution which your partners sell support for.
I wouldn't find this annoying at all - I would expect to have to do this for hundreds of packages.
Without unpaid volunteers things like Debian do not exist. Don't malign the situation and circumstances of other projects, especially if they are your competitors.
Compete by being better, not by complaining louder.
Sure, individual maintainers offer general purpose consulting services, but if we all did that for the next 20 years to keep the lights on we will never make a tiny fraction of the money we could have made by paywalling the binary artifacts like Chainguard and others do.
Stagex is and will forever be a community owned project.
>Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.
Unpaid volunteer hackers provide their work for free under licenses designed for the purpose of allowing companies like OpenAI to use their work without paying or contributing in any form. OpenAI wants to make the most money. Why would they spend any time or money on something they can get for free?
And at this point, it appears running code through an LLM to translate it eliminates copyright (and thus the licence), so $Anycorp can use it.
Our stuff is AGPL3 licenced and if this present trend continues we might just switch to MIT so at least the little guys can take advantage of it the way the big guys can.
Only if they provide the software or software as a service. Then I suspect it's good enough if the modifications or forks made are shared internally if software is used only internally, but on the other hand I'm not a lawyer.
Employers might argue that such internal use and distribution would fall under the “exclusively under your behalf” clause in the GPLv3, which is inherited by the AGPLv3.
This is the point. They can use and modify it, but they also have to share their modifications, i.e., help its development. Yet most megacorps never even touch this license.
Never let the left hand know what the right hand is doing. I suppose it works both ways here, but the specific end user is not why people make code available, it’s in the hope of improving things, even just the tiniest bit.
> Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.
Didn't the acquisition only happen a few weeks ago? Wouldn't it be more alarming if OpenAI had gone in and forced them to change their build process? Unless you're claiming that the article is lying about this being a description of what they've already been doing for a while (which seems a bit outlandish without more evidence), it's not clear to me why you're attributing this process to the parent company.
Don't get me wrong; there's plenty you can criticize OpenAI over, and I'm not taking a stance on your technical claims, but it seems somewhat disingenuous to phrase it like this.
I was just calling them by their new name, but yes clearly I am not the biggest fan of OpenAI and me invoking their name so soon betrays that. Sam altmans vision for handling the "proof of human" problem WoT solves is having everyone scan their eyes into magic orbs you can't audit at runtime and letting them sign stuff for us. Cool. I will take WoT over that every time.
Yeah, I'll just establish for the record that we've been thinking about this for a long time, and that it has nothing to do with anybody except our own interests in keeping our development and release processes secure.
I promise we are actively working on a much better solution we hope any distro can use, but... for now we just enforce signed merge commits by a different maintainer other than the author as something they only do for code they personally reviewed.
The biggest problem with crev is it is (last I checked) entirely centralized on git/github making it not all that useful for early supply chain dependencies that often just live on random tar files on servers, or svn, or cvs, or mercurial.
Crev also lacks support for public identity bound keys which would make us give up the highly valuable 25+ year web of trust we have built in identity bound keys that predate AI and cannot be easily impersonated.
Also they don't sign commits or reviews themselves because they think crev eliminates the need for such things, which I consider ridiculous.
I really like dpc and worked next to him when he was designing crev and tried to explain these exact problems, but in the end he wanted to ship something that only solved the limited set of problems he cared about at the time which was blessing rust packages on github, which he is of course entitled to do.
We will still certainly cite crev and we are incorporating some of what we feel are the good ideas such as the actual general shape of the reviews, confidence, etc.
>Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.
To be frank. Because more effort doesn't actually mean that something is more secure. Just because you check extra things or take extra steps that doesn't mean it actually results in tangibly better security.
If anyone from Astral sees this: at this level of effort, how do you deal with the enormous dependence on Github itself? You maintain social connections with upstream, and with PyPA... what if Github is compromised/buggy and changes the effect of some setting you depend on?
We talk to GitHub as well! You're right that they are an enormous and critical dependency, and we pay close attention to the changes they make to their platform.
What if? GitHub has is extremely buggy! I'm getting increasingly frustrated with the paper cuts that have become endemic across the entire platform. For example its not uncommon for one of our workflows to fail when cloning a branches of the repo they are running in.
I deliberately didn't mention this because I think most of the pain with Github over the last year is probably caused to some degree by their scale, which seems like an unrelated issue. (But maybe not.)
One (amongst other) big problem with current software supply chain is that a lot of tools and dependencies are downloaded (eg from GitHub releases) without any validation that it was published by the expected author. That's why I'm working on an open source, auditable, accountless, self hostable, multi sig file authentication solution. The multi sig approach can protect against axios-like breaches. If this is of interest to you, take a look at https://asfaload.com/
Yes, that's why I aim to make the checks transparant to the user. You only need to provide the download url for the authentication to take place. I really need to record a small demo of it.
Artifact attestation are indeed another solution based on https://www.sigstore.dev/ . I still think Asfaload is a good alternative, making different choices than sigstore:
- Asfaload is accountless(keys are identity) while sigstore relies on openid connect[1], which will tie most user to a mega corp
- Asfaload ' backend is a public git, making it easily auditable
- Asfaload will be easy to self host, meaning you can easily deploy it internally
- Asfaload is multisig, meaning event if GitHub account is breached, malevolent artifacts can be detected
- validating a download is transparant to the user, which only requires the download url, contrary to sigstore [2]
So Asfaload is not the only solution, but I think it has some unique characteristics that make it worth evaluating.
Overall I believe this is the right approach and something like this is what's required. I can't see any code or your product though so I'm not sure what to make of it.
The open source ecosystem has come very far and proven to be resilient. And while trust will remain a crucial part of any ecosystem, we urgently need to improve our tools and practices when it comes to sandboxing 3rd party code.
Almost every time I bump into uv in project work, the touted benefit is that it makes it easier to run projects with different python versions and avoiding clashes of 3rd dependencies - basically pyenv + venv + speed.
That sends a cold shiver down my spine, because it tells me that people are running all these different tools on their host machine with zero sandboxing.
Mainly the "project" system. I'm only developing python in my free time, not professionally so I'm not as well versed in its ecosystem as I would be in PHP. The fact that there's tons of way to have project-like stuff I don't want to deal with thoses. I used to do raw python containers + requirements.txt but the DX was absolutely not enjoyable. I'm just used to it now
Pinning github actions by commit SHA does not solve the supply chain problem if the pinned action itself is pulling in other dependencies which themselves could be compromised. An action can pull in a docker image as a dependency for example. It is effectively security theatre. The real fix is owning the code that runs in your CI pipelines. Or fork the action itself and maintain it as part of your infrastructure.
Shouldn't you always read & double-check the 3rd-party GitHub actions you use, anyway? (Forking or copying their code alone doesn't solve the issue you mention any more than pinning a SHA does.)
Double checking Github actions does not mitigate threats from supply chain vulnerabilities. Forking an action moves the trust from a random developer to yourself. You still have to make sure the action is pulling in dependencies from trusted sources which can also be yourself depending on how far you want to go.
Owning, auditing, and maintaining your entire supply chain stack is more secure than pinning hashes, but it is not practical for most projects.
Pinning your hashes is more secure than not pinning, and is close to free.
At the end of the day, the line of trust is drawn somewhere (do you audit the actions provided by GitHub?). It is not possible to write and release software without trusting some third party at some stage.
The important part is recognizing where your "points of trust" are, and making a conscious decision about what is worth doing yourself.
With the recent incidents affecting Trivy and litellm, I find it extremely useful to have a guide on what to do to secure your release process.
The advices here are really solid and actionable, and I would suggest any team to read them, and implement them if possible.
The scary part with supply chain security is that we are only as secure as our dependencies, and if the platform you’re using has non secure defaults, the efforts to secure the full chain are that much higher.
I maintain `repomatic`, a Python CLI + reusable workflows. It bakes most of the practices from this post into a drop-in setup for Python projects (uv-based, but works for others too). The goal is to make the secure default the easy default for maintainers who just want to ship packages. Also addresses a lot of GitHub Actions own shortcomings.
But thanks to the article I added a new check for the fork PR workflow approval policy.
The entire paragraph about version pinning using hashes (and using a map lookup for in-workflow binary deps) reminds me that software engineers are forever doomed to reinvent worse versions of nixpkgs and flakes.
I don't even love Nix, it's full of pitfalls and weirdnesses, but it provides so much by-default immutability and reproducibility that I sometimes forget how others need to rediscover this stuff from first principles every time a supply chain attack makes the news.
Nix provides declarative, reproducible builds. So, ostensibly, if you had your build system using Nix, then some of the issues here go away.
Unfortunately, Nix is also not how most people function. You have to do things the Nix way, period. The value in part comes from this strong opinion, but it also makes it inherently niche. Most people do not want to learn an entire new language/paradigm just so they can get this feature. And so it becomes a chicken and egg problem. IMHO, I think it also suffers from a little bit of snobbery and poor naming (Nix vs. NixOS vs. Nixpkgs) which makes it that much harder to get traction.
Nix, if not used incorrectly (and they really make it hard to use it, both correctly and incorrectly lol), gives you reproducible and verifiable builds.
Unfortunately I have to agree with the sibling comment that it suffers from poor naming and the docs are very hard to grok which makes it harder to get traction.
I really hate the idea of `it's all sales at the end of the day` but if Nix could figure how to "sell" itself to more people then we would probably have less of those problems.
I don't have much experience with GitHub's CI offering. But if this is an accurate description of the steps you need to take to use it securely ... then I don't think it can, in fact, ever be used securely.
Even if you trust Microsoft's cloud engineering on the backend, this is a system that does not appear to follow even the most basic principles of privilege and isolation? I'm not sure why you would even try to build "supply-chain security" on top of this.
I would agree with this. I recently tried to figure out how to properly secure agent-authored code in GitHub Actions. I believe I succeeded in doing this[1] but the secure configuration ended up being so delicate that I don’t have high hopes of this being a scalable path.
Now, as other commenter pointed out, maybe this is just inherent complexity in this space. But more secure defaults could go a long way making this more secure in practice.
[1] https://github.com/airutorg/sandbox-action
Yeah, this is usually where things break in practice
Out of curiosity, is there a build setup you have seen in the past that you think could be a good replacement for this complex GitHub CI setup? Asking for a friend ;)
Update: now I've finished reading the article, my impression is that complexity is mostly inherent to this problem space. I'd be glad to be proven wrong, though!
I think any of the webhook-based providers are better, because you can isolate your secrets. PRs go to a PR webhook that runs in an environment that just doesn’t have access to any secrets.
Releases go to the release webhook, which should output nothing and ideally should be a separate machine/VM with firewall rules and DNS blocks that prevent traffic to anywhere not strictly required.
Things are a lot harder to secure with modern dynamic infrastructure, though. Makes me feel old, but things were simpler when you could say service X has IP Y and add firewall rules around it. Nowadays that service probably has 15 IP addresses that change once a week.
The complexity comes from how the whole system is designed.
There’s no single repository or curated packages as is typical in any distribution: instead actions pull other actions, and they’re basically very complex wrapper around scripts which downloads binaries from all over the place.
For lots of very simple actions, instead of installing a distribution package and running a single command, a whole “action” is used which creates and entire layer of abstraction over that command.
It’s all massive complexity on top of huge abstractions, none of which were designed with security in mind: it was just gradually bolted on top over the years.
Yes, this problem space has inherent complexity, but no, this inherent complexity does not require Github's insanely insecure defaults and incoherent security model.
As a practical step, one could try using webhooks to integrate their github repo with literally any other CI provider. This would at least give you a single, low-coupling primitive to build your workflows on. It would not, in any way, eliminate the domain's inherent complexity (secrets, 3rd party contributions, trusted publishing, etc.), but it starts out safe because by default it doesn't do anything - it's just an HTTP call that gets fired under certain conditions.
If you have deterministic full source bootstrapped builds, it gets pretty simple and all of the third party trust can go away.
You build it, I build it, we get the same hash. It allows anyone to prove a published binary is a faithful compilation of given input source code.
This is no different from any package registry getting some packages compromised.
Not many of them allow for immutable relases. And if they do, nothing blocks you from releasing a patch version that will most likely be automatically pulled in by many many projects during build.
The whole dependencies ecosystem is currently broken. Thats why its so easy (relatively) to attack via supply-chain.
Only way to be really secured is to have own registry of vetted dependencies pinned to exact version and maintain own upgrade pipeline.
NOONE (beside google) is going to do that. Its too costly, you need two big teams just to handle that one part.
> NOONE (beside google) is going to do that. Its too costly, you need two big teams just to handle that one part.
And yet my team and I at stagex are building a decentralized code review system to handle this anyway. Not waiting around with our fingers crossed for the corpos to solve supply chain security for us. Has to be a community led effort.
The only binaries of uv in the world you can get that were full source bootstrapped from signed package commits to signed reviews to multi-signed deterministic artifacts are the ones from my teammates and I at stagex.
All keys on geodistributed smartcards held by maintainers tied to a web of trust going back 25 years with over 5000 keys.
https://stagex.tools/packages/core/uv/
Though thankful for clients that let individual maintainers work on stagex part time once in a while, we have had one donation ever for $50 as a project. (thanks)
Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.
I am annoyed.
This is the market telling you what matters.
OpenClaw has been an outstanding success, it is providing people the ability to leak their keys, secrets, and personal data, and allowing people to be subject to an incredible number of supply chain attacks when its users have felt their attack surface was just too low.
Your efforts have been on increasing security and reducing supply chain attacks, when the market is strongly signaling to you that people want reduced security and more supply chain attacks!
(I’m the author of TFA.)
> All keys on geodistributed smartcards held by maintainers tied to a web of trust going back 25 years with over 5000 keys.
Neither the age nor the cardinality of the key graph tells me anything if I don’t trust the maintainers themselves; given that you’re fundamentally providing third-party builds, what’s the threat model you’re addressing?
It’s worth nothing that all builds of uv come from a locked resolution and, as mentioned in TFA, you can get signed artifacts from us. So I’m very murky on the value of signed package commits that come from a different set of identities than the ones actually building the software.
StageX does reproducible builds, so they are signed independently and can also be verified locally. I don't think it applies to Astral, but it's useful for packages with a single maintainer or a vulnerable CI, where there is only one point of failure.
But I also think it'd be nice if projects provided a first-party StageX build, like many do with a Dockerfile or a Nix flake.
Once we have better support for multiarch in stagex, since StageX is distributed as OCI images, you could just replace your existing Dockerfile bases with stagex.
You definitely trust the same web of trust key graph already in every single layer of your current CI solution. Everything at Astral and by all indications also OpenAI is built with third party services, third party (blind) signing, using third party binaries signed by those 5000 keys directly or indirectly.
That web of trust is the trust foundation of the entire internet and likely every server that powers Github, Astral, and OpenAI including every CI system you described.
https://kron.fi/en/posts/stagex-web-of-trust/
One node in that graph is also nowhere near good enough to stop supply chain attacks, which is why we use -multiple- points thanks to full source bootstrapped deterministic builds.
Let me flip it and ask why anyone should trust that an Astral/OpenAI employee that does not sign their commits and does not sign their reviews, has not been impersonated or had an account takeover due to the phishable 2FA that is allowed, and won't just make a commit to CI stack for uv (or uv itself!) under a pseudonym then merge their pseudonym's code.
One person can burn it all down in spite of the practices in this blog post. Letting machines blindly sign whatever non-deterministic outputs come out of an automated process does not actually buy you much in practice against many of the supply chain attack tactics actually used in the wild. Also of course the same applies to the third party build systems you trust. Github themselves also don't use any of these basic supply chain security practices either so many many points of failure here.
Astral/OpenAI are actually giving -thousands- of randos other than the authors the ability to backdoor the uv binaries you produce, and without a reproducible full source bootstrapped build process, no one would be able to quickly or easily prove it.
To package or change uv in stagex one maintainer must sign the commit, and another must sign the review/merge commit. Then -multiple- maintainers must compile 180 bytes of human readable machine code, build up to tinycc, then gcc, then llvm, and eventually to a rust compiler, that we then use to build uv, all deterministically.
So, we actually don't trust any third parties other than the actual authors of the source code to a limited extent in our process. That said we are working on a solution for decentralized review of upstream code as well right now because we largely don't trust upstreams to not let their identities get stolen because most teams for whatever reason refuse to sign their commits and reviews, so we will have to do that for them too. Regardless, we can prove we faithfully deliver honest compilations of whatever upstream code is published without any single points of failure.
We ask users downloading binaries to trust that a bunch of maintainers are putting their personal reputations and keys (which long predate AI and are hard to impersonate) on the line to sign their bit for bit identical builds of uv, and the entire toolchain underneath it, and provide faithful compilations of upstream source code.
It would make everyone a lot safer if upstreams, especially well funded ones, could meet or exceed the threat model we must support downstream.
> You definitely trust the same web of trust key graph already in every single layer of your current CI solution. Everything at Astral and by all indications also OpenAI is built with third party services, third party (blind) signing, using third party binaries signed by those 5000 keys directly or indirectly.
I don't think we do; there are places we trust distribution signers, but we don't do so in a "web" topology; we trust them because a small set of keys is pre-baked into VMs, Docker images, etc. The web of trust, as it existed 20 years ago, is dead[1].
Topologically this is a lot like a CA ecosystem, except worse in material ways: even distros (full of talented, motivated people!) struggle to operationalize PGP, so we end up with a bunch of de facto unexpirable and irrevocable keys[2] that nobody is really tracking. Consequently, nobody is really factoring these into their security story, whether or not they're a web.
[1]: https://inversegravity.net/2019/web-of-trust-dead/
[2]: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1461834
You can call it dead, and yet, it is the only system signing internet infrastructure at scale right now because of Debian, Suse, Fedora, Arch, Gentoo, Ubuntu, Redhat, etc. It is very much alive for those use cases that need it most. If one of those keys was compromised or the people that hold them, most of the internet is backdoored. Except for things built on stagex, in which case you would have to compromise several. Still not good enough, but better.
PGP Web of Trust for all its faults and early design facepalms (of which there are many) is the only proof-of-human system where humans meet humans and sign each others keys that we ever built before AI. No one can reasonably expect any recently created keys were not created by made up LLM identities unless signed into the web of trust by well published existing keys held by well known and trusted humans.
But even if you don't want to look at the Web of Trust you can prove the key I sign stagex releases with is mine via all sorts of other ways thanks to keyoxide: https://keyoxide.org/E90A401336C8AAA9
Also PGP specs supports modern crypto now, attestation via dns, and even hackernews. You can attest my PGP key is tied to my HN profile right now. I would agree -gpg- is dead, with no real reason to use it anymore now that we have modern rust tooling with modern crypto.
But! If someone wants to generate an ssh key on a smartcard or something and sign with that instead, we would absolutely consider it. Not married to supporting only a single spec, but we absolutely need human beings to hold their own private keys on smartcards which are themselves attested by other human held private keys and the online services shared by the same identities.
No, I call it dead because it's dead. The SKS network is dead, the strong set is moribund, and the remaining real users of PGP are instead slinging key bundles around by baking them into pre-trusted artifacts (like ISOs). But that's not a "web of trust," it's just bespoke centralized key distribution with a certification format that every single serious cryptographer agrees is terrible.
(And this is before a more brute statistical argument: even at its greatest extent, the PGP ecosystem was minuscule[1].)
[1]: https://moxie.org/2015/02/24/gpg-and-me.html
I am deeply aware of Moxies views on this, and we have talked about them at length, and he is wrong. Also SKS and GnuPG are not OpenPGP. GnuPG no longer conforms to modern OpenPGP standards and is the IE6 grade implementation that we should stop talking about and using and on that point at least moxie and I agree. I found a major CVE in gpg myself.
But regardless of tooling, it is about the keys and who holds them and who they endorse. It does not really matter how keys are distributed. It matters that keys signed other keys and that we have a way of downloading them and verifying that.
We cache a copy of all 5444 keys in the web of trust of stagex maintainers in our keys repo and you can draw a line from our keys to the keys that signed commits to the linux kernel today. These also sync and update from a dozen SKS keyservers that are still online for anyone that wants to build a key directory as we did.
Though SKS is being rapidly replaced with WKD where every domain hosts their own keys and they are automatically discovered.
Are you really going to say this has no trust or security value?
We should all just stop and let Github sign everything for us even though they don't full source bootstrap anything or sign commits or use deterministic builds?
What is the outcome you are actually arguing for here.
> It does not really matter how keys are distributed. It matters that keys signed other keys and that we have a way of downloading them and verifying that.
I think it matters if you want to call it a WoT. But also, I don't think any signatures originating from these keys are being verified usefully at any meaningful scale.
> Are you really going to say this has no trust or security value?
I think it has marginal security value, maybe net-negative if you balance it with the fact that cryptographers and cryptographic engineers have to waste time arguing against using PGP.
> What is the outcome you are actually arguing for here.
I like binary transparency. I also think identity-based signing is significantly more ergonomic, and has seen more adoption in the last 4 years than PGP has in the last 35. And I think this is actually a stunning indictment, because I'd say that identity-based signing schemes like Sigstore are still running behind my expectations.
> I think it matters if you want to call it a WoT. But also, I don't think any signatures originating from these keys are being verified usefully at any meaningful scale.
Web of trust is a web of mutually trusting keys, not a network of servers. That web can be verified on any computer as in the blog post by kron I linked earlier, and it is verified for every package install in our soon-to-be published sxctl tool we will be presenting at some conferences next month.
> I think it has marginal security value, maybe net-negative if you balance it with the fact that cryptographers and cryptographic engineers have to waste time arguing against using PGP.
So again, are you really saying all the maintainers of most services running the internet should stop using the only IETF standard built for human-identity-bound signing with keys held by those humans?
The alternative everyone seems to be suggesting with a straight face is login with github or google and let them sign for you with "keyless signing"? That is the only alternative that is gaining adoption, and it is a ridiculous downgrade. I consider it mostly security theater.
The whole point of humans holding their own signing keys locally is to be able to make it not matter if your centralized online accounts are taken over. Something that is usually easy to do because no one uses hardware 2FA or renews their personal email domains.
But, if they did use hardware 2FA, hey look they have a local signing key... why not just... sign the binaries with that hardware directly instead of using that to login and let someone else sign for you. And then if you are going to do that, you don't want to be impersonated, so why not publish those public keys, and have other maintainers sign them. And now we have re-invented the web of trust.
I don't think you are annoyed. You have done this to produce a reproducible linux distribution which your partners sell support for.
I wouldn't find this annoying at all - I would expect to have to do this for hundreds of packages.
Without unpaid volunteers things like Debian do not exist. Don't malign the situation and circumstances of other projects, especially if they are your competitors.
Compete by being better, not by complaining louder.
Sure, individual maintainers offer general purpose consulting services, but if we all did that for the next 20 years to keep the lights on we will never make a tiny fraction of the money we could have made by paywalling the binary artifacts like Chainguard and others do.
Stagex is and will forever be a community owned project.
>Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.
Unpaid volunteer hackers provide their work for free under licenses designed for the purpose of allowing companies like OpenAI to use their work without paying or contributing in any form. OpenAI wants to make the most money. Why would they spend any time or money on something they can get for free?
Not sure if you're fully over the context that openAI bought Astral - who "own" uv.
Yep. Permissive licenses, "open source", it's all just free work for the worst corporations you can think.
It's free work for anyone.
Seems like the most cynical take on OSS possible.
Like anything good you do an evil person could benefit from - is the solution to never do any good?
The solution is to use AGPLv3.
I’m maybe daft but AGPLv3 doesnt prevent $Evilcorp from using it, they just need to share any modifications or forks they made?
And at this point, it appears running code through an LLM to translate it eliminates copyright (and thus the licence), so $Anycorp can use it.
Our stuff is AGPL3 licenced and if this present trend continues we might just switch to MIT so at least the little guys can take advantage of it the way the big guys can.
Only if they provide the software or software as a service. Then I suspect it's good enough if the modifications or forks made are shared internally if software is used only internally, but on the other hand I'm not a lawyer.
> if software is used only internally
Internal users are still users tho. They are entitled to see source code and license allows them to share it with the rest if of the world.
Employers might argue that such internal use and distribution would fall under the “exclusively under your behalf” clause in the GPLv3, which is inherited by the AGPLv3.
Oh, I guess it would. Ignore me.
This is the point. They can use and modify it, but they also have to share their modifications, i.e., help its development. Yet most megacorps never even touch this license.
Never let the left hand know what the right hand is doing. I suppose it works both ways here, but the specific end user is not why people make code available, it’s in the hope of improving things, even just the tiniest bit.
> Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.
Didn't the acquisition only happen a few weeks ago? Wouldn't it be more alarming if OpenAI had gone in and forced them to change their build process? Unless you're claiming that the article is lying about this being a description of what they've already been doing for a while (which seems a bit outlandish without more evidence), it's not clear to me why you're attributing this process to the parent company.
Don't get me wrong; there's plenty you can criticize OpenAI over, and I'm not taking a stance on your technical claims, but it seems somewhat disingenuous to phrase it like this.
I was just calling them by their new name, but yes clearly I am not the biggest fan of OpenAI and me invoking their name so soon betrays that. Sam altmans vision for handling the "proof of human" problem WoT solves is having everyone scan their eyes into magic orbs you can't audit at runtime and letting them sign stuff for us. Cool. I will take WoT over that every time.
Yeah, I'll just establish for the record that we've been thinking about this for a long time, and that it has nothing to do with anybody except our own interests in keeping our development and release processes secure.
That fits what I had assumed (and would expect), but it definitely doesn't hurt to have that confirmed, so thank you!
What are you using for signed reviews?
I promise we are actively working on a much better solution we hope any distro can use, but... for now we just enforce signed merge commits by a different maintainer other than the author as something they only do for code they personally reviewed.
Are you looking at crev at all?
https://github.com/crev-dev/
The biggest problem with crev is it is (last I checked) entirely centralized on git/github making it not all that useful for early supply chain dependencies that often just live on random tar files on servers, or svn, or cvs, or mercurial. Crev also lacks support for public identity bound keys which would make us give up the highly valuable 25+ year web of trust we have built in identity bound keys that predate AI and cannot be easily impersonated.
Also they don't sign commits or reviews themselves because they think crev eliminates the need for such things, which I consider ridiculous.
I really like dpc and worked next to him when he was designing crev and tried to explain these exact problems, but in the end he wanted to ship something that only solved the limited set of problems he cared about at the time which was blessing rust packages on github, which he is of course entitled to do.
We will still certainly cite crev and we are incorporating some of what we feel are the good ideas such as the actual general shape of the reviews, confidence, etc.
crev is on the StageX team's radar, and is rather close to ideal, but falls short in some aspects I don't recall at this point.
The private jet wont fuel itself now will it.
>Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.
To be frank. Because more effort doesn't actually mean that something is more secure. Just because you check extra things or take extra steps that doesn't mean it actually results in tangibly better security.
Exactly. Deterministic artifacts alone are not necessarily more secure and are tangential to a lot of what is being described in the blog post.
The blog is mostly focused on hardening the CI/CD pipeline.
FYI it was actually William Woodruff (the article author) and his team at Trail of Bits that worked with PyPI to implement Trusted Publishing.
If anyone from Astral sees this: at this level of effort, how do you deal with the enormous dependence on Github itself? You maintain social connections with upstream, and with PyPA... what if Github is compromised/buggy and changes the effect of some setting you depend on?
We talk to GitHub as well! You're right that they are an enormous and critical dependency, and we pay close attention to the changes they make to their platform.
> what if Github is compromised/buggy
What if? GitHub has is extremely buggy! I'm getting increasingly frustrated with the paper cuts that have become endemic across the entire platform. For example its not uncommon for one of our workflows to fail when cloning a branches of the repo they are running in.
I deliberately didn't mention this because I think most of the pain with Github over the last year is probably caused to some degree by their scale, which seems like an unrelated issue. (But maybe not.)
One (amongst other) big problem with current software supply chain is that a lot of tools and dependencies are downloaded (eg from GitHub releases) without any validation that it was published by the expected author. That's why I'm working on an open source, auditable, accountless, self hostable, multi sig file authentication solution. The multi sig approach can protect against axios-like breaches. If this is of interest to you, take a look at https://asfaload.com/
I’m maybe not understanding here, but isn’t it the point of release attestations (to authenticate that the release was produced by the authors)?
[0] https://docs.github.com/en/actions/how-tos/secure-your-work/...
The problem is nobody checks.
All the axios releases had attestations except for the compromised one. npm installed it anyway.
Yes, that's why I aim to make the checks transparant to the user. You only need to provide the download url for the authentication to take place. I really need to record a small demo of it.
Artifact attestation are indeed another solution based on https://www.sigstore.dev/ . I still think Asfaload is a good alternative, making different choices than sigstore:
- Asfaload is accountless(keys are identity) while sigstore relies on openid connect[1], which will tie most user to a mega corp
- Asfaload ' backend is a public git, making it easily auditable
- Asfaload will be easy to self host, meaning you can easily deploy it internally
- Asfaload is multisig, meaning event if GitHub account is breached, malevolent artifacts can be detected
- validating a download is transparant to the user, which only requires the download url, contrary to sigstore [2]
So Asfaload is not the only solution, but I think it has some unique characteristics that make it worth evaluating.
1:https://docs.sigstore.dev/about/security/
2: https://docs.sigstore.dev/cosign/verifying/verify/
> without any validation that it was published by the expected author
SPOF. I'd suggest use automatic tools to audit every line of code no matter who the author is.
Overall I believe this is the right approach and something like this is what's required. I can't see any code or your product though so I'm not sure what to make of it.
Here's the GitHub repo of the backend code: https://github.com/asfaload/asfaload
There's also a spec of the approach at https://github.com/asfaload/spec
I'm looking for early testers, let me know if you are interested to test it !
The open source ecosystem has come very far and proven to be resilient. And while trust will remain a crucial part of any ecosystem, we urgently need to improve our tools and practices when it comes to sandboxing 3rd party code.
Almost every time I bump into uv in project work, the touted benefit is that it makes it easier to run projects with different python versions and avoiding clashes of 3rd dependencies - basically pyenv + venv + speed.
That sends a cold shiver down my spine, because it tells me that people are running all these different tools on their host machine with zero sandboxing.
meh not always. I do use uv IN docker all the time, its quite handy
Honest question - what are the main benefits for you when you use it in docker?
ps. I feel like I've been doing python so long that my workflows have routed around a lot of legit problems :)
For us, the DX of uv for dependency management is much better than just using pip and requirements.txt.
To be clear though, we only use uv in the builder stage of our docker builds, there is no uv in the final image.
Main reason I now use uv is being able to specify a cool down period. pip allows it but it's with a timestamp so pretty much useless..
And that doesn't prevent me from running it into a sandbox or vm for an additional layer of security.
> pip allows it but it's with a timestamp
A PR to be able to use a relative timestamp in pip was merged just last week
https://github.com/pypa/pip/pull/13837/commits
Mainly the "project" system. I'm only developing python in my free time, not professionally so I'm not as well versed in its ecosystem as I would be in PHP. The fact that there's tons of way to have project-like stuff I don't want to deal with thoses. I used to do raw python containers + requirements.txt but the DX was absolutely not enjoyable. I'm just used to it now
Pinning github actions by commit SHA does not solve the supply chain problem if the pinned action itself is pulling in other dependencies which themselves could be compromised. An action can pull in a docker image as a dependency for example. It is effectively security theatre. The real fix is owning the code that runs in your CI pipelines. Or fork the action itself and maintain it as part of your infrastructure.
We do address this in the article! It's defense in depth, not theater.
We audit all of our actions, check if they pull in mutable dependencies, contribute upstream fixes, and migrate off using any action when we can.
(I work at Astral)
Do you fork them and have a team maintaining forks ?
If not you cant be sure of anything. Its just a security theater.
Shouldn't you always read & double-check the 3rd-party GitHub actions you use, anyway? (Forking or copying their code alone doesn't solve the issue you mention any more than pinning a SHA does.)
Double checking Github actions does not mitigate threats from supply chain vulnerabilities. Forking an action moves the trust from a random developer to yourself. You still have to make sure the action is pulling in dependencies from trusted sources which can also be yourself depending on how far you want to go.
> It is effectively security theatre.
I disagree. Security is always a trade-off.
Owning, auditing, and maintaining your entire supply chain stack is more secure than pinning hashes, but it is not practical for most projects.
Pinning your hashes is more secure than not pinning, and is close to free.
At the end of the day, the line of trust is drawn somewhere (do you audit the actions provided by GitHub?). It is not possible to write and release software without trusting some third party at some stage.
The important part is recognizing where your "points of trust" are, and making a conscious decision about what is worth doing yourself.
With the recent incidents affecting Trivy and litellm, I find it extremely useful to have a guide on what to do to secure your release process.
The advices here are really solid and actionable, and I would suggest any team to read them, and implement them if possible.
The scary part with supply chain security is that we are only as secure as our dependencies, and if the platform you’re using has non secure defaults, the efforts to secure the full chain are that much higher.
I'm in the process of standing up a new container registry, going to be studying this closely.
I maintain `repomatic`, a Python CLI + reusable workflows. It bakes most of the practices from this post into a drop-in setup for Python projects (uv-based, but works for others too). The goal is to make the secure default the easy default for maintainers who just want to ship packages. Also addresses a lot of GitHub Actions own shortcomings.
But thanks to the article I added a new check for the fork PR workflow approval policy.
More at: https://github.com/kdeldycke/repomatic
This is a really great overview; what a useful resource for other open-source projects.
The entire paragraph about version pinning using hashes (and using a map lookup for in-workflow binary deps) reminds me that software engineers are forever doomed to reinvent worse versions of nixpkgs and flakes.
I don't even love Nix, it's full of pitfalls and weirdnesses, but it provides so much by-default immutability and reproducibility that I sometimes forget how others need to rediscover this stuff from first principles every time a supply chain attack makes the news.
>worse versions of nixpkgs and flakes
You mean statically-compiled binaries and hash pinning? Those have been around a bit longer than Nix :-)
Were they deployed at scale in such a way that most (open and some non-free) software is packaged as such? I've never seen this happen until nixpkgs.
Every generation thinks they invented sex. And hash pinning, which now sounds dirty.
Earlier submission from author: https://news.ycombinator.com/item?id=47691466
Super useful info... but I feel so tired after reading it
Lengths people will go to rediscover Nix/Guix is beyond me
If it doesn't work on Windows, it is not a full replacement.
Isn't Nix just reinventing what Vesta did for software reproducibility decades earlier? https://vesta.sourceforge.net/
I don't see the connection though?
Nix provides declarative, reproducible builds. So, ostensibly, if you had your build system using Nix, then some of the issues here go away.
Unfortunately, Nix is also not how most people function. You have to do things the Nix way, period. The value in part comes from this strong opinion, but it also makes it inherently niche. Most people do not want to learn an entire new language/paradigm just so they can get this feature. And so it becomes a chicken and egg problem. IMHO, I think it also suffers from a little bit of snobbery and poor naming (Nix vs. NixOS vs. Nixpkgs) which makes it that much harder to get traction.
There are different notions of "reproducible". Nix does not automatically make builds reproducible in the way that matters here:
https://reproducible.nixos.org
It is still good at that but the difference to other distros is rather small:
https://reproducible-builds.org/citests/
Nix, if not used incorrectly (and they really make it hard to use it, both correctly and incorrectly lol), gives you reproducible and verifiable builds.
Unfortunately I have to agree with the sibling comment that it suffers from poor naming and the docs are very hard to grok which makes it harder to get traction.
I really hate the idea of `it's all sales at the end of the day` but if Nix could figure how to "sell" itself to more people then we would probably have less of those problems.
Reading the paragraph on hash pinning and "map lookup files" (lockfiles) made me audibly sigh.
Created an agent skill based on this blog. Assessing my own repos now.
https://github.com/backnotprop/oss-security-audit