I wonder why this was flagged. Maybe too controversial an opinion for HN? It is rather unconventional.
Though this isn't directly related to the topic, I do personally believe that too much updating can be a bad thing. We've been fed this narrative to always update because there are so many exploits that need patching all the time. This is certainly not untrue, but it's also not that simple. I think what we almost always ignore is the cost in the churn of both software development (or just patching bugs) and updates being constantly pushed. Every change to code adds the potential for a new exploit, even if a change was intended to fix an existing exploit. The more often you pull updates to a piece of software, the more often you are opening yourself up to either a new exploit or an outright malicious piece of code being snuck in. You also open yourself up to bugs that, while not necessarily exploitable, can cause catastrophic failures or features being broken indefinitely, as we have witnessed time and again with Microslop.
When it comes to developer dependencies specifically, my philosophy is to try and only use libraries that are simple enough that the likelihood they will actually need to be updated is minimal. I also tend to avoid updates for things that I don't truly need, such as a new framework feature. If the current version of a framework I'm using is working totally fine, then I don't necessarily have a reason to update and migrate to a new version just because other developers have introduced a new opinion for how their users should write their applications. Eventually I might do an upgrade, but by not upgrading to every single major, minor, or fix version, the fewer chances there are of bad code ruining my day (or career). Of course I don't treat all software the same. If I'm relying on cryptographic libraries for the purposes of security or privacy, then it makes sense to upgrade rather often.
In my personal life, I rarely update anything like my OS or individual pieces of software. I've had this attitude for over a decade and so far I've yet to have noticed my systems being hacked or my bank accounts being drained. As far as I can tell, the need to always update is massively overrated.
IMO the best practice is to leave dependencies unpinned, but use a lock file, and only update the lock file a few times a year.
Upgrade enough that you don’t get stuck, but not often enough to expose yourself to supply chain attacks every time CI runs.
It's another startup sales pitch. Their argument is the entire ecosystem is screwed, but it's okay, you can run their uber complicated toolchain in a CI pipeline and that will fix everything, by reviewing all of the code of your dependency updates. (because all the other backdoored code was never reviewed? because you're better at reviewing upstream code than the upstream maintainers?)
My take is the "hot take" nobody likes to hear. I think you should actually follow standard security best practices. Don't update constantly to the latest bleeding edge versions, but do update to the latest security patched versions. Do pin your versions (and SHA hashes of releases). Do keep an artifact repository. Do cryptographic verification of artifacts. Do validate every dependency you add, understand who made it, what it does. Do try to minimize your dependencies. Do review every new dependency and see what it is you're pulling into your application and whitelist the sources and their signed keys. Do use code signing. Do use OAuth, ephemeral keys, MFA, certificates.
Linux distributions have been secure for a long time by following these practices. Even their unstable branches. If they can do it, you can do it.
> The old operating model was indeed fine in a much smaller, simpler tech world,
The thing with the 'old days' that does not happen now is developers would go out of their way to make sure new releases were API backward compatible. Now it seems anything goes. If the API changed, a new function may be created or the old function would use variable arguments to allow the older functionality.
With that said, providing backward compatibility seems to be hard these days due to a BOM (Bill of material/dependencies) that make a BOM of submarine look simple.
That ssh/xz issue kind of highlighted how complex things have gotten.
Usually, there are CI/CD pipelines, but also change and testing regimes of some sort. Dev/QA too before prod.
Here is my suggestion though:
- Don't use package repos on the internet, host local caches
- Update your package caches twice a week, not more frequently
- Unpin and update packages in dev as it makes sense, as soon as the package repos update
- QA whenever your team can actually do testing
- Prod, with your next release, no more frequently than 90 days
There is a reason public CVE disclosure is ballpark 90 days per ethical security research and testing norms. That's how long proper testing and release is expected to take.
If you have no specific reason to update a package, and it passes integration tests with more up to date packages, then don't update. Supply chain compromises aside, bugs have a way of creeping. Things get missed with tests. Either you take this careful approach, or you take the other extreme approach where you design everything so well and with so much costly infrastructure and people to manage it, that impact is priced into the availability and threat modeling, which almost no one can pull off except on paper (well.. I said almost...).
I wonder why this was flagged. Maybe too controversial an opinion for HN? It is rather unconventional.
Though this isn't directly related to the topic, I do personally believe that too much updating can be a bad thing. We've been fed this narrative to always update because there are so many exploits that need patching all the time. This is certainly not untrue, but it's also not that simple. I think what we almost always ignore is the cost in the churn of both software development (or just patching bugs) and updates being constantly pushed. Every change to code adds the potential for a new exploit, even if a change was intended to fix an existing exploit. The more often you pull updates to a piece of software, the more often you are opening yourself up to either a new exploit or an outright malicious piece of code being snuck in. You also open yourself up to bugs that, while not necessarily exploitable, can cause catastrophic failures or features being broken indefinitely, as we have witnessed time and again with Microslop.
When it comes to developer dependencies specifically, my philosophy is to try and only use libraries that are simple enough that the likelihood they will actually need to be updated is minimal. I also tend to avoid updates for things that I don't truly need, such as a new framework feature. If the current version of a framework I'm using is working totally fine, then I don't necessarily have a reason to update and migrate to a new version just because other developers have introduced a new opinion for how their users should write their applications. Eventually I might do an upgrade, but by not upgrading to every single major, minor, or fix version, the fewer chances there are of bad code ruining my day (or career). Of course I don't treat all software the same. If I'm relying on cryptographic libraries for the purposes of security or privacy, then it makes sense to upgrade rather often.
In my personal life, I rarely update anything like my OS or individual pieces of software. I've had this attitude for over a decade and so far I've yet to have noticed my systems being hacked or my bank accounts being drained. As far as I can tell, the need to always update is massively overrated.
IMO the best practice is to leave dependencies unpinned, but use a lock file, and only update the lock file a few times a year. Upgrade enough that you don’t get stuck, but not often enough to expose yourself to supply chain attacks every time CI runs.
It's another startup sales pitch. Their argument is the entire ecosystem is screwed, but it's okay, you can run their uber complicated toolchain in a CI pipeline and that will fix everything, by reviewing all of the code of your dependency updates. (because all the other backdoored code was never reviewed? because you're better at reviewing upstream code than the upstream maintainers?)
My take is the "hot take" nobody likes to hear. I think you should actually follow standard security best practices. Don't update constantly to the latest bleeding edge versions, but do update to the latest security patched versions. Do pin your versions (and SHA hashes of releases). Do keep an artifact repository. Do cryptographic verification of artifacts. Do validate every dependency you add, understand who made it, what it does. Do try to minimize your dependencies. Do review every new dependency and see what it is you're pulling into your application and whitelist the sources and their signed keys. Do use code signing. Do use OAuth, ephemeral keys, MFA, certificates.
Linux distributions have been secure for a long time by following these practices. Even their unstable branches. If they can do it, you can do it.
Agree on how scary it is to do npm upgrade right now.
But I don't understand why Dependabot is bad but Mendral is good. Why Dependabot will be an attack vector and Mendrol will not?
I think a way to do is to reduce how many dependencies are needed.
All of this is really about web dev, not s/w dev in general.
npm and pip are curses on the planet.
One point right from the beginning of the article: Why would you EVER leave a public port open straight to phpMyAdmin?
This is what SSH tunneling is for...
> The old operating model was indeed fine in a much smaller, simpler tech world,
The thing with the 'old days' that does not happen now is developers would go out of their way to make sure new releases were API backward compatible. Now it seems anything goes. If the API changed, a new function may be created or the old function would use variable arguments to allow the older functionality.
With that said, providing backward compatibility seems to be hard these days due to a BOM (Bill of material/dependencies) that make a BOM of submarine look simple.
That ssh/xz issue kind of highlighted how complex things have gotten.
you just update them and that's it?
Usually, there are CI/CD pipelines, but also change and testing regimes of some sort. Dev/QA too before prod.
Here is my suggestion though:
- Don't use package repos on the internet, host local caches - Update your package caches twice a week, not more frequently - Unpin and update packages in dev as it makes sense, as soon as the package repos update - QA whenever your team can actually do testing - Prod, with your next release, no more frequently than 90 days
There is a reason public CVE disclosure is ballpark 90 days per ethical security research and testing norms. That's how long proper testing and release is expected to take.
If you have no specific reason to update a package, and it passes integration tests with more up to date packages, then don't update. Supply chain compromises aside, bugs have a way of creeping. Things get missed with tests. Either you take this careful approach, or you take the other extreme approach where you design everything so well and with so much costly infrastructure and people to manage it, that impact is priced into the availability and threat modeling, which almost no one can pull off except on paper (well.. I said almost...).