Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.
> It's not physically possible to run post-mortems for issues at those rates.
Not at all, you merely move the goal post of at what layer the "root cause" actually could come from! At that speed, it's always something short and sweet, while when you actually want to long-term address things, you have to have time to even investigate organizational issues or whatever the actual problems stem from.
But you have half a day? "Post-mortem: Push X wasn't properly analyzed before deployment, in future more testing" and call it a day.
“A failure occurred. This was caused by something going wrong. Changes to operating guidelines have been instituted to ensure that things will not go wrong in the future unless we happen to do the same thing again.”
FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.
GH going down used to be quite rare. If it failed to load I'd spend a bunch of time trying to figure out what was wrong with my internet connection, just to read on HN that it was down for everyone.
This week GH failed to load and I automatically assumed it was a GH issue - just for it to be followed up a few minutes later by a marketing coworker complaining about internet connectivity. Turns out the office internet connection was dropping about 50% of all packets.
It is bad enough that business-side managers are noticing that GH issues are slowing work down. That would've been unimaginable a few years ago.
One tangent, I believe sev-0 is actually "critical" (at least as how I'm used to reading it), and the higher you go the less critical something is.
IMO as a github-watcher, I think they changed their definition of what constitutes a sev-0 between sev-1 for the better. In particular, they had a few "sev-1"'s around the turn of the year that would be classified as sev-0's if they happened today.
Pre-4/22 GitHub sev-1 was a normal SaaS company's sev-0, imo. So I think their new system is more reflective of reality. My guess is that a few of their big customers bullied them to have more accurate SEV categorization.
To be clear, your observation that "they changed their definition of what constitutes a sev-0" is based just on your external observation of incidents and their designations, correct? I.e. they haven't officially released a statement saying they have changed their standards
Is it? Seems a text description of "Make a website outlining 'How cooked GitHub' is with a modern style" to basically any LLM would produce exactly that UI and design, literally nothing of that design a human had any influence on, besides the ones selecting what training data the used LLMs was trained with.
I think most of us who've tried using LLMs for web-design can recognize that style and design at this point, regardless of model actually used.
Oh wow, I'm in the position to be able to give a peek behind the curtain of something (validly!!) critiqued as AI slop! Exciting.
I originally made the core data functionality of this site for myself because I was curious what the uptime stats for each service were (I build something that heavily depends on GitHub), and to viz the distribution/severity of those incidents, again per-service, over time.
It involved a lot of back-and-forth, and is not a one-shotter; maybe closer to 40-50 shots over maybe ~10 hours of human time. A couple memorable things that made it complicated, irrespective of the UI: sneaky bugs around double-counting time for overlapping incidents, no GitHub API for incidents so you need to puppeteer-scrape the backlog of incidents to get historical data. Although, you all are right to call out that the CSS was three shots, though, and it shows :) I thought it looked so cool in ~January 2026 and now it gives me the ick, too!
For people who are curious about how much direction went into the information architecture/presentation, it was fairly substantial. I wanted a contribution graph style viz and it took many turns to get it working the way I wanted. The swimlane viz for selected-day-incident visualization was also me, because I love swimlane graphs.
I ended up sharing it with some folks and they wanted to reference it, so I put it on a website. So it's jokey for sure, but I take my jokes seriously! I'm grateful that people have feedback on how it can better functionally and visually :)
> Although, you all are right to call out that the CSS was three shots, though, and it shows :)
Totally, my comment was all about the styling and design literally, and is in no way a comment about the data or actual contents of the website, hope you didn't take it that way as well, as it does seem proper in that regard!
Thank you for sharing it, and even greater thank you for sharing the process behind building it, for me that's more interesting almost :)
Compared to near unusable pages that large organizations produce, yes this page is highly effective at conveying information. Who cares how it was produced?
Well, we're at least two people who care, since we were conversing about how good/bad the webdesign is, then you jumped in here :) If you don't care, why bother to reply to people who seemingly do care? What kind of conversation are you expecting here, "Yeah, do tooo"? :|
Why I care about understanding people who think differently than me? I don't know, always did, not sure exactly why, always been interesting to understand people's perspectives, especially when I personally feel differently, tends to help myself understand me better too, so it's basically a win-win to get people to explain their reasoning.
A lack of effort put into presentation is a signal for low quality. In the past, this might've manifested in using default Wordpress templates, and now it manifests as stock LLM templates instead. Can you make high quality content and present it on a stock template? Sure, you can, but without any prior reason to believe that you put more effort into the content than the template, I'm liable to believe the content is LLM-generated slop as well and therefore untrustworthy and not worth my time.
In this case, it appears more effort was put into content than presentation, which is a possibility and the creator is in the comments saying as such, but humans operate on heuristics by default. The majority of sites that look like this have been a complete waste of my time and I usually just click away at this point.
> this page is highly effective at conveying information
Is it though? If the page is near unreadable?
* Almost pure-black background rendering every not-pure-white colour barely readable
* Dark-grey and low saturation colours used almost everywhere, for both fonts and other coloured elements (the orange cells in the calendar are the most readable thing)
* Thin fonts - coupled with the dark grey colours this just adds to the readability issues
* Yet another incredibly long info-dump of a page
And then as far as actual information:
* Vanity metrics as the main information, that is a lot of things with no context or historical information
* A lot of aggregates and rollups that aren't that useful
No, I haven't tried Reader Mode.
It's a good demo for UI state syncing though, I'll give it that.
Outside design systems I rarely get good CSS from LLMs.
3D type stuff too, it's useless outside boilerplate.
Very little spatial reasoning training, no end-user subjective reasoning inference (Google is starting to though even in unrelated chats), so it's no surprise the LLM doesn't know what you want.
Since I don't even know what I want half the time until I saw it, the subjective reasoning piece is key - that is, being able to predict what I'll want to pretty good accuracy. Then you have your agents etc.
Have an opinion on the design, imagine something, then tell it to do just that, then iterate. It's when you're unspecific you get the generic, bland and typical LLM design, you just have to be subjective and influence it in some (human) direction.
Here is a provocative thought - maybe these are the so-called "better designs" from LLMs? It's not like writing English sentences is some huge secret you are sitting on that no one else knows.
> It's not like writing English sentences is some huge secret you are sitting on that no one else knows.
I'd actually say what really makes an excellent engineer stick out among many great engineers, is their ability to communicate clearly and knowing what needs to be communicated vs not, basically being way better at language and communication in general, and they also understand the important of it.
My point being that not everyone writes as good prompts as everyone else, the way you communicate, how clearly and how exact you are matters a lot, much more than you seemingly is under the impression of.
Same goes with the "LLM does web design" example from before, a web designer with great communication skills in web design, will (naturally) have a better prompt for something that'll potentially could look good, compared to a web designer that isn't at good at communicating what they actually want.
I'm deleting my GitHub repos today (been planned for a bit) in favour of a local Forgejo Git. I also have not experienced any service disruption since I migrated well over a month ago.
Can you elaborate on how your Forgejo frontend will be different than the default one? I'm asking because I've only ever used GitHub, GitLab and Forgejo for longer periods and Forgejo was the fastest and easiest to use for me.
Most part screen is taken by picture.
Contrast ratio is really low. Hard to read
Should they remove that useless banner, current status which is the most interesting part coud've been made visible right away.
Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.
I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch.
I don't envy their position of having to scale that fast on something that has to be instant and real-time. As far as I know, you can't do CDN/edge caching shenanigans with a remote git repository like Google can with a YouTube video. It's gotta always be reading/writing to the latest, single source of truth.
Sure, backseat commenting is easier and I wouldn't wanna be in charge at github right now, but on the other side there also a reality where we'd see video essays about githubs downfall because their reliability crashed so hard that businesses could not trust them and moved to competitors / self hosted instances which then meant less paid users to subsidize the ever growing demand of the free users.
It’s not quite as cacheable as YouTube, but a lot of it is still pretty cacheable. Actions aren’t. Issues, wikis, and READMEs are. File views are. Most projects aren’t changing daily. The few that change daily aren’t changing hourly. There are a few that change constantly and would require constant cache updates. But the long tail is pretty static.
Yes it's potentially a write-heavy workload which also needs to be consistent aka the worst case scenario.
The easy solutions like caching and read replicas don't work and you're forced to go the route of sharding or similar techniques that have much more painful tradeoffs.
I'm not sure if that's why everything keeps breaking but at that scale write-heavy workloads are never going to be easy
However, they have reported numbers along rather inconsistent dimensions. Like, historically they've focused on number of repos and users and later PR's and issues, and often catch-all terms like "contributions" which includes all of those + comments etc... but the number of commits alone (which apparently is the main culprit now?) has been mentioned very sporadically. This has made it hard to get a consistent sense of historical growth.
Without any other information, however, it is reasonable to assume that a 14x in commits is the prime candidate for instability. Especially since commits are write traffic, which is much harder to scale than read traffic. Plus every 3 - 5x increase in scale can reveal bottlenecks in your distributed systems that you never knew existed, so they probably have like 2 - 3 "generations" of bottlenecks to figure out!
They are highly responsible for all of that.
They are diversified a lot with a lot of random things instead of focusing on their core business. They have actively pushed people to use the service and feature more.
Think about countless actions that have to run almost at every push and PR push!
Also, remember that we were used to use external services for "actions", and they basically killed the competition by offering their own CI actions at no cost to most users.
Also, they did a lot of reworks in the last years, not necessarily for the best like the PR diff page, and probably not in the most efficient way.
They are already cooked as this has been happening ever since the Microsoft acquisition and it was run to the ground before 2023.
At this point you would get better uptime by just self-hosting your own GitLab, Forgejo or Codeberg instance instead of dealing with Github's unreliablity.
There is no defending them with their clear neglet and carelessness of the platform.
If all you need is a repository, you don't even need any of these. You need SSH access to a server, and optionally, one of several web front-ends. Git comes with a CGI script that handles public anonymous checkouts via HTTP(S), although since nginx doesn't support CGI, integrating those is a little bit tricky as you need a FastCGI wrapper.
I moved most of my projects off GitHub to Forgejo and will be using Tangled too for public repositories. I don’t think people realize that if you self host Forgejo, you get 99% of the functionality of GitHub with zero of the limitations. Especially if you have the hardware to spare for CI runners. And if self hosting isn’t your thing you can always just use Codeberg and Tangled directly.
I’m working on an open source Forgejo browser called Joui. It’s coming along nicely, and is so much snappier than GitHub in every single way.
It looks like it is the number of consecutive days with no incident. If you look at 31 Dec 2025, that corresponds to an 8-day period with no incidents.
Java or Bedrock edition, and have you tried logging into your EntraID Microsoft Teams for Xbox account lately? Make sure to check the box to keep you logged in!
And if someone makes a server that doesn't do the chat verification, Microsoft blacklists that server in the client-side server address textbox. This system was developed to destroy pay-to-win servers, but they're now applying it against servers that refuse to censor "fuck".
Not as bad as it is now. All I see are suggested posts from people I never connected with and those are full of instagramesque self-promoting banal vibes.
Has nothing to do with Microsoft acquisition... AI usage has increased demand and load. More PRs, more Action runners, more of everything firing. GitHub just wasn't ready for the scale and are now having issues catching up with it as it continues to increase exponentially.
This is a convenient lie that GH likes to tell. Growth is nothing like exponential, its at most 300% over several years according to their own public numbers (presented misleadingly on graphs)
But a couple of years ago they were crowing about how much work they were doing to prepare for “a billion developers”. If they had actually done that then the actual load from agents should have been no problem.
There was an x post in another thread under this post that showed all the standard usage numbers are way up: 14x, 2.1x, etc. And the OP hinted at the usage growth being non-linear for 2026
Yeah, that and Microsoft has been slow to move the infrastructure to something that scales better to handle that load.
The more surpassing part is that Microsoft hasn't figured out a way to manage/contain the AI-sourced traffic better so it doesn't create all this noisy neighbor problems for non-AI usage/users.
Github's core platform doesn't really make that separation, anything a human can leverage on github an AI agent can as well, just faster and with heavier usage. End of day agents and humans are using the same services.
Sure, still need to enable access the same info but feels like bucketing the clients into
bucket1 = clients that were working just fine before (users and whatever automation they had in place)
bucket2 = ai clients that contributed to, if not flat out caused, the scale problems
then slowing down/limiting the bucket2 clients while keeping the bucket1 clients rolling as-is, is both doable and keeps existing customers happy while the underlying infra gets scale/perf improvements needed to support ai clients at scale.
FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.
MSFT was pretty arms length for the first 5-6 years. I was honestly kind of impressed and it made my opinion of MSFT better. But then AI made it too attractive of a target and MSFT couldn't help but make it a place the former CEO wanted to leave (and it has been running headless for about a year now).
It's quite disappointing objectively, but I expected worse from MSFT.
I wouldn't know, somehow this game I bought maybe 15 years ago is no longer playable for me, my account was supposed to be migrated from Mojang to Microsoft or similar, but then that never happened or something, and trying to login now asks me to contact Microsoft support, which I've tried 3-4 times, never had anyone respond to me so who knows how the game is today? I stopped trying at this point...
Personally, once a game I own is janked from my hands because of organizational decisions, that's the time I'll stop consider the game "in good shape", but I'm sure the people who had to buy the same game a second time still enjoy it.
Yes, the account migration was a mess. Support response times were at least 30 days, if you ever actually received a response at all (I never did). I did buy the game a second time in order to play with my kids.
They deleted my account from 2010 because I didn't convert it to a Microsoft one. They baked an incredibly aggressive chat filter into multiplayer, even if you're not playing on official servers. They've added microtransactions for things that we previously free (skins, resource packs). They force you into their shitty, bloated, user-hostile launcher with adverts.
It's been nonstop content-slop since the acquisition. New mobs, new blocks, new items, new blocks, new items, new mobs, new mobs, new biomes. Some of them are good but the totality of adding a bunch of stuff has been to destroy the simplicity that was one of the draws of the original game. Now it's an exploration and niche-mechanics-exploitation game more than a virtual legos game. You don't go mining any more, you find trading loops with villagers.
This was happening to some degree pre-acquisition, but since the acquisition it's been this non-stop.
Some of it's good. The Nether and the oceans were really boring before their respective updates.
They should have called Minecraft "done" around the acquisition time and started on Minecraft 2.
Like those aviators who draw a picture on flightradar24, if you filter by All Services - Critical, somebody almost about to draw a swastika just in May... Are the AI agents revolting?
This is getting ridiculous. One particularly concerning thing I’m seeing is that pull requests on both the web UI and API aren’t reflecting all commits or branch changes consistently. It would be very easy to merge something without realizing you’re not actually reviewing the full diff.
Yeah, I've had several occasions recently (seemingly not related to any incidents on the status page) where I've had to wait 20 minutes to an hour to be able to open a PR, because Github didn't recognize my branch had any new commits compared to the base branch.
New PR: revert GitHub software and infrastructure to version of June 1st, 2018.
New PR: disable new user signups for 6 months
HR initiative: all future KPIs automatically require three-nines availability; all bonuses are forfeited, regardless of accomplishments, if annual availability falls below target
GitHub’s instability started when they announced they were wholesale migrating to azure. They’ve been struggling with uptime since before the enormous surge in agentic coding they’re seeing. I don’t doubt that AI has massively increased their load, but u think this is more suspect than that.
Yes, because that caused the usage of the services to skyrocket, GitHub runs on Azure and Azure is experiencing capacity strain due to AI, so GitHub's services are struggling to auto-scale
Per a report that came out the other day, the GitHub move to Azure has been slowed down (i.e. I don't think it's done). But maybe you have newer/better info than me
GitHub’s data architecture (lots and lots of projects in a hierarchical distribution with immutable, highly transaction-based data elements) ought to be easily shardable to allow for massive scale-out. Especially a decade after acquisition by Microsoft. Even a major increase in activity (which was surely predicted internally years ago, given the marketing around AI) should be easily handled by a company with their resources. These repeated failures are indicative of massive mismanagement and misinvestment.
Microsoft has been letting GitHub operate independently since up to very recently. Seems to imply the problems could be systemically GitHub’s and Microsoft is now stepping in and correcting the ship.
They are definitely more outages but the question is if these outages are due to the providers using LLMs to build there products and are therefore not delivering the quality they did before or have LLMs enabled a completely new user base to create projects which they deploy in the free tiers of named providers and they simply cannot keep up with the growth and the new influx of free users is skewing their mixed calculations (free vs paid) so heavily that they cannot scale without losing money. I'd probably say it's a mix of both.
No, it's not just you. It is fairly obvious what's happening - the same old Entshittificators now have a great tool to up the speed of entshitification by 100x - thus these crappy outages every other day.
Well, the significant growth comes from freemium usage. A whole lot of vibe slop triggering actions, with no supporting business. So revenue has not tracked all other growth and the billing system isn’t stressed.
I'd appreciate if they'd not mark the incident as resolved when there's still fallout - ie: my commits didn't display on the branch, my actions didn't run
It's the same issue as the other day - display message at the top admitting that cache needs to be refreshed (or whatever the wording was)
I'm finding this amusing because today, I am in the process of removing all my code from GitHub and cloning it to a local Git that has a webhook to a local releases server.
Multiple companies are trying to create new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.
It's like switching from horse buggies to automobiles, the whole worlds needs re-architecturing to handle the new load.
The age of boutique hand-coding is being replaced by the age of industrial software factories.
> new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.
This is not a particularly novel level of scale. Facebook's mercurial backend had to handle >5,000 developers committing to the singular monorepo long before LLMs were a thing
I mean, if we're talking about "fixing" the symptoms of the downtimes rather than the sources and causes, I guess they could just null route github.com until they have things under control?
Personally, I think they'd have more luck actually attacking the source, what that might be. Somehow I think Microsoft's push for "Every developer only use AI for development, no manual thinking/coding from now on" is the detrimental step, seemingly many companies are still discovering the right approach. Put a freeze to that, and I'm fairly sure you'd see less downtime pretty much immediately, unless all real engineers already left the company, I'm sure I would have at this point.
I personally like Blender’s Gitea theme better than the rest but I guess that’s subjective. In dark mode I do not like the low contrast Codeberg theme or the default Forgejo theme, but all of the instances custom themes look great.
As far as Git forges go in general though.. tangled is very pretty https://tangled.org/tangled.org/core I think more power user oriented software should be comfortable with compact interfaces
That Blender Gitea theme is really nice! I wonder why exactly it's so much easier on the eyes? In a lot of ways all of these are "just Github" with minor changes, so the one that is actually better really stands out.
It's fascinating how fast the Forgejo I host at my university's laboratory loads from my home network. Every page load is <100ms. I think it goes to show how much bloat we don't realise exists in modern webapps.
Ideally those forgejos would safe enough to be on the public internet (and using a federation protocol like activitypub) so we don't have to go through a centralized service such as github and not locked behind private networks (such as tailscale nets)
It's so unfortunate that Gitlab is a complete mess, that GitHub has no real competition now. I can only think that few months to a year from now there will be _something_ that works on an enterprise scale.
Have you forgotten about Azure Dev Ops aka Visual Studio Team System aka Team Foundation Server*?
Yes, it's still Microsoft, but they've forgotten about it, so it runs entirely adequately and is actually a surprisingly okay github replacement. It does nothing special, but it does do everything, just in a way you often would rather it wouldn't. It doesn't have the flexibility of JIRA for the ticketing, and the deployment machinery doesn't have the fanciness ( and vendor threat ) of chaining github actions, but it does handle both.
I haven't used gitlab, so I'm curious to hear what makes it a "complete mess" too.
* Microsoft's headless chicken naming strategy in full force, it's a miracle they haven't yet renamed and rebranded it to align with copilot yet.
How is it painful to use GitLab? Curious, as a user of both, I find them both nice. I like GitLab CI/CD more than I do GHA, but that's personal preference/bias more than anything objective.
Gitlab CI has some tech debt from accumulating geological layers of different ways to do things, but overall it's pretty good, it scales to more complicated setups, and it's not too painful.
Now the best way to use GHA is to do the bare minimum. Put all your CI logic in a script that you can test locally, and just have GHA run your script. Even that is painful. And, somehow, impossible to make secure without having spent 5,000 hours reading all the previous ways people got pwn'd by Github Action's horrendous security model.
My main problem with Gitlab is that after years I still can't find what I'm looking for in the UI. It's always exactly in the third place I look. Otherwise Gitlab has been good. Even self-hosted works pretty well.
if you just want somewhere to stick a code repo and build a release every so often — dont use gitlab, you will not enjoy it.
> My main problem with Gitlab is that after years I still can't find what I'm looking for in the UI.
i still get lost too after several years daily driving gitlab. this is the Ops centric thing. they provide a lot of options. lots of options is good for Ops.
> Now the best way to use GHA is to do the bare minimum
yeah, i’m an ops guy, so the maintaining custom actions stuff on github is horrible for me vs click a button and move on with my day — once i find the button that is! xD
As a daily GitLab user, I'd say that would be the main criticism I could levy at it as well. It does feel like there are a number of "and the kitchen sink" type features that are just there to check a box in a RFP or something.
That said, are the majority of people actually even _using_ those features? For us we're essentially just using GitLab for git, merge requests, and CI pipelines. A couple places we use the static page hosting. (First thing I do whenever I create a new repository is go into the settings and just uncheck _all_ the boxes.)
All of that core functionality works really well and is more than polished enough from my point of view.
Gitlab is pretty decent. Honestly I would say there's not much between GitHub and Gitlab. Gitlab's CI is more powerful than GitHub's IMO, but the UX is a bit worse. But it's really marginal.
They're both slow and have tons of long-standing missing features. But they're ok. I'd definitely rather use Gitea/Gogs/Forgejo, and maybe Tangled if it supported normal setups (e.g. private repos).
It was just yesterday [0] that GA was down and another incident today? I am convinced that Copilot and Tay.ai are destroying GitHub and there is no CEO of GitHub to contact.
Now will you please self-host as I said 6 years ago? [1]
At this point, you might as well say that is what's happening at GitHub with the help of GitHub Copilot since nothing has changed and has only gotten worse over time.
Yesterday my CI runs wouldn't even be created because Actions was eating shit, and today my CI runs get created but fail because the API is eating shit. Fun.
They may have gotten down to only 2 nines on most of their services, but at least the LLM is still running at full power! must increase value for shareholders
For years we had a GitHub status thing in our Slack but I had to remove it about a year ago because the noise got too much, it would be unbearable in 2026.
If you go to www.githubstatus.com, the downtime is not showing in the chart. I was annoyed enough yesterday when I visited this page to figure out why my Actions had failed and was greeted with big green ticks and only a tiny red rectangle halfway down the page to indicate the problem.
This time they've just scrubbed the evidence outright?
among all the recent incidents, this was probably the first time i experienced any service disruption.
even comparing branches were giving 500 in the web ui. even github copilot in their new session limits now also mention if github is going through an incident, which is plainly comical.
as a github user, we are paying for the slow git operations through our github action minutes, if someone from GH is here, will you be compensating for it?
I think it is time to decouple GitHub from Microsoft. Microsoft has shown irresponsible behaviour - and this continues. They keep on going at it until nothing works anymore. Typical microslop work.
You don't get to control that. It is Microsoft's right to do whatever it wants with GitHub - it could shut down tomorrow, or demand face ID. If you want to control what happens with a thing, you have to make the thing instead of letting someone else make the thing and sell it to Microsoft.
Your choice is to accept the product that exists on the market, switch to another product that exists on the market (such as Codeberg or self-hosted Forgejo), make your own product, or not use any.
You missed the option to "lobby the government to tell Microsoft to straighten up and fly right". Microsoft is a corporation, a legal entity only created with the permission and grace of the State of Washington.
I know it’s super fun to shit on GitHub and everyone’s favorite thing to say is “build a competitor!”
They’re trying to scale from 1 billion commits last year to over 14 billion this year. I have zero desire to try and manage that scaling. Basically being DDOS’d by agents all day now.
> I have zero desire to try and manage that scaling.
Last time I checked, this is called "doing their job".
You are free to choose what company you want to work for, but for github, if they are offering the services, often as paid services, they have an obligation to keep things running which is called SLA. Otherwise people will leave (and are leaving)
We've banned this account for repeatedly posting abusive comments. We've asked you before to observe the guidelines, and the recent trend is getting much worse. When this happens we eventually have to assume you want to be banned. If you don't want to be banned, you can email us (hn@ycombinator.com) and commit to using HN as intended.
https://isgithubcooked.com
Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.
It's not physically possible to run post-mortems for issues at those rates.
They should install OpenClaw for that as well.
AI: The cause of, and solution to, all of your tech debt.
Perhaps best to simply declare indefinite-mortem
They should be running pre-mortems every morning at this point
The operational equivalent of pre-crime.
> It's not physically possible to run post-mortems for issues at those rates.
Not at all, you merely move the goal post of at what layer the "root cause" actually could come from! At that speed, it's always something short and sweet, while when you actually want to long-term address things, you have to have time to even investigate organizational issues or whatever the actual problems stem from.
But you have half a day? "Post-mortem: Push X wasn't properly analyzed before deployment, in future more testing" and call it a day.
“A failure occurred. This was caused by something going wrong. Changes to operating guidelines have been instituted to ensure that things will not go wrong in the future unless we happen to do the same thing again.”
Of all the sites/graphs I've seen of GH outages, this one is the most striking IMO:
https://damrnelson.github.io/github-historical-uptime/
Unfortunately, it doesn't look like it's being updated with new data. But it wouldn't look any better for GH if it was.
FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.
Sure, but it isn't completely wrong either.
GH going down used to be quite rare. If it failed to load I'd spend a bunch of time trying to figure out what was wrong with my internet connection, just to read on HN that it was down for everyone.
This week GH failed to load and I automatically assumed it was a GH issue - just for it to be followed up a few minutes later by a marketing coworker complaining about internet connectivity. Turns out the office internet connection was dropping about 50% of all packets.
It is bad enough that business-side managers are noticing that GH issues are slowing work down. That would've been unimaginable a few years ago.
Apparently, you aren't alone. :)
https://github.com/DaMrNelson/github-historical-uptime/issue...
I wonder what the cause of this was? Microsoft Politics? Bureaucracy? Forced move to azure?
My guess would be a obnoxious and lethal mixture of all of the above.
Also AI mandates.
Wow, it seems that 100% of sev-3 ("critical") incidents in the last year (=365 days) have occurred between April 22, 2026 and now.
Is it possible that there has been a change in the way the data are collected/recorded that even partially accounts for this sudden onset?
One tangent, I believe sev-0 is actually "critical" (at least as how I'm used to reading it), and the higher you go the less critical something is.
IMO as a github-watcher, I think they changed their definition of what constitutes a sev-0 between sev-1 for the better. In particular, they had a few "sev-1"'s around the turn of the year that would be classified as sev-0's if they happened today.
Pre-4/22 GitHub sev-1 was a normal SaaS company's sev-0, imo. So I think their new system is more reflective of reality. My guess is that a few of their big customers bullied them to have more accurate SEV categorization.
Ah, thank you for the correction on sev-0.
To be clear, your observation that "they changed their definition of what constitutes a sev-0" is based just on your external observation of incidents and their designations, correct? I.e. they haven't officially released a statement saying they have changed their standards
Waves around it had to break eventually eh?
The UI of that page is so nice, should build a github competitor.
The user profile / contributions and PR UX is pretty much the entire "hub" product since git is a fully separate offline app.
> The UI of that page is so nice
Is it? Seems a text description of "Make a website outlining 'How cooked GitHub' is with a modern style" to basically any LLM would produce exactly that UI and design, literally nothing of that design a human had any influence on, besides the ones selecting what training data the used LLMs was trained with.
I think most of us who've tried using LLMs for web-design can recognize that style and design at this point, regardless of model actually used.
Oh wow, I'm in the position to be able to give a peek behind the curtain of something (validly!!) critiqued as AI slop! Exciting.
I originally made the core data functionality of this site for myself because I was curious what the uptime stats for each service were (I build something that heavily depends on GitHub), and to viz the distribution/severity of those incidents, again per-service, over time.
It involved a lot of back-and-forth, and is not a one-shotter; maybe closer to 40-50 shots over maybe ~10 hours of human time. A couple memorable things that made it complicated, irrespective of the UI: sneaky bugs around double-counting time for overlapping incidents, no GitHub API for incidents so you need to puppeteer-scrape the backlog of incidents to get historical data. Although, you all are right to call out that the CSS was three shots, though, and it shows :) I thought it looked so cool in ~January 2026 and now it gives me the ick, too!
For people who are curious about how much direction went into the information architecture/presentation, it was fairly substantial. I wanted a contribution graph style viz and it took many turns to get it working the way I wanted. The swimlane viz for selected-day-incident visualization was also me, because I love swimlane graphs.
I ended up sharing it with some folks and they wanted to reference it, so I put it on a website. So it's jokey for sure, but I take my jokes seriously! I'm grateful that people have feedback on how it can better functionally and visually :)
> Although, you all are right to call out that the CSS was three shots, though, and it shows :)
Totally, my comment was all about the styling and design literally, and is in no way a comment about the data or actual contents of the website, hope you didn't take it that way as well, as it does seem proper in that regard!
Thank you for sharing it, and even greater thank you for sharing the process behind building it, for me that's more interesting almost :)
The Bootstrap of 2020s.
At least Boostrap pages were readable ;)
Compared to near unusable pages that large organizations produce, yes this page is highly effective at conveying information. Who cares how it was produced?
> Who cares how it was produced?
Well, we're at least two people who care, since we were conversing about how good/bad the webdesign is, then you jumped in here :) If you don't care, why bother to reply to people who seemingly do care? What kind of conversation are you expecting here, "Yeah, do tooo"? :|
Could you explain why you care?
Why I care about understanding people who think differently than me? I don't know, always did, not sure exactly why, always been interesting to understand people's perspectives, especially when I personally feel differently, tends to help myself understand me better too, so it's basically a win-win to get people to explain their reasoning.
A lack of effort put into presentation is a signal for low quality. In the past, this might've manifested in using default Wordpress templates, and now it manifests as stock LLM templates instead. Can you make high quality content and present it on a stock template? Sure, you can, but without any prior reason to believe that you put more effort into the content than the template, I'm liable to believe the content is LLM-generated slop as well and therefore untrustworthy and not worth my time.
In this case, it appears more effort was put into content than presentation, which is a possibility and the creator is in the comments saying as such, but humans operate on heuristics by default. The majority of sites that look like this have been a complete waste of my time and I usually just click away at this point.
A lot of software engineers do still care how software is produced. That's a good thing!
People who make web sites care? Isn't this a place to talk about how tech things are made!?
> this page is highly effective at conveying information
Is it though? If the page is near unreadable?
* Almost pure-black background rendering every not-pure-white colour barely readable
* Dark-grey and low saturation colours used almost everywhere, for both fonts and other coloured elements (the orange cells in the calendar are the most readable thing)
* Thin fonts - coupled with the dark grey colours this just adds to the readability issues
* Yet another incredibly long info-dump of a page
And then as far as actual information:
* Vanity metrics as the main information, that is a lot of things with no context or historical information
* A lot of aggregates and rollups that aren't that useful
No, I haven't tried Reader Mode.
It's a good demo for UI state syncing though, I'll give it that.
What really grinds my gears is how easy it is to get better designs out of LLMs. But if you don't ask, you get the default.
Outside design systems I rarely get good CSS from LLMs.
3D type stuff too, it's useless outside boilerplate.
Very little spatial reasoning training, no end-user subjective reasoning inference (Google is starting to though even in unrelated chats), so it's no surprise the LLM doesn't know what you want.
Since I don't even know what I want half the time until I saw it, the subjective reasoning piece is key - that is, being able to predict what I'll want to pretty good accuracy. Then you have your agents etc.
as someone who doesn't know how to get better design out of LLMs, can you elaborate?
Have an opinion on the design, imagine something, then tell it to do just that, then iterate. It's when you're unspecific you get the generic, bland and typical LLM design, you just have to be subjective and influence it in some (human) direction.
Also check out https://impeccable.style/, it's really good
what would you ask to get a better design?
I say listen up Gemini you mother FUCKER
Here is a provocative thought - maybe these are the so-called "better designs" from LLMs? It's not like writing English sentences is some huge secret you are sitting on that no one else knows.
> It's not like writing English sentences is some huge secret you are sitting on that no one else knows.
I'd actually say what really makes an excellent engineer stick out among many great engineers, is their ability to communicate clearly and knowing what needs to be communicated vs not, basically being way better at language and communication in general, and they also understand the important of it.
I agree. But I was talking about the "super secret" ability to write prompts, which pretty much anyone can do.
My point being that not everyone writes as good prompts as everyone else, the way you communicate, how clearly and how exact you are matters a lot, much more than you seemingly is under the impression of.
Same goes with the "LLM does web design" example from before, a web designer with great communication skills in web design, will (naturally) have a better prompt for something that'll potentially could look good, compared to a web designer that isn't at good at communicating what they actually want.
I’m actively working on an alternative Frontend for Forgejo at the moment, completely self hostable, free, and open source.
Moving everything from GitHub to Forgejo and Tangled for now. These outages haven’t effected me for the past month because of this.
I'm deleting my GitHub repos today (been planned for a bit) in favour of a local Forgejo Git. I also have not experienced any service disruption since I migrated well over a month ago.
Can you elaborate on how your Forgejo frontend will be different than the default one? I'm asking because I've only ever used GitHub, GitLab and Forgejo for longer periods and Forgejo was the fastest and easiest to use for me.
It’s still early days, but I already have it in a useable state so I could share more such as early screencaps.
I plan on focusing primarily on these areas:
- mobile experience is first class, even on old/slow devices
- diff viewer is fast even on extremely large pull requests
- stacked pull request support
- user interface is modern, accessible, and theme-able with a light touch of whimsy
- search is accessible from anywhere
- opinionated keyboard shortcuts and commandk palette from day one
Many other longer term goals that I’m not mentioning here for now while the roadmap is forming.
>"The UI of that page is so nice"
Most part screen is taken by picture. Contrast ratio is really low. Hard to read Should they remove that useless banner, current status which is the most interesting part coud've been made visible right away.
I would call this whole thing highly un-ergonomic
The UI is in the default claude code style
Lol it's pretty bad UI
May has been filled with critical issues. It seems it's getting worse over time.
Commits are up 14x year-over-year
https://x.com/kdaigle/status/2040164759836778878
Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.
I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch.
I don't envy their position of having to scale that fast on something that has to be instant and real-time. As far as I know, you can't do CDN/edge caching shenanigans with a remote git repository like Google can with a YouTube video. It's gotta always be reading/writing to the latest, single source of truth.
Sure, backseat commenting is easier and I wouldn't wanna be in charge at github right now, but on the other side there also a reality where we'd see video essays about githubs downfall because their reliability crashed so hard that businesses could not trust them and moved to competitors / self hosted instances which then meant less paid users to subsidize the ever growing demand of the free users.
It’s not quite as cacheable as YouTube, but a lot of it is still pretty cacheable. Actions aren’t. Issues, wikis, and READMEs are. File views are. Most projects aren’t changing daily. The few that change daily aren’t changing hourly. There are a few that change constantly and would require constant cache updates. But the long tail is pretty static.
Yes it's potentially a write-heavy workload which also needs to be consistent aka the worst case scenario.
The easy solutions like caching and read replicas don't work and you're forced to go the route of sharding or similar techniques that have much more painful tradeoffs.
I'm not sure if that's why everything keeps breaking but at that scale write-heavy workloads are never going to be easy
Not a valid excuse without knowing what their historical growth rate has been. And how much of the instability is load related.
GitHub has been publishing their growth numbers since at least 2016: https://octoverse.github.com/2016/
However, they have reported numbers along rather inconsistent dimensions. Like, historically they've focused on number of repos and users and later PR's and issues, and often catch-all terms like "contributions" which includes all of those + comments etc... but the number of commits alone (which apparently is the main culprit now?) has been mentioned very sporadically. This has made it hard to get a consistent sense of historical growth.
Without any other information, however, it is reasonable to assume that a 14x in commits is the prime candidate for instability. Especially since commits are write traffic, which is much harder to scale than read traffic. Plus every 3 - 5x increase in scale can reveal bottlenecks in your distributed systems that you never knew existed, so they probably have like 2 - 3 "generations" of bottlenecks to figure out!
They are highly responsible for all of that. They are diversified a lot with a lot of random things instead of focusing on their core business. They have actively pushed people to use the service and feature more.
Think about countless actions that have to run almost at every push and PR push! Also, remember that we were used to use external services for "actions", and they basically killed the competition by offering their own CI actions at no cost to most users.
Also, they did a lot of reworks in the last years, not necessarily for the best like the PR diff page, and probably not in the most efficient way.
They are already cooked as this has been happening ever since the Microsoft acquisition and it was run to the ground before 2023.
At this point you would get better uptime by just self-hosting your own GitLab, Forgejo or Codeberg instance instead of dealing with Github's unreliablity.
There is no defending them with their clear neglet and carelessness of the platform.
If all you need is a repository, you don't even need any of these. You need SSH access to a server, and optionally, one of several web front-ends. Git comes with a CGI script that handles public anonymous checkouts via HTTP(S), although since nginx doesn't support CGI, integrating those is a little bit tricky as you need a FastCGI wrapper.
I moved most of my projects off GitHub to Forgejo and will be using Tangled too for public repositories. I don’t think people realize that if you self host Forgejo, you get 99% of the functionality of GitHub with zero of the limitations. Especially if you have the hardware to spare for CI runners. And if self hosting isn’t your thing you can always just use Codeberg and Tangled directly.
I’m working on an open source Forgejo browser called Joui. It’s coming along nicely, and is so much snappier than GitHub in every single way.
Is the “streak” days of continuous uptime, or of days with at least one downtime incident? I think it’s the latter :]
It looks like it is the number of consecutive days with no incident. If you look at 31 Dec 2025, that corresponds to an 8-day period with no incidents.
I guess that also means this year GitHub has not yet made it a single week without an outage of some kind.
It's a streak for continuous uptime, and yeah it is fairly depressing to imagine overseeing that :/
Name one thing Microsoft didn't run into the ground post-acquisition
hey now, LinkedIn was terrible before Microsoft.
Java or Bedrock edition, and have you tried logging into your EntraID Microsoft Teams for Xbox account lately? Make sure to check the box to keep you logged in!
Last I heard UK Minecraft players aren't even allowed to talk anymore without ID verification.
And if someone makes a server that doesn't do the chat verification, Microsoft blacklists that server in the client-side server address textbox. This system was developed to destroy pay-to-win servers, but they're now applying it against servers that refuse to censor "fuck".
Not as bad as it is now. All I see are suggested posts from people I never connected with and those are full of instagramesque self-promoting banal vibes.
TBH, even LinkedIn seemed to provide me with posts advertising events that happened two weeks ago a bit less pre-acquisition.
GH was acquired by microsoft some eight years ago. It has been working quite well until recently.
People may have had complaints about functionality, features, commercial issues, but the thing used to at least have a decent uptime until recently.
Has nothing to do with Microsoft acquisition... AI usage has increased demand and load. More PRs, more Action runners, more of everything firing. GitHub just wasn't ready for the scale and are now having issues catching up with it as it continues to increase exponentially.
This is a convenient lie that GH likes to tell. Growth is nothing like exponential, its at most 300% over several years according to their own public numbers (presented misleadingly on graphs)
But a couple of years ago they were crowing about how much work they were doing to prepare for “a billion developers”. If they had actually done that then the actual load from agents should have been no problem.
Is this growth in resource usage or growth in revenue? Because those numbers aren't necessarily coupled. I.e most action runners are free
usage
There was an x post in another thread under this post that showed all the standard usage numbers are way up: 14x, 2.1x, etc. And the OP hinted at the usage growth being non-linear for 2026
Are you sure? Seems like they "completed" a migration about the same time all these problems started to become daily. https://www.theverge.com/tech/796119/microsoft-github-azure-...
Yeah, that and Microsoft has been slow to move the infrastructure to something that scales better to handle that load.
The more surpassing part is that Microsoft hasn't figured out a way to manage/contain the AI-sourced traffic better so it doesn't create all this noisy neighbor problems for non-AI usage/users.
Github's core platform doesn't really make that separation, anything a human can leverage on github an AI agent can as well, just faster and with heavier usage. End of day agents and humans are using the same services.
Sure, still need to enable access the same info but feels like bucketing the clients into
bucket1 = clients that were working just fine before (users and whatever automation they had in place) bucket2 = ai clients that contributed to, if not flat out caused, the scale problems
then slowing down/limiting the bucket2 clients while keeping the bucket1 clients rolling as-is, is both doable and keeps existing customers happy while the underlying infra gets scale/perf improvements needed to support ai clients at scale.
MSFT is also forcing its subsidiaries to “lean into AI” so that they can fire people to cover for Satya’s bad investments
> It has been working quite well until recently.
I'm not sure how reliable the data is, but average uptime seems to have dipped measurably starting within a year of the aquisition, according to https://damrnelson.github.io/github-historical-uptime/
FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.
They moved to Azure. Nothing improves on Azure.
It also used to be run as an independent company with access to MS's resources.
Now it's a unit in their AI hype machine.
MSFT was pretty arms length for the first 5-6 years. I was honestly kind of impressed and it made my opinion of MSFT better. But then AI made it too attractive of a target and MSFT couldn't help but make it a place the former CEO wanted to leave (and it has been running headless for about a year now).
It's quite disappointing objectively, but I expected worse from MSFT.
I think Minecraft is still in good shape
I wouldn't know, somehow this game I bought maybe 15 years ago is no longer playable for me, my account was supposed to be migrated from Mojang to Microsoft or similar, but then that never happened or something, and trying to login now asks me to contact Microsoft support, which I've tried 3-4 times, never had anyone respond to me so who knows how the game is today? I stopped trying at this point...
Personally, once a game I own is janked from my hands because of organizational decisions, that's the time I'll stop consider the game "in good shape", but I'm sure the people who had to buy the same game a second time still enjoy it.
Yes, the account migration was a mess. Support response times were at least 30 days, if you ever actually received a response at all (I never did). I did buy the game a second time in order to play with my kids.
They deleted my account from 2010 because I didn't convert it to a Microsoft one. They baked an incredibly aggressive chat filter into multiplayer, even if you're not playing on official servers. They've added microtransactions for things that we previously free (skins, resource packs). They force you into their shitty, bloated, user-hostile launcher with adverts.
It's been nonstop content-slop since the acquisition. New mobs, new blocks, new items, new blocks, new items, new mobs, new mobs, new biomes. Some of them are good but the totality of adding a bunch of stuff has been to destroy the simplicity that was one of the draws of the original game. Now it's an exploration and niche-mechanics-exploitation game more than a virtual legos game. You don't go mining any more, you find trading loops with villagers.
This was happening to some degree pre-acquisition, but since the acquisition it's been this non-stop.
Some of it's good. The Nether and the oceans were really boring before their respective updates.
They should have called Minecraft "done" around the acquisition time and started on Minecraft 2.
[dead]
Dave Cutler?
Like those aviators who draw a picture on flightradar24, if you filter by All Services - Critical, somebody almost about to draw a swastika just in May... Are the AI agents revolting?
This is getting ridiculous. One particularly concerning thing I’m seeing is that pull requests on both the web UI and API aren’t reflecting all commits or branch changes consistently. It would be very easy to merge something without realizing you’re not actually reviewing the full diff.
Yeah, I've had several occasions recently (seemingly not related to any incidents on the status page) where I've had to wait 20 minutes to an hour to be able to open a PR, because Github didn't recognize my branch had any new commits compared to the base branch.
A taste of the agenticaly-developed world.
[flagged]
Before clicking, I assumed this was going to be a write-up of the one from a few days ago instead of an entirely new incident.
I assumed it was the one from yesterday! Silly me.
New PR: revert GitHub software and infrastructure to version of June 1st, 2018.
New PR: disable new user signups for 6 months
HR initiative: all future KPIs automatically require three-nines availability; all bonuses are forfeited, regardless of accomplishments, if annual availability falls below target
HR initiative: fire CEO and CTO
New PR: disable Github API New PR: block (ai) bots through attestation to make usage predictable
Finance initiative: Undo the Microsoft purchase
Github does not have a CEO
We're halfway there!
Imagine if we did this for countries.
is it me or ever since AI coding became the norm, there have been way more outages with otherwise reliable services?
I get downtime on Supabase every few weeks. Even Cloudflare. And now Github
GitHub’s instability started when they announced they were wholesale migrating to azure. They’ve been struggling with uptime since before the enormous surge in agentic coding they’re seeing. I don’t doubt that AI has massively increased their load, but u think this is more suspect than that.
I think they’re referring to services using AI for coding and shipping bugs more often as a result.
Yes, because that caused the usage of the services to skyrocket, GitHub runs on Azure and Azure is experiencing capacity strain due to AI, so GitHub's services are struggling to auto-scale
Per a report that came out the other day, the GitHub move to Azure has been slowed down (i.e. I don't think it's done). But maybe you have newer/better info than me
GitHub’s data architecture (lots and lots of projects in a hierarchical distribution with immutable, highly transaction-based data elements) ought to be easily shardable to allow for massive scale-out. Especially a decade after acquisition by Microsoft. Even a major increase in activity (which was surely predicted internally years ago, given the marketing around AI) should be easily handled by a company with their resources. These repeated failures are indicative of massive mismanagement and misinvestment.
Microsoft has been letting GitHub operate independently since up to very recently. Seems to imply the problems could be systemically GitHub’s and Microsoft is now stepping in and correcting the ship.
They are definitely more outages but the question is if these outages are due to the providers using LLMs to build there products and are therefore not delivering the quality they did before or have LLMs enabled a completely new user base to create projects which they deploy in the free tiers of named providers and they simply cannot keep up with the growth and the new influx of free users is skewing their mixed calculations (free vs paid) so heavily that they cannot scale without losing money. I'd probably say it's a mix of both.
Enterprise software and support services have consideraby (in my personal experience) degraded over the last year.
Less care on process, or quality, more focus on "just ship".
I've also seen and heard from peers this happening in multiple smaller outsorcing companies.
Correct. There’s no incentive to be careful anymore when you can just prompt an LLM to fix it
No but everyone is pretending that everything is fine. Actually, no, no one is pretending anything. No one cares, really.
Not just you, but uncertain whether it's due to unreviewed slop going to production, or increased demand due to slop generation.
> is it me or
No, of course not.
[dead]
No, it's not just you. It is fairly obvious what's happening - the same old Entshittificators now have a great tool to up the speed of entshitification by 100x - thus these crappy outages every other day.
Good that the Billing functionality is still at 3 nines at least.
They got their priorities right for sure
Well, the significant growth comes from freemium usage. A whole lot of vibe slop triggering actions, with no supporting business. So revenue has not tracked all other growth and the billing system isn’t stressed.
Someone linked this third-party "honest" status page:
https://mrshu.github.io/github-statuses/
Seems more accurate with my experience of GitHub.
I'd appreciate if they'd not mark the incident as resolved when there's still fallout - ie: my commits didn't display on the branch, my actions didn't run
It's the same issue as the other day - display message at the top admitting that cache needs to be refreshed (or whatever the wording was)
I'm finding this amusing because today, I am in the process of removing all my code from GitHub and cloning it to a local Git that has a webhook to a local releases server.
GitHub is not agent scale.
Multiple companies are trying to create new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.
It's like switching from horse buggies to automobiles, the whole worlds needs re-architecturing to handle the new load.
The age of boutique hand-coding is being replaced by the age of industrial software factories.
> new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.
This is not a particularly novel level of scale. Facebook's mercurial backend had to handle >5,000 developers committing to the singular monorepo long before LLMs were a thing
Yes, on a single repo. Now multiply that per bazillion companies on github, some of which are trying that.
Why the heck would you want to do this. Using git as your undo chain sounds like a pretty awful thing to do.
I think it'd be pretty neat to be able to rebase my undo history on to a remote branch someone else is working on.
This is smart but also hilarious.
This seems odd to me. Why would you need to commit every second?
And push to remote as well? Seems not thought out
> GitHub is not agent scale.
Is the scaling issue with git or github?
[dead]
Maybe GitHub needs to freeze free repository creation until they get this under control because this is ridiculous.
I mean, if we're talking about "fixing" the symptoms of the downtimes rather than the sources and causes, I guess they could just null route github.com until they have things under control?
Personally, I think they'd have more luck actually attacking the source, what that might be. Somehow I think Microsoft's push for "Every developer only use AI for development, no manual thinking/coding from now on" is the detrimental step, seemingly many companies are still discovering the right approach. Put a freeze to that, and I'm fairly sure you'd see less downtime pretty much immediately, unless all real engineers already left the company, I'm sure I would have at this point.
It’s not just their own slop that’s causing this, it’s also caused by the tsunami of slop being uploaded by vibe coders.
If you want to upload to GitHub, you should pay. The days of charitably giving away compute for the “open source communities” are over.
Grandfather existing public repositories in, then cut it off. Stop the bleeding. It doesn’t have to be forever.
Or maybe they need to bring back quality assurance expertise to the company.
Yes that would be part of getting things under control, of course.
And/or move more contextually aware humans with 10K+ hours of hard won experience and fear of failure/sense of pride back into the loop.
https://GiveUpGithub.org
Good link - why and how to ditch GitHub.
Are they running paid marketing campaigns for Gitlab ?
Self-hosting forgejo under tailscale + mirroring public repos through GitHub
Has worked wonders for me :)
Forgejo is fantastic. I do think it could use a fresh coat of paint from a designer but it’s otherwise really good.
Gitea (what Forgejo forked from) recently stole the sidebar on repos from GitHub and I think that would be great for Forgejo to steal too…
Forgejo themed by Codeberg: https://codeberg.org/forgejo/forgejo (the codeberg theme is extremely low contrast)
Forgejo default: https://v15.next.forgejo.org/pparaxan/quark
Forgejo themed by Lix: https://git.lix.systems/lix-project/lix
Gitea: https://gitea.com/gitea/awesome-gitea
Gitea themed by Blender: https://projects.blender.org/blender/blender
I personally like Blender’s Gitea theme better than the rest but I guess that’s subjective. In dark mode I do not like the low contrast Codeberg theme or the default Forgejo theme, but all of the instances custom themes look great.
As far as Git forges go in general though.. tangled is very pretty https://tangled.org/tangled.org/core I think more power user oriented software should be comfortable with compact interfaces
That Blender Gitea theme is really nice! I wonder why exactly it's so much easier on the eyes? In a lot of ways all of these are "just Github" with minor changes, so the one that is actually better really stands out.
I like it too, is it available for download anywhere?
I think I found it : https://projects.blender.org/infrastructure/gitea-custom/src...
It's fascinating how fast the Forgejo I host at my university's laboratory loads from my home network. Every page load is <100ms. I think it goes to show how much bloat we don't realise exists in modern webapps.
GitHub used to be like that before they rebuilt everything in React.
No, they didn’t rebuilt everything in React and current sluggishness of page loads is not related to React.
See: https://news.ycombinator.com/item?id=47940456
Ideally those forgejos would safe enough to be on the public internet (and using a federation protocol like activitypub) so we don't have to go through a centralized service such as github and not locked behind private networks (such as tailscale nets)
It's so unfortunate that Gitlab is a complete mess, that GitHub has no real competition now. I can only think that few months to a year from now there will be _something_ that works on an enterprise scale.
Have you forgotten about Azure Dev Ops aka Visual Studio Team System aka Team Foundation Server*?
Yes, it's still Microsoft, but they've forgotten about it, so it runs entirely adequately and is actually a surprisingly okay github replacement. It does nothing special, but it does do everything, just in a way you often would rather it wouldn't. It doesn't have the flexibility of JIRA for the ticketing, and the deployment machinery doesn't have the fanciness ( and vendor threat ) of chaining github actions, but it does handle both.
I haven't used gitlab, so I'm curious to hear what makes it a "complete mess" too.
* Microsoft's headless chicken naming strategy in full force, it's a miracle they haven't yet renamed and rebranded it to align with copilot yet.
One does not mention TFS in polite company
Only alternative outside of GitHub and GitLab I've used was Bitbucket, and it was worse - but this was time when GitHub was good.
There are plenty of open source forges with a better UX.
Don't forget CodePlex!
I'd consider self-hosting GitHub Enterprise before putting my team through the pain of Gitlab.
How is it painful to use GitLab? Curious, as a user of both, I find them both nice. I like GitLab CI/CD more than I do GHA, but that's personal preference/bias more than anything objective.
Gitlab CI has some tech debt from accumulating geological layers of different ways to do things, but overall it's pretty good, it scales to more complicated setups, and it's not too painful.
Now the best way to use GHA is to do the bare minimum. Put all your CI logic in a script that you can test locally, and just have GHA run your script. Even that is painful. And, somehow, impossible to make secure without having spent 5,000 hours reading all the previous ways people got pwn'd by Github Action's horrendous security model.
My main problem with Gitlab is that after years I still can't find what I'm looking for in the UI. It's always exactly in the third place I look. Otherwise Gitlab has been good. Even self-hosted works pretty well.
expanding on the parent a little
* GiLab — Ops centric
* GitHub — Developer centric
if you just want somewhere to stick a code repo and build a release every so often — dont use gitlab, you will not enjoy it.
> My main problem with Gitlab is that after years I still can't find what I'm looking for in the UI.
i still get lost too after several years daily driving gitlab. this is the Ops centric thing. they provide a lot of options. lots of options is good for Ops.
> Now the best way to use GHA is to do the bare minimum
yeah, i’m an ops guy, so the maintaining custom actions stuff on github is horrible for me vs click a button and move on with my day — once i find the button that is! xD
>GitHub — Developer centric
I'm looking at the branch listing and I can't see a single hash. I dunno who GitHub is for but it isn't devs.
Everything about their UI/UX screams of doing the bare minimum to check off a box on a feature list. It reminds me of Jira.
As a daily GitLab user, I'd say that would be the main criticism I could levy at it as well. It does feel like there are a number of "and the kitchen sink" type features that are just there to check a box in a RFP or something.
That said, are the majority of people actually even _using_ those features? For us we're essentially just using GitLab for git, merge requests, and CI pipelines. A couple places we use the static page hosting. (First thing I do whenever I create a new repository is go into the settings and just uncheck _all_ the boxes.)
All of that core functionality works really well and is more than polished enough from my point of view.
Gitlab is pretty decent. Honestly I would say there's not much between GitHub and Gitlab. Gitlab's CI is more powerful than GitHub's IMO, but the UX is a bit worse. But it's really marginal.
They're both slow and have tons of long-standing missing features. But they're ok. I'd definitely rather use Gitea/Gogs/Forgejo, and maybe Tangled if it supported normal setups (e.g. private repos).
I think it’s about time to backup all of my GH repos.
Again?
It was just yesterday [0] that GA was down and another incident today? I am convinced that Copilot and Tay.ai are destroying GitHub and there is no CEO of GitHub to contact.
Now will you please self-host as I said 6 years ago? [1]
[0] https://news.ycombinator.com/item?id=48278635
[1] https://news.ycombinator.com/item?id=22867803
GitHub famously does not have a single 9 of uptime.
You think a Microsoft chatbot from 2016 is destroying GitHub?
Microsoft corporate culture destroys Github reputation and tech.
At this point, you might as well say that is what's happening at GitHub with the help of GitHub Copilot since nothing has changed and has only gotten worse over time.
Yesterday my CI runs wouldn't even be created because Actions was eating shit, and today my CI runs get created but fail because the API is eating shit. Fun.
They may have gotten down to only 2 nines on most of their services, but at least the LLM is still running at full power! must increase value for shareholders
For years we had a GitHub status thing in our Slack but I had to remove it about a year ago because the noise got too much, it would be unbearable in 2026.
If you go to www.githubstatus.com, the downtime is not showing in the chart. I was annoyed enough yesterday when I visited this page to figure out why my Actions had failed and was greeted with big green ticks and only a tiny red rectangle halfway down the page to indicate the problem.
This time they've just scrubbed the evidence outright?
It was previously showing, but I believe the incident has bee resolved now. At least, PRs work for me when they previously didn't.
The "Git Operations" chart is showing all green, but several of the recent blocks have a note showing there was an outage.
Today's is green, even though there was an outage.
Maybe we should start posting av story when GitHub has been fine for some time instead of posting every incident
git is supposed to be decentralized.
maybe it's time to revert back to the central idea of git & not centralize around a particular provider.
for issues - mailing list will do. you can always slap a beautiful ui if you want to or a tui (as is the fad) these days.
actions can also be decentralized via an API spec & webhooks.
Reminder that after more than a month, many repos _still_ don't show all PRs and Issues https://github.com/orgs/community/discussions/193463
among all the recent incidents, this was probably the first time i experienced any service disruption.
even comparing branches were giving 500 in the web ui. even github copilot in their new session limits now also mention if github is going through an incident, which is plainly comical.
fun times!
Github Pull Requests are down to 1 nine of availability
https://mrshu.github.io/github-statuses/
The software reliability and uptime is going bad across the industry, railway, github etc
wild that there is a large pattern forming up of unreliable software being pushed
Tried to do a git push - it succeeded after 3 mins. Then I wanted to open a PR and it failed with a 500 error.
Facepalmed and decided that this is it for today.
I'm so done with GitHub.
GitHub Incident again/
At this point we can even stop specifying that it's GitHub...
as a github user, we are paying for the slow git operations through our github action minutes, if someone from GH is here, will you be compensating for it?
I think it is time to decouple GitHub from Microsoft. Microsoft has shown irresponsible behaviour - and this continues. They keep on going at it until nothing works anymore. Typical microslop work.
You don't get to control that. It is Microsoft's right to do whatever it wants with GitHub - it could shut down tomorrow, or demand face ID. If you want to control what happens with a thing, you have to make the thing instead of letting someone else make the thing and sell it to Microsoft.
Your choice is to accept the product that exists on the market, switch to another product that exists on the market (such as Codeberg or self-hosted Forgejo), make your own product, or not use any.
You missed the option to "lobby the government to tell Microsoft to straighten up and fly right". Microsoft is a corporation, a legal entity only created with the permission and grace of the State of Washington.
Sure, I guess skywriting prayers to the spaghetti monster is also an option, but neither of those will do anything.
But everything else still works fine, right?
I know it’s super fun to shit on GitHub and everyone’s favorite thing to say is “build a competitor!”
They’re trying to scale from 1 billion commits last year to over 14 billion this year. I have zero desire to try and manage that scaling. Basically being DDOS’d by agents all day now.
https://x.com/kdaigle/status/2040164759836778878
> I have zero desire to try and manage that scaling.
Last time I checked, this is called "doing their job".
You are free to choose what company you want to work for, but for github, if they are offering the services, often as paid services, they have an obligation to keep things running which is called SLA. Otherwise people will leave (and are leaving)
If it were humans using the site it might be super motivating, but knowing that it's 99% bots producing AI slop, yeah I'd be looking for a new job.
I thought it was the yesterday's thread but no, here we go again
dthtrj
Fed up and bored of this
It seems before AI eats software, its going to first eat GH and Microsoft.
they didn’t think the leopard would eat their face!
[flagged]
[flagged]
We've banned this account for repeatedly posting abusive comments. We've asked you before to observe the guidelines, and the recent trend is getting much worse. When this happens we eventually have to assume you want to be banned. If you don't want to be banned, you can email us (hn@ycombinator.com) and commit to using HN as intended.
[flagged]
The moral of the story is: we don’t even need one 9.
OK now people can layoff even more engineers and feed the tasks to AI.
/s