We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.
Most likely, those were debris from the interception of missiles flying overhead and being destroyed on their way to a military target.
AFAIK, there have been no confirmed signs of civilian sites being targeted directly, and it would also be unlikely that actual missiles would cause so little damage that you could patch your datacenter up and get it ready to go within hours.
Thanks for the links, which I've reviewed. Allow me to clarify: I meant sources that confirm that the civilian places hit (eg. hotels and residential buildings) were the actual targets.
Local and official news all say that these were hit by debris from intercepted missiles/drones (on their way to somewhere else). There is a major difference between this, vs. if those buildings were directly being targeted.
AFAICT your linked sources indicate that the oil installations and ports were targets, but not the hotels and buildings.
I'm asking in good faith as this makes a significant difference.
I don't see the large difference between a civilian port, a civilian oil facility or a civilian aluminum factory vs a hotel on the topic of whether the Iranians are capable of targeting a civilian data center, however, assuming you are curious, here goes
Finding these take time so I am sorry if this is going to be the last of these sources I'll paste, for example Bahrain luxury apartments building being hit:
This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc…
Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances.
A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for.
“Oh, that’s in case there’s a fire!”
“What does it do?”
“It cuts… the power… uhh… for the safety of the fire department.”
“So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?”
Equinix in Sydney plonked 2 datacenters right on top of each other, and still insists that they are useful as redundant sites.
There was a locally very funny situation for a while when a tech influencer was insisting both equinix sites could be shut down by a single building collapse. He was wrong, but he wasn't so wrong that people shouldn't make better infrastructure decisions.
The building in question wasnt really tall enough. And would have to be precision demolished to collapse in the way he was afraid of.
It would still cause chaos and possible power issues.
Needs to be taken in context with some Sydney buildings having maintenance defects a few years after they open. Largely due to inferior materials imported from china. The building in question developed some cracks in supporting beams and was briefly evacuated. There was never a chance it was going to topple on its own in a way that impacted more than 1/2 datacenters, so he pivoted to possible terrorism, but even thats largely nonsensical.
I just went hunting for the case and couldnt find it. The gentleman in question had published the claim to his business that was it happens trying to build contacts with defense and intelligence agencies for third party threat assessment. As far as I can tell the business no longer exists and he has deleted their footprint.
But he also posted the claim on public mailing lists so I can probably trawl it up if necessary.
My boss at my first job hit the Big Red Button by swinging his arms too big in our datacenter one day, shutting down hundreds of servers and the mainframe, wreaking havoc for days!
That was when we installed the Big Clear Button Cover.
> we will ensure we restore power and connectivity safely
this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost.
IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the relevant team in AWS. What if they try to bring the service back but then missiles are struck again and what if human lives might be lost on it. Let's just hope that it could be part of a evaluation as well.
But I mean,are the employees safe at home?
I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?
> But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?
My gut feeling says that they would be safer at homes than at datacenters. The only large info I have heard is attack on hotels, this datacenter etc. (atleast till right now).
> but in the fog of war maybe the data center wasn't the target?
We can't say this for sure but even if that was the case, I do think that they would see some damage was caused and then, try to double tap it for even more damage. So chances are, even if it wasn't the target previously, it might be the target now?
100% absolutely but its a bit worrying if in the future multiple AZ/datacenters could be start to get targeted?
attacking datacenters within a particular region so that service would have a hard time.
I guess someone can use some other regions DC to have more than (regional?) AZ but for mission critical infra, I can see that having sometimes issue too and you genuinely can't predict any of all of this.
That being said, There should be more than one AZ reliance but IMO also off-site or multi-cloud backups should also be preferred/used as well.
> What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.
Off-site backups/Multi Cloud Strategy while encrypting data (and keeping the key safe, key point) might be a better strategy for such mission critical infrastructure.
I'm sure bezos will be really happy someone is being a hero for him in a war zone while he sails his newest yacht to wherever the new version of the island is.
on second thought there is a difference between restoring critical infrastructure in times of crisis vs restoring bot infrastructure for indian spamming operations. choose wisely
Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future.
Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services.
Again, this is just me thinking out loud on a tangent and this doesn't have much to do with this story, but I felt it was an interesting thought to share nonetheless.
The more interesting question, is how many datacenters are just plonked next to a high-value military target?
For infrastructure reasons, we plonk datacenters down next to airports big enough to fly major hardware into, and near where the big oceanic cables come ashore… and for strategic reasons those are also the perfect places to place military bases
You sound like you know things, and I have questions! What are the chances that we see infrastructure move to less-developed-but-more-stable regions of the world? What even are the candidate locations that wouldn't be an absolute headache to set up? I'm thinking perhaps West Africa? Or South Africa if it can stabilize a bit? Maybe other coastal locations that have good-enough transportation hubs nearby?
Data centers in space no longer look so unreasonable when the requirement is “redundancy against multi site bomb strikes mid op”. A little depressing when some pieces start to fit together.
I’m not exactly fully bought into the idea (for many practicality reasons) but it seems easier to build many (and replace) ground stations than data centers.
Additionally, StarLink et al are now able to directly communicate with cell phones. It therefore should be possible to route entirely in space between “data center satellites” and communications satellites and communicate directly with an end user device, avoiding the entire terrestrial internet.
Routing? Why route? the link at the point of comms is direct. There's no extra ground station, unless you mean the person/people doing things. But either they're communicating direct to a DC on the ground, or one in space, or wherever. Setup so that-- sure-- they can work and coordinate better if someone isn't dropping bombs on ground nodes but that all nodes have some independent capabilities as well.
That's so interesting. Are any of the US military (or other satellite state of the US) systems running in "normal" datacenters or do they have a few protected DoD datacenters in the US?
I do think that though, atleast from the Anthropic decision prior, we know that Anthropic which was used by DoD should be on normal AWS datacenters.
I am saying this because, Dod Threatened to force take the source code of Anthropic if they don't agree to aggregious demands so that means that they don't have the source code.
Perhaps DoD used Anthropic within AWS Military modular DC's but I find it extremely unlikely.
I am almost certain that even with OpenAI who bent its knee to DoD, its still hosted on regular infrastructure and DoD is using these AI models on pretty sensitive tasks (During the Venezeula Maduro's capture, Anthropic/Claude were used iirc to handle some data analysis)
IMO Tho, Any Employee from Anthropic/OpenAI might know better tho about how these models are deployed.
> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.
The bigger part of me seems that if we someone nukes 50 datacenters all at once or say all of Amazon's datacenters at once, then the data stored in there would simply be gone and given so many datacenters are located in Virginia,USA iirc or just so many companies being reliant on few datacenter providers.
The larger threat to me with the lose of data is firstly the panic within public fronting services but also, with Hedge Funds, Pension funds or banking datacenters who might be using these and if they lose the data, then its gonna cause even more public mayhem.
Some might be saying oh off-site backups exist but there has atleast been one instance, where a single Google accident had led to massive issues for a 135 Billion $ pension fund.
In my experience many middle eastern companies tend to only operate out of the Middle East AZ’s. They’re not backhauling their data and customers to us-east-1. If the goal is to severely disrupt middle eastern rivals, then you don’t need to hit every possible AWS datacenter.
Interesting, I didn't know that so thanks for telling me something new.
But why is this the case? Like, saving costs? Doesn't this recent attack on AWS DC does show that they aren't as safe as previously thought especially in a region of conflict.
Is there any particular reason as to why this is the case?
Notably they did have backups. As you would expect for a $135 billion dollar undertaking. It's just that restoring from a calamity tends to be time consuming (a key difference between failover and a backup).
IIUC part of the reason ballistic missiles have multiple warheads is that some of them detonate high up to knock out air defenses and other electronics allowing the rest to fall through to their targets. The last time we tried this experiment as a species was the starfish prime tests in 1962 which caused some electrical havoc in Hawaii. These days our systems are probably more delicate and sensitive? All that is to say, in a scenario where nukes are going off I'm not sure you'd even need to target any datacenters in particular.. they're probably all toast by default.
> The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event.
> Due to the ongoing conflict in the Middle East, both affected regions have experienced physical impacts to infrastructure as a result of drone strikes.
It depends on the service if things move gracefully or not. The incident explains it's only EC2 (and dependent services) in that AZ, so if they try to route traffic for services hosted on EC2 to that AZ it's not working (and customers running instances in that AZ have lost access).
The other ones are not impacted. They always like to tell you to pay for more than one instance in different AZs so if this happens you don't get impacted.
They've been preparing for this for 40+ years - this has been the existential threat for Iran since the revolution. You're right they can't win, but I'd expect this to get messy.
> I’ve done some light game theory reading - but this isn’t covered - if they’re gone, who are they getting better terms for?
I think the idea can be that they are getting worse terms for America. Saddam Hussein from Iraq lost as well but the war was so messy that so so many americans died in bloodshed and gruesome ways and there was a whole gulf war syndrome within Army with severe PTSD.
This makes the people highly unreceptive of war and the govt should have more pressure from people. So in theory, America should've thought twice before attacking the same region.
I also think, that its hard to ever really remove an ideology. You definitely see ideologies of the past spring up in future and I think that sure they might lose the govt power but they would still have some influence for what its worth.
Also, Ayatollah was killed by these forces. Atleast the highly religious of people of Iran/politicians might spin the story to become religious which does have such ideas of inflicting maximum damage to the opponent at times.
I don't think that game theory can be applicable for religious/personal matters at times
Looking at Google Maps, there's Al Dhafra Air Base a couple of miles to the datacenter's south, an oil refinery a bit to the east, ports to the north, and a military academy to the west.
Why not? It's a physical building with lots of equipment that produces products shipped to its customers.
Its products are sequences of electrons, instead of atoms. But so are power plants. And in the context of what happens when they're hit by missiles, a factory, data center, and power plant all behave the same.
Google Cloud also has middle east locations. As does Azure, Oracle and Alibaba. Afaik, IBM Cloud does not. I think those five and AWS are the top 6 global public access clouds.
Yes of course. But they also have to know that concentrated compute is an attractive target for marauding countries that decide on a whim to bomb you from outside.
Not really, although things may be trending in the wrong direction. Pick any “weird” current event and you can find much worse examples in relatively modern history. Just compare whatever you think of to the siege of Leningrad, for instance.
Has this ever happened ever in history of Cloud providers before this because of war?
They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was caused.
Moving forward, will there be a demand (all be small) for nuclear bunker esque datacenters which can withstands missiles? I know absolutely nothing about constructing underground but can explosives not be used to create underground datacenters comparatively cheaply? One can also use revamped Nuclear bunkers (although the scale of AWS datacenters might be huge tho who knows)
I am curious but what are the safety attempts made by Internet Exchange Providers or (had to search it up) but Submarine Cable landing stations, to me it feels like blowing these up leads to internet downtime across whole country / between providers.
Historically in the US, some portion of Bell installations were designed to be resistant to attack. But it comes at large expense for construction and maintenance. Underground facilities also bring increased risk of flooding.
Competition and deregulation and lack of attacks leads towards less robust installations to reduce costs. Geographically redundant installations help as long as all installations aren't targetted; and are valuable for operational concerns other than just attacks.
Data centers are usually built to withstand local natural risks e.g. weather. All bets, SLAs, and insurance are usually off when it comes to acts of war.
There's also just an upper limit to the kind of risk you can reasonably defend against.
An out-of-control wildfire levels the entire city? The Big One hits the Bay Area? The entire city is flooded for a few months because the levees break during a Cat5 hurricane? Yeah, your DC will be completely ruined. And even if it isn't, you're probably not getting any outside power, generator fuel, or repair technicians for a while.
No matter how much money you pump into hardening your own super-bunker DC, there will always be disasters you aren't prepared for. At a certain point it just makes more financial sense to abandon the idea of invulnerability and build a redundant site a few states over. Accept that you will occasionally lose one, and only protect against incidents where mitigation is cheaper than occasionally rebuilding.
In terms of deliberate attacks on a DC, an adversary that can bomb a blast resistant bunker can easily bomb the data lines going in and out, to say nothing of external power supplies. I guess you can at least repair the lines after the attack, but it's pretty easy to knock it offline if that's your goal. I just don't see the value in making a DC blast resistant compared to the huge costs.
That's what I'm trying to understand too. It's this a meteor,tree,etc? Or a human made object,and if so accidental or intentional one. Further risk assessment would be dependent on root cause.
The earliest phrasing I saw internally was "Root cause is identified as a drone attack to DXB61 site". That's somewhat open to interpretation, and could also have simply been incorrect. It was scrubbed from the ticket, though, and it now merely vaguely gestures toward a "power event". The ticket I'd expect to have further detail was locked down.
I actually like the way they said it. I don't know if it's a different cultural tradition, but the cool steely-eyed fact-based conversation always really felt so much more inspiring:
Conrad: I got three fuel cell lights, an AC bus light, a fuel cell disconnect, AC bus overload 1 and 2, Main Bus A and B out.
Aaron: Flight, EECOM. Try SCE to Aux.
Modern culture in the movies and whatnot is that someone should be yelling "Everything's failing. Give me something, Houston. All lights are on! MAYDAY MAYDAY!" and some sort of flavour commentary like that. But reading engineering updates that go like this feels like watching maximal professionalism under fire:
> At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event. EC2 Instance launches will continue to be impaired in the impacted AZ. We recommend that customers continue to retry any failed API requests. If immediate recovery of an affected resource (EC2 Instance, EBS Volume, RDS DB Instance, etc.) is required, we recommend restoring from your most recent backup, by launching replacement resources in one of the unaffected zones, or an alternate AWS Region. We will provide an update by 12:30 PM PST, or sooner if we have additional information to share.
This has that same mechanical tone of an ice-cold captain dealing with a proximate situation providing exactly the information they know. No flavour commentary. Amazing. I fucking love it.
The actual Whitest Kids You Know Youtube channel has been reuploading old skits with the title changed and maybe a single word audio dub as direct criticisms of this administration's horseshit.
It's insane. Jokes written 20 years ago work better today.
It's been extremely on point and it's sad how much criticism from the mid 2000s which was dismisses as "Not supporting America" was actually 100% on point and still fully valid and correct and prescient and on point 20 years later because the same exact old assholes are still in power and calling the shots and finally allowed to go weapons free on the geopolitical agenda's they've held for decades.
People didn't listen that the Bush Jr admin, who started middle east wars under fabricated pretenses and booted up the surveillance regime with the PATRIOT act and called war protests "Anti-american", was Fascist and they still aren't listening as a racist, sexist, rapist moron drives the US off the deep end and starts even more illegal and false wars and directs that same DHS Bush created to destroy the enemies of the state within.
In Southern Europe some smaller web servers are intermittently not working, while big servers like YouTube are working fine. But I don't think it is related.
The important quote from the timeline:
Mar 01 9:41 AM PST
We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.
> impacted by objects
Well that's a way of phrasing it.
I assume these were missiles.
Most likely, those were debris from the interception of missiles flying overhead and being destroyed on their way to a military target.
AFAIK, there have been no confirmed signs of civilian sites being targeted directly, and it would also be unlikely that actual missiles would cause so little damage that you could patch your datacenter up and get it ready to go within hours.
That's incorrect, there have been multiple hotels being attacked and recently oil facilities in SA
Can you please share sources?
refineries, gas fields - https://www.reuters.com/business/energy/saudi-aramco-shuts-r...
dubai airport, residential buildings, hotels, ports - https://www.reuters.com/world/middle-east/several-loud-blast...
https://www.lemonde.fr/en/international/article/2026/03/01/i...
Bahrain - https://bh.usembassy.gov/security-alert-update-5-u-s-embassy...
You can google for other targets in Qatar, Oman, Jordan, Iraq, Cyprus and Kuwait
Thanks for the links, which I've reviewed. Allow me to clarify: I meant sources that confirm that the civilian places hit (eg. hotels and residential buildings) were the actual targets.
Local and official news all say that these were hit by debris from intercepted missiles/drones (on their way to somewhere else). There is a major difference between this, vs. if those buildings were directly being targeted.
AFAICT your linked sources indicate that the oil installations and ports were targets, but not the hotels and buildings.
I'm asking in good faith as this makes a significant difference.
I don't see the large difference between a civilian port, a civilian oil facility or a civilian aluminum factory vs a hotel on the topic of whether the Iranians are capable of targeting a civilian data center, however, assuming you are curious, here goes
Finding these take time so I am sorry if this is going to be the last of these sources I'll paste, for example Bahrain luxury apartments building being hit:
https://edition.cnn.com/world/video/bahrain-iran-drone-strik...
US warning that high rise buildings in Bahrain are being targeted by Iranians drones:
https://x.com/TravelGov/status/2027843430987010446
Impact assessment: yes.
This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc…
Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances.
A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for.
“Oh, that’s in case there’s a fire!”
“What does it do?”
“It cuts… the power… uhh… for the safety of the fire department.”
“So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?”
“… yes.”
“Not 100% then, is it?”
Equinix in Sydney plonked 2 datacenters right on top of each other, and still insists that they are useful as redundant sites.
There was a locally very funny situation for a while when a tech influencer was insisting both equinix sites could be shut down by a single building collapse. He was wrong, but he wasn't so wrong that people shouldn't make better infrastructure decisions.
The two in Alexandria are of course just down the road from the airport. A widebody jet cratering in that location will put them both out of business.
None of the runways point towards them, its hard to imagine that scenario.
34R does. The immediate right turn in the 34R ILS and RNP missed approach procedures takes you pretty much directly overhead Equinix.
Aircraft routinely overfly the location either on departure from 34R or approach for 16L.
They're also below sea level.
Theres an Energex office building in Brisbane that has a backup generator below sea level. (Guess when they need to use the generator lmao.)
I'm intrigued...how was he wrong?
The building in question wasnt really tall enough. And would have to be precision demolished to collapse in the way he was afraid of.
It would still cause chaos and possible power issues.
Needs to be taken in context with some Sydney buildings having maintenance defects a few years after they open. Largely due to inferior materials imported from china. The building in question developed some cracks in supporting beams and was briefly evacuated. There was never a chance it was going to topple on its own in a way that impacted more than 1/2 datacenters, so he pivoted to possible terrorism, but even thats largely nonsensical.
I just went hunting for the case and couldnt find it. The gentleman in question had published the claim to his business that was it happens trying to build contacts with defense and intelligence agencies for third party threat assessment. As far as I can tell the business no longer exists and he has deleted their footprint.
But he also posted the claim on public mailing lists so I can probably trawl it up if necessary.
Should have pushed it.
at The Planet in Dallas c. 2002 the EPO button was exposed with no cover, and in very very close proximity to the "Exit" button for the doors...
one day, a colo customer hit the wrong button on the way out, and uhh, there was an outage
Obligatory far side cartoon: https://static1.srcdn.com/wordpress/wp-content/uploads/2023/...
My boss at my first job hit the Big Red Button by swinging his arms too big in our datacenter one day, shutting down hundreds of servers and the mainframe, wreaking havoc for days!
That was when we installed the Big Clear Button Cover.
> we will ensure we restore power and connectivity safely
this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost.
IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the relevant team in AWS. What if they try to bring the service back but then missiles are struck again and what if human lives might be lost on it. Let's just hope that it could be part of a evaluation as well.
I wouldn't risk it.
Both Americans and Israelis are known for double taps. Surely Iran can adapt their tactics too.
But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?
> But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?
My gut feeling says that they would be safer at homes than at datacenters. The only large info I have heard is attack on hotels, this datacenter etc. (atleast till right now).
> but in the fog of war maybe the data center wasn't the target?
We can't say this for sure but even if that was the case, I do think that they would see some damage was caused and then, try to double tap it for even more damage. So chances are, even if it wasn't the target previously, it might be the target now?
> this would require human intervention
Amazon has self-propelled robots that handle their logistics and fulfillment, don't they? Send in the robots.
> this would require human intervention
that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month.
same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.
Man, don’t be a “hero” trying to restore a lower ping to someone trying to buy a kindle in Jeddah.
What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.
Their lack of multiple AZ’s isn’t the guy making 30k a year’s problem.
In that case, the hero was the person who avoided relying on a single AZ when they deployed to cloud.
100% absolutely but its a bit worrying if in the future multiple AZ/datacenters could be start to get targeted?
attacking datacenters within a particular region so that service would have a hard time.
I guess someone can use some other regions DC to have more than (regional?) AZ but for mission critical infra, I can see that having sometimes issue too and you genuinely can't predict any of all of this.
That being said, There should be more than one AZ reliance but IMO also off-site or multi-cloud backups should also be preferred/used as well.
> What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.
Off-site backups/Multi Cloud Strategy while encrypting data (and keeping the key safe, key point) might be a better strategy for such mission critical infrastructure.
I'm sure bezos will be really happy someone is being a hero for him in a war zone while he sails his newest yacht to wherever the new version of the island is.
on second thought there is a difference between restoring critical infrastructure in times of crisis vs restoring bot infrastructure for indian spamming operations. choose wisely
Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future.
Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services.
Again, this is just me thinking out loud on a tangent and this doesn't have much to do with this story, but I felt it was an interesting thought to share nonetheless.
The more interesting question, is how many datacenters are just plonked next to a high-value military target?
For infrastructure reasons, we plonk datacenters down next to airports big enough to fly major hardware into, and near where the big oceanic cables come ashore… and for strategic reasons those are also the perfect places to place military bases
You sound like you know things, and I have questions! What are the chances that we see infrastructure move to less-developed-but-more-stable regions of the world? What even are the candidate locations that wouldn't be an absolute headache to set up? I'm thinking perhaps West Africa? Or South Africa if it can stabilize a bit? Maybe other coastal locations that have good-enough transportation hubs nearby?
Is there acrually some meaningful physical separation between military and civilian server deployments?
We seem to be really bad at separating those two. For example Starlink is basically military infrastructure now, used to guide bombs.
Arpa net
i do believe that was decommissioned in the late 80s
A datacenter IS a high value military target.
Exactly. 2 is only sufficient for HA against random failures. It's not enough for HA against a determined adversary willing to use targeted force.
> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.
Previous outage news makes it sound like the cloud providers still have quite a few logical single points of failure.
> doesn't seem it was the case in this instance
To me it absolutely looks like the case in here, what are the chances three availability zones across two different countries have been attacked?
Data centers in space no longer look so unreasonable when the requirement is “redundancy against multi site bomb strikes mid op”. A little depressing when some pieces start to fit together.
How would that help? The ground station that handles local routing to and from the orbital station becomes the target.
I’m not exactly fully bought into the idea (for many practicality reasons) but it seems easier to build many (and replace) ground stations than data centers.
Additionally, StarLink et al are now able to directly communicate with cell phones. It therefore should be possible to route entirely in space between “data center satellites” and communications satellites and communicate directly with an end user device, avoiding the entire terrestrial internet.
Routing? Why route? the link at the point of comms is direct. There's no extra ground station, unless you mean the person/people doing things. But either they're communicating direct to a DC on the ground, or one in space, or wherever. Setup so that-- sure-- they can work and coordinate better if someone isn't dropping bombs on ground nodes but that all nodes have some independent capabilities as well.
That's so interesting. Are any of the US military (or other satellite state of the US) systems running in "normal" datacenters or do they have a few protected DoD datacenters in the US?
Found this relevant article: https://serverlift.com/blog/military-modular-data-centers/ (AWS Military Modular Data centers)
I do think that though, atleast from the Anthropic decision prior, we know that Anthropic which was used by DoD should be on normal AWS datacenters.
I am saying this because, Dod Threatened to force take the source code of Anthropic if they don't agree to aggregious demands so that means that they don't have the source code.
Perhaps DoD used Anthropic within AWS Military modular DC's but I find it extremely unlikely.
I am almost certain that even with OpenAI who bent its knee to DoD, its still hosted on regular infrastructure and DoD is using these AI models on pretty sensitive tasks (During the Venezeula Maduro's capture, Anthropic/Claude were used iirc to handle some data analysis)
IMO Tho, Any Employee from Anthropic/OpenAI might know better tho about how these models are deployed.
they have a press release stating that Claude is available on bedrock in secret regions
https://aws.amazon.com/about-aws/whats-new/2025/11/amazon-be...
https://aws.amazon.com/federal/secret-cloud/
> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.
The bigger part of me seems that if we someone nukes 50 datacenters all at once or say all of Amazon's datacenters at once, then the data stored in there would simply be gone and given so many datacenters are located in Virginia,USA iirc or just so many companies being reliant on few datacenter providers.
The larger threat to me with the lose of data is firstly the panic within public fronting services but also, with Hedge Funds, Pension funds or banking datacenters who might be using these and if they lose the data, then its gonna cause even more public mayhem.
Some might be saying oh off-site backups exist but there has atleast been one instance, where a single Google accident had led to massive issues for a 135 Billion $ pension fund.
Relevant Kevin Faang video about it: https://www.youtube.com/watch?v=3GOAUyipnM4 [Google Accidentally Deletes $135 Billion Pension Fund, Chaos Ensues]
In my experience many middle eastern companies tend to only operate out of the Middle East AZ’s. They’re not backhauling their data and customers to us-east-1. If the goal is to severely disrupt middle eastern rivals, then you don’t need to hit every possible AWS datacenter.
Interesting, I didn't know that so thanks for telling me something new.
But why is this the case? Like, saving costs? Doesn't this recent attack on AWS DC does show that they aren't as safe as previously thought especially in a region of conflict.
Is there any particular reason as to why this is the case?
Notably they did have backups. As you would expect for a $135 billion dollar undertaking. It's just that restoring from a calamity tends to be time consuming (a key difference between failover and a backup).
IIUC part of the reason ballistic missiles have multiple warheads is that some of them detonate high up to knock out air defenses and other electronics allowing the rest to fall through to their targets. The last time we tried this experiment as a species was the starfish prime tests in 1962 which caused some electrical havoc in Hawaii. These days our systems are probably more delicate and sensitive? All that is to say, in a scenario where nukes are going off I'm not sure you'd even need to target any datacenters in particular.. they're probably all toast by default.
This is the data center version of https://xkcd.com/538/. Realistically if there is a hot war what you’re saying seems accurate.
Just one AZ, not the whole region:
> The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event.
This has recently been updated:
> Due to the ongoing conflict in the Middle East, both affected regions have experienced physical impacts to infrastructure as a result of drone strikes.
So they're confirming actual aggression
Amazon usually has 3 AVs per region, looks like there are surviving AVs but the system didn't switch over gracefully.
I bet that was an interesting sev2 ticket!
It depends on the service if things move gracefully or not. The incident explains it's only EC2 (and dependent services) in that AZ, so if they try to route traffic for services hosted on EC2 to that AZ it's not working (and customers running instances in that AZ have lost access).
The other ones are not impacted. They always like to tell you to pay for more than one instance in different AZs so if this happens you don't get impacted.
I doubt that there is enough capacity to evict a full AV to another one.
Iran may be heavily overpowered but they play their cards well. They will lose the war but the economic damage to the gulf is irreparable
They've been preparing for this for 40+ years - this has been the existential threat for Iran since the revolution. You're right they can't win, but I'd expect this to get messy.
Irreparable? That's a big claim.
Long term I think if the regime in Iran changes and better economic cooperation becomes possible everyone will benefit.
Iran is currently winning the war.
> they play their cards well. They will lose the war
so…
so they self-destruct to force better terms of negotiation.
I’ve done some light game theory reading - but this isn’t covered - if they’re gone, who are they getting better terms for?
> I’ve done some light game theory reading - but this isn’t covered - if they’re gone, who are they getting better terms for?
I think the idea can be that they are getting worse terms for America. Saddam Hussein from Iraq lost as well but the war was so messy that so so many americans died in bloodshed and gruesome ways and there was a whole gulf war syndrome within Army with severe PTSD.
This makes the people highly unreceptive of war and the govt should have more pressure from people. So in theory, America should've thought twice before attacking the same region.
I also think, that its hard to ever really remove an ideology. You definitely see ideologies of the past spring up in future and I think that sure they might lose the govt power but they would still have some influence for what its worth.
Also, Ayatollah was killed by these forces. Atleast the highly religious of people of Iran/politicians might spin the story to become religious which does have such ideas of inflicting maximum damage to the opponent at times.
I don't think that game theory can be applicable for religious/personal matters at times
We have business in UAE. For whatever reason I defaulted to us-west-2 since these particular applications are not latency sensitive.
I wonder if this was bad targeting job or intentional. I appreciate the transparency and optimism in the status updates though!
Looking at Google Maps, there's Al Dhafra Air Base a couple of miles to the datacenter's south, an oil refinery a bit to the east, ports to the north, and a military academy to the west.
might just be shrapnel from capture ops
We are living in increasingly weirder times.
A factory not working because of a missile strike seems pretty classic actually.
Sure, but it's not a factory.
It's a big building with a lot of capital assets inside that are the means of production for a business…
Why not? It's a physical building with lots of equipment that produces products shipped to its customers.
Its products are sequences of electrons, instead of atoms. But so are power plants. And in the context of what happens when they're hit by missiles, a factory, data center, and power plant all behave the same.
When I first learned that there were AWS Middle East regions, my first thought was "wow they are more optimistic than I am ."
Google Cloud also has middle east locations. As does Azure, Oracle and Alibaba. Afaik, IBM Cloud does not. I think those five and AWS are the top 6 global public access clouds.
No, they are more aware of the customer demand for compute in the region.
Yes of course. But they also have to know that concentrated compute is an attractive target for marauding countries that decide on a whim to bomb you from outside.
I think you mean when the US and Israel decide to start an unprovoked war “on a whim”.
And demand for data sovereignty.
Public cloud or cloud run by external organisations can never provide data sovereignty, by definition.
Absolutely, especially in the KSA.
Not really, although things may be trending in the wrong direction. Pick any “weird” current event and you can find much worse examples in relatively modern history. Just compare whatever you think of to the siege of Leningrad, for instance.
Not really. It's just been pretty damn quiet for years.
Is this the one in Bahrain?
UAE. Abu Dhabi
Has this ever happened ever in history of Cloud providers before this because of war?
They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was caused.
Moving forward, will there be a demand (all be small) for nuclear bunker esque datacenters which can withstands missiles? I know absolutely nothing about constructing underground but can explosives not be used to create underground datacenters comparatively cheaply? One can also use revamped Nuclear bunkers (although the scale of AWS datacenters might be huge tho who knows)
Had some ideas which show that this idea might be interesting, https://www.nature.com/articles/s44284-026-00406-2
I am curious but what are the safety attempts made by Internet Exchange Providers or (had to search it up) but Submarine Cable landing stations, to me it feels like blowing these up leads to internet downtime across whole country / between providers.
Historically in the US, some portion of Bell installations were designed to be resistant to attack. But it comes at large expense for construction and maintenance. Underground facilities also bring increased risk of flooding.
Competition and deregulation and lack of attacks leads towards less robust installations to reduce costs. Geographically redundant installations help as long as all installations aren't targetted; and are valuable for operational concerns other than just attacks.
Cold War era definitely resulted in a lot of comms infrastructure being hardened against attack.
not even a decade ago for an example (NSVLTNMT):
https://en.wikipedia.org/wiki/2020_Nashville_bombing
> but given the situation, How does that prevent the situation from happening again
You don't. Instead, you make sure your failover or DR setup is regularly tested and works.
Data centers are usually built to withstand local natural risks e.g. weather. All bets, SLAs, and insurance are usually off when it comes to acts of war.
There's also just an upper limit to the kind of risk you can reasonably defend against.
An out-of-control wildfire levels the entire city? The Big One hits the Bay Area? The entire city is flooded for a few months because the levees break during a Cat5 hurricane? Yeah, your DC will be completely ruined. And even if it isn't, you're probably not getting any outside power, generator fuel, or repair technicians for a while.
No matter how much money you pump into hardening your own super-bunker DC, there will always be disasters you aren't prepared for. At a certain point it just makes more financial sense to abandon the idea of invulnerability and build a redundant site a few states over. Accept that you will occasionally lose one, and only protect against incidents where mitigation is cheaper than occasionally rebuilding.
In terms of deliberate attacks on a DC, an adversary that can bomb a blast resistant bunker can easily bomb the data lines going in and out, to say nothing of external power supplies. I guess you can at least repair the lines after the attack, but it's pretty easy to knock it offline if that's your goal. I just don't see the value in making a DC blast resistant compared to the huge costs.
> will there be a demand (all be small) for nuclear bunker esque datacenters
Those already exist. See for example Bahnhof's "Pionen - White Mountain" data center in Stockholm, or Cyberfort's "The Bunker" a bit west of London.
but but they said this is a cloud!
> one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire.
God forbid we'd ever say that it was struck by a missile or a munition in an act of war.
That's what I'm trying to understand too. It's this a meteor,tree,etc? Or a human made object,and if so accidental or intentional one. Further risk assessment would be dependent on root cause.
The earliest phrasing I saw internally was "Root cause is identified as a drone attack to DXB61 site". That's somewhat open to interpretation, and could also have simply been incorrect. It was scrubbed from the ticket, though, and it now merely vaguely gestures toward a "power event". The ticket I'd expect to have further detail was locked down.
Maybe a missile, maybe a drone, maybe debris
Doesn’t really matter, we know trumps latest war is the cause
I actually like the way they said it. I don't know if it's a different cultural tradition, but the cool steely-eyed fact-based conversation always really felt so much more inspiring:
Modern culture in the movies and whatnot is that someone should be yelling "Everything's failing. Give me something, Houston. All lights are on! MAYDAY MAYDAY!" and some sort of flavour commentary like that. But reading engineering updates that go like this feels like watching maximal professionalism under fire:> At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event. EC2 Instance launches will continue to be impaired in the impacted AZ. We recommend that customers continue to retry any failed API requests. If immediate recovery of an affected resource (EC2 Instance, EBS Volume, RDS DB Instance, etc.) is required, we recommend restoring from your most recent backup, by launching replacement resources in one of the unaffected zones, or an alternate AWS Region. We will provide an update by 12:30 PM PST, or sooner if we have additional information to share.
This has that same mechanical tone of an ice-cold captain dealing with a proximate situation providing exactly the information they know. No flavour commentary. Amazing. I fucking love it.
For those who wanna hear it https://youtu.be/-rqL035klC4?t=101
(Lightning at 1:03)
High signal/noise ratio is extra important when things are going badly
Potential moon bear attack; we're waiting on satellite imagery to confirm it. https://youtu.be/pvjgIxuVdo4?t=96
The actual Whitest Kids You Know Youtube channel has been reuploading old skits with the title changed and maybe a single word audio dub as direct criticisms of this administration's horseshit.
Their version from 2 days ago: https://www.youtube.com/watch?v=S3o-3pL79O4
It's insane. Jokes written 20 years ago work better today.
It's been extremely on point and it's sad how much criticism from the mid 2000s which was dismisses as "Not supporting America" was actually 100% on point and still fully valid and correct and prescient and on point 20 years later because the same exact old assholes are still in power and calling the shots and finally allowed to go weapons free on the geopolitical agenda's they've held for decades.
People didn't listen that the Bush Jr admin, who started middle east wars under fabricated pretenses and booted up the surveillance regime with the PATRIOT act and called war protests "Anti-american", was Fascist and they still aren't listening as a racist, sexist, rapist moron drives the US off the deep end and starts even more illegal and false wars and directs that same DHS Bush created to destroy the enemies of the state within.
In Southern Europe some smaller web servers are intermittently not working, while big servers like YouTube are working fine. But I don't think it is related.