H/T to reader Skip..
Microsoft 365 took a broad hit today (Nov. 12). Outlook was the epicenter, but Teams, Copilot, OneDrive, SharePoint, Office on the web, and even Defender XDR were caught in it.
Peak: roughly 130,000 Outlook reports on Downdetector around 1 p.m. ET, with tens of thousands more on 365 and Teams.
Scope: started in the U.S./Americas, then Microsoft acknowledged all regions were affected.
Not resolved cleanly: by late afternoon they were still at it — "We're opening all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief." That's not a confident postmortem voice.
What Microsoft actually said the cause was: a large number of servers became unexpectedly unavailable, failed automated health checks, and the remaining infrastructure got strained by the load. They explicitly said the elevated load was not a traffic spike. They also said they found no recent changes that correlate with the start of impact — which is corporate for "we don't know why our own servers fell over."
Worth noticing: AWS and Google users were reporting problems at the same time. Amazon insisted its services were "operating normally" and blamed "an unrelated disruption elsewhere on the internet" for inaccurate reports. When three major providers wobble in the same window, the interesting question is what they share — DNS, CDNs, transit providers, upstream dependencies — and that's precisely the layer nobody gives straight answers about. but "coincidence" isn't a technical explanation either.
Wierd my Microsoft Outlook work email disabled but I can get it on the cellphone.
ReplyDeleteNothing like putting so much of your world on a internet system, eh?
Haven’t been getting emails on my Live/Outlook account since mid afternoon. Still not working at 2048 CT
ReplyDeleteIf the average person knew the condition of the fiber-optic cables these networks run on, they would be amazed anything works at all. I run a small fiber contracting company that does a find-and-fix for one of the network providers for these big companies. Based on what I've seen, there should be no fear of AI taking over. One good thump on any number of splices and the whole thing goes down with loss of signal.
ReplyDeleteSame thing with the Inside plant, all the companies view technicians as expendable. Wages have been stagnant for over 20 years, a good tech can diagnose and resolve a problem quickly. Nobody is interested in a quick MTTR anymore, as long as they get a body out there to work the issue. Until the FCC or the Commerce comission start fining the piss out of them it will stay that way.
DeleteSame here in the UK
ReplyDelete"Not a Hack. Not a Mystery. Just One Company Too Big to Fail, Grading Its Own Outage." Microsoft already told you it wasn't a cyberattack. So drop the dramatic frame — this wasn't an adversary, it was a config change, a health-check system that ejected servers into a stampede, and a "fix" that didn't hold. Fourteen hours, 70+ countries, Congress and two spy agencies locked out of their own email.
ReplyDeleteHere's the eye-opener: the actual postmortem — timeline, root cause, remediation — goes to enterprise admins through a portal Microsoft owns, on a timeline Microsoft's own service description sets, in a format Microsoft picks. No public version promised. Personal Outlook users aren't even in the room.
So the company that caused the outage is the company that writes the report, decides who reads it, and decides when the 48-hour clock starts. That's not accountability. That's a closed loop with a status page bolted on.
One config change, four continents, zero public sentences owed. That's what "too big to fail" actually looks like.
While I agree with the assessment, I just can't get behind blindly trusting anyone who says it wasn't an attack. I've had healthcare admins openly tell the media they were not hacked/compromised while I'm simultaneously fending off a ddos incursion. Knowing how fragile the systems really are 1 or 2 dedicated people can easily cripple a system and have it look like multiple other issues. Our power grid is no different. Alternating micro feedback can cause undo strain on a transformer causing it to wear out quickly, or even blow, but if you look at it system wide it would appear to be nothing more than normal variance.
DeleteMicrosft has always sucked, this should come as no surprise.
ReplyDeleteBACK TO THE TIDDIES!!!
Switched to pure Linux last year; no problem
ReplyDeleteJust know all the pathways they opened leave you exposed
ReplyDeleteBill Gates tried to rule the world; now he throws a temper-tantrum because Elon Musk does rule the world.
ReplyDeleteThis is what we get for allowing critical infrastructure hire nothing but H1B’s. The code is junk and it barely functions when everything is perfect. It can be fixed but, corporate looses the tax break.
ReplyDeleteKen in Atlanta
it was an H1b failure.
ReplyDelete