Rackspace hosted Exchange suffered a catastrophic failure beginning December 2, 2022 and is still continuous since 12:37 AM December 4th. Initially described as connectivity and login problems, the assistance was ultimately updated to announce that they were handling a security event.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be dealt with.
Clients on Buy Twitter Verified reported that Rackspace was not reacting to support e-mails.
This has been quite the day with #Rackspace. Every hosted exchange customer has been down for 14 hours or two. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace consumer privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Uncertain the number of companies that is, however it’s significant.
They’re serving a 554 long delay bounce so people emailing in aren’t knowledgeable about the bounce for a number of hours.”
The main Rackspace status page offered a running upgrade of the blackout however the initial posts had no details aside from there was an outage and it was being examined.
The very first authorities update was on December 2nd at 2:49 AM:
“We are investigating a concern that is affecting our Hosted Exchange environments. More information will be published as they appear.”
Thirteen minutes later Rackspace began calling it a “connection concern.”
“We are examining reports of connection concerns to our Exchange environments.
Users might experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their email customer(s).”
By 6:36 AM the Rackspace updates explained the continuous problem as “connectivity and login issues” then later on that afternoon at 1:54 PM Rackspace revealed they were still in the “examination stage” of the blackout, still trying to find out what went wrong.
And they were still calling it “connection and login concerns” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
Four hours later Rackspace described the scenario as a “significant failure”and began providing their consumers complimentary Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround up until they comprehended the problem and could bring the system back online.
The official assistance specified:
“We experienced a substantial failure in our Hosted Exchange environment. We proactively closed down the environment to prevent any more issues while we continue work to restore service. As we continue to resolve the source of the issue, we have an alternate service that will re-activate your ability to send and receive e-mails.
At no charge to you, we will be supplying you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 until additional notice.”
Rackspace Hosted Exchange Security Incident
It was not till nearly 24 hr later at 1:57 AM on December 3rd that Rackspace formally revealed that their hosted Exchange service was suffering from a security event.
The announcement even more revealed that the Rackspace technicians had powered down and detached the Exchange environment.
“After further analysis, we have identified that this is a security event.
The recognized effect is separated to a part of our Hosted Exchange platform. We are taking essential actions to evaluate and secure our environments.”
Twelve hours later on that afternoon they upgraded the status page with more information that their security team and outdoors professionals were still dealing with solving the outage.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has actually not launched information of the security event.
A security occasion usually involves a vulnerability and there are 2 extreme vulnerabilities presently in the wile that were patched in November 2022.
These are the 2 most existing vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to check out and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an attacker is able to run destructive code on a server.
An advisory published in October 2022 described the effect of the vulnerabilities:
“A verified remote assailant can perform SSRF attacks to escalate opportunities and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mail box server, the assaulter can possibly gain access to other resources via lateral motion into Exchange and Active Directory site environments.”
The Rackspace blackout updates have actually not indicated what the specific issue was, just that it was a security event.
The most existing status update as of December fourth stated that the service is still down and clients are encouraged to move to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make development in dealing with the event. The accessibility of your service and security of your data is of high value.
We have devoted extensive internal resources and engaged first-rate external know-how in our efforts to minimize unfavorable impacts to clients.”
It’s possible that the above noted vulnerabilities are related to the security incident affecting the Rackspace Hosted Exchange service.
There has actually been no statement of whether consumer details has actually been jeopardized. This event is still continuous.
Included image by Best SMM Panel/Orn Rin