Being one of the 1% sucks if you're a Rackspace user

[theregister.com] 1 month ago

As the Rackspace email fiasco approaches week three with the company's hosted Exchange customers' data in limbo, Rackspace execs still won't put an exact number on how many customers were affected by the ransomware-induced email outage, or when — if — they'll be able to recover their old messages and contacts.

When asked for an exact customer count, "It's 1 percent of our overall company revenue," Rackspace Chief Product Officer Josh Prewitt told The Register, confirming a figure from an earlier filing with the US Securities and Exchange. 

Rackspace also told the SEC that this 1 percent, which brings in about $30 million in annual revenue, is "primarily small and medium business" customers, who are likely bearing the financial brunt of losing access to their users. When asked how the company can recover from the security snafu and gain back customer trust, Prewitt returned to the 1 percent.

"From an overall perspective for Rackspace, at 1 percent of the company's revenue it's not as impactful to the company overall," he said in an interview. "But I recognize that we've lost a lot of customer trust in this process. The best way for us to recover from that is to first and foremost, help customers gain access back to their data, and help them to be able to restore it through whatever means necessary."

One Reg reader, Erin Lutz, said that while math-wise, this may be an "isolated disruption" because it only hit 1 percent of Rackspace's $3 billion business, it's still a slap in the face to the affected companies. 

"It's  insulting to say this when you literally took down thousands of businesses — people's livelihoods — and then neglected to help them or provide timely updates on what was happening," she said in an email. "To me, that is basically telling small businesses that they don't care — that this is a side business to them — and that they are not in touch with the small business community."  

Prewitt defended the company's response. "I've seen the criticism on communications," he said. "The important thing here is we don't want to say anything and ever have to walk it back. We want everybody, all of our customers, the industry, all of the media, we want them to know that when we say something, it's true."

He said Rackspace looks forward to sharing all the gory details about what happened and what the company learned along the way. But not quite yet. 

"The more that we can share our findings and our learnings, share the root cause, share how we responded to it, and let the tech industry and security industry learn from it, the better off everybody's gonna be," Prewitt said. "And I also think that's gonna go a long way in helping to restore trust."

It's all very complicated

Prewitt said as of now, three-quarters of affected users have been moved from Rackspace's hosted environment to Microsoft Office 365 and now are able to send and receive emails. But as these affected users have said on social media, Reddit threads, and in conversations with The Register, many of them lost years' worth of data that they haven't been able to recover after a cyberattack took down the hosted Microsoft Exchange email on December 2.

On December 19, in an update posted on its Hosted Exchange Issues status page, the company said it was making progress in its data recovery efforts and would start "transferring data to our Hosted Exchange email environment customers in the coming days."

Customers will soon be able to download their historical data from the control panel, Prewitt said. "We strongly believe that the vast majority of customers are going to have all access restored to their data," he added. "And we're working through that process right now. We've already enabled it for a number of customers."

The process takes time. "But that process is well underway," he said. "We're scaling it up at this point, we already enabled it for tons of our customers, and it's going to continue to accelerate and scale quickly over the next few days and week."

When asked how many customers have regained access to their data, Prewitt said he didn't have an exact count. "It's definitely a significant number," he said.

He also said he can't give a timeframe for when all of the affected customers will be able to recover the contents of their old inboxes. "For some particular mailboxes and in some instances, we'll run into some issues," Prewitt said. "But I would expect that we've made tremendous progress within the next few days to one week."

Whether or not all customers will be able to recover all of their data, also gets an it's-complicated answer. 

Rackspace has recovered all of the PST files on the compromised servers, Prewitt said. But that doesn't guarantee all of the data stored on all of the files will be recovered, he added. 

"There's a chance that an Exchange PST file could be a corrupt file, there's a chance there would be something wrong with an individual customer's PST, so it would be premature for me to say absolutely every single customer, 100 percent" will recover their data, Prewitt said. 

"We strongly believe it's going to be a vast majority of our customers," he added.

The company also hasn't decided if it'll kill the legacy hosted service after the incident response team wraps up, either. "Our main priority is how do we help customers access their data, how do we get them back up and running," Prewitt said, adding that migrating them to Microsoft Office 365 is the best fix for now. "But we're not ready to make a decision yet on the hosted Exchange platform long term."

In a similar vein, the company's not ready to disclose how the intruders gained access to the hosted Exchange servers, the amount of the ransom demand, or who is behind the attack. 

One of Rackspace's external advisors said the attack was not related to any ProxyNotShell vulnerabilities, despite widespread speculation. The advisor described the attack as "fairly unique," and promised the upcoming forensic investigation would reveal more details.

The company also declined to comment on the growing class-action lawsuit, which now includes nationwide claims as well as state-specific charges for 18 states [PDF]. ®