Who, Me? Welcome once again to yet another Monday and another instalment of Who, Me? in which Register readers own up to the … let's say "learning experiences" … they've enjoyed up in their careers.
This week's eager learner is someone we'll Regomize as "Pat" who is a senior programmer and help desk supervisor for an agricultural concern. In that role he and a bunch of other bods enjoy admin privileges for their various domains, but – presumably for good reasons – each of the admins also has a regular user account without admin privileges. They do most of their business in their regular user accounts wherein no harm can be done, only employing the admin accounts when needed to do admin stuff.
It seems a sensible arrangement, really. Spend all your time in an admin account and things can get broken, right?
Anyway, a few years ago Pat was in the process of upgrading the system from Exchange 2007 to Exchange 2016 – skipping over the 2010 and 2013 releases, which was not uncommon. Obviously, there were quite a few differences between the two, both in terms of user interface and functionality.
The consultant the biz had brought in to help with the implementation noted that the admin users’ accounts all had mailboxes attached to them and wondered if this was entirely necessary. Given the way they worked – admins using their admin accounts only for admin stuff and not for regular user stuff – Pat agreed that it probably wasn't. But rather than just leap in, he thoughtfully contacted all the other admins to see what they thought.
Once they had all agreed that mailboxes were unnecessary on the admin accounts, Pat jumped into the Exchange Administration Console, which was entirely different to its equivalent in the 2007 version. He spotted that there was a "Disable" option but decided that wasn't really what he wanted to do with these mailboxes.
Then he found the "Delete" option and surmised that that was the appropriate function to remove the mailboxes from the admin accounts. As Pat wrote to Who, Me?: "Delete what? Well, I'm in the Exchange Admin Console, so it's probably mailboxes, right? FULL STEAM AHEAD!"
- Python script saw students booted off the mainframe for sending one insult too many
- Techie made a biblical boo-boo when trying to spread the word
- Bargain-hunting boss saw his bonus go up in a puff of self-inflicted smoke
- Developer tried to dress for success, but ended up attired for an expensive outage
He deleted what he believed to be the mailboxes and continued with his work.
It was about this time he started getting phone calls from the other admins. Why can’t I authenticate? Why is my login being rejected? Can you make sure my account hasn't been deactivated?
Realizing what had happened – all of the admin accounts except his own had been deleted – Pat immediately tried to reverse what he had done by retrieving the accounts from the Active Directory recycle bin.
Only to discover that he had not yet enabled the Active Directory recycle bin.
His own account had presumably been spared from delete-aggeddon only by the fact that he was logged into it at the time. Thankfully this allowed him sufficient privileges to rebuild the other administrators' accounts and privileges from scratch. Of course the other admins' accounts had all developed their little quirks and workarounds over the years, so Pat found that nothing really worked quite right again after that.
Until the org retired those servers and rebuilt the system from the ground up.
Pat learned never to delete anything without knowing a) exactly what was being deleted, and b) that it could be recovered if necessary.
Learning from experience can be hard, but it drives the lesson home. If you've got a learning experience you think other readers could benefit from – or at least have a laugh – click here to send an email to Who, Me? and we might share it on some future Monday. ®