We’ve all heard about ransomware – the bad guys lock up your files using strong encryption and then force you to pay thousands of dollars to get them back. That sounds scary in theory. But what does a real ransomware attack look like?
Our CEO sat down with an old childhood friend whose business was recently hit by ransomware. We’ll protect his and his company’s name, but let’s call this friend Robert. Everything else about this interview is true.
Robert works for a software company that employs approximately 100 people, and serves as a manager for the software team. He doesn’t explicitly work in security, but regularly interacts with his company’s security team. Robert’s story should be best-case – a knowledgable staff with the resources to prevent such attacks.
Unfortunately, as you’ll see, ransomware causes unbelievable damage to even the best-prepared companies.
Do you have anti-virus software running in your company?
Yes, we run [a major AV program] on all of the workstations. We are also running the same company’s products on our servers. We keep that software up to date, but it was unable to prevent our systems from being infected with ransomware.
Lots of people point to backups as a solution to ransomware. Do you have backups?
We are supposed to have warm and cold backups of all of our critical infrastructure. In the case of our database server, the warm backup was also corrupted and the cold backups were not there. To this day, we don’t know for sure whether they were deleted as a part of the attack or if they weren’t being created and simply no one noticed.
People think that backups are a simple solution. They aren’t. Actually making sure that all important data gets backed up is hard. I’m still surprised how much data in devices like laptops just doesn’t make it into the backup schedule. Even when they do, restoring full systems can take days.
What was the impact on your company?
We were offline for 3 days, which means customers had no access to the site. It crippled not only our production infrastructure but also our development servers and even our individual workstations, all of which had to be rebuilt. It probably set development back on major projects at least 8 weeks.
We still aren’t 100% back (three months later), although part of that is due to the (necessary) stricter network security that was put in place after this that development now has to work around.
How long did it take to recover, and approximately what did it cost?
Well, as previously mentioned we are still dealing with the aftermath, although at this point it is mostly minor things. It took us about 60 hours to bring the production site backup. That involved our IT staff (4 people) working pretty much around the clock. After that, I’d say it was a good 3-4 weeks of rebuilding critical infrastructure to get us at least 90% of the way back. We’re still working on the last 10%.
Between lost sales, the new infrastructure we needed to buy, and person time, I’d say this easily cost us in the $500k range. If you add in the lost development time and the delay in delivering major projects, you could probably argue it’s closer to $750k or more.
Did your IT staff make any changes afterwards? How are you protecting yourselves against the next attack?
Yes, the network and all production infrastructure was rebuilt from scratch. The production network is now much more isolated than it was before. We changed our patching policies so that critical patches get applied pretty much immediately. We are also engaging with a firm to do a full security audit of our IT infrastructure and will make additional changes based on the outcome of that.
Do you have any advice for others who may have this problem?
I think the thing that we realized very quickly after the initial attack ended was that we were in a true disaster recovery (DR) situation. We thought we had a good DR plan in place but when faced with executing it, it was clear we were completely unprepared. We had antivirus software from a top firm and we were infected anyway. Our backups failed to protect all of our data. Even without paying the ransom, the real costs and lost revenue of this attack were huge.
My advice for another firm is to take DR seriously, to make it a priority, and to actually practice executing your DR plan on a regular basis to make sure that it will actually work.