Many potential ransomware attack victims remain in the dark about ransomware, even though attacks are on the increase. If you look at news articles of recent attacks they often focus on the attack, the threat actors, and the type of ransomware. So, it’s no wonder they are in the dark about how an attack could affect them. That’s to say, important facts are left out such as the discussions about long recovery delays, that can blow out to 287 days on average according to the Ransomware Task Force. Secondly, that the safety net of backups often fail.
Meanwhile, there is not enough talk about how ransomware affects the people. These attacks hit the business, and the people, not just the machines.
Ski Kacaroski a systems administrator gives us greater insight into understanding ransomware, as Ski in 2019 managed to pull his school district out of a ransomware nightmare. For those who don’t know much about ransomware Kacaroski talks at length about his experience where attackers had encrypted crucial data, locked up vital systems, and even threatened employee pay.
So, we’ve included some of his insight and surprising lessons
The first few hours are critical
On the 2oth of September 2019 at Northshore School District, which is north of Seattle in Washington State, cybercriminals launched a ransomware attack late in the night called Ryuk. The school district relied on a datacentre of 300 Windows and Linux black box servers. They also manage devices for 4,000 + staff members that include Windows, Mac, and Chromebook workstations, and iPad tablets.
The day after the attack, the school district’s database administrators called Kacaroski about the issue with the database server. So Kacaroski logged into the system using his employer VPN and started looking around. That’s when he discovered the server had been hit with the Ryuk ransomware. He could only see one unencrypted file which was a note from the threat actors. As everything else had been encrypted with Ryuk.
During the first few hours is where Kacaroski admits he made a big mistake.
Firstly, he explains that given the chance to do it again, “the minute he saw the first one [hit], he would’ve just pulled the power on every single box, ASAP.” In other words, it cost a few boxes by not doing it quick enough. When someone reports an issue with the system your first thought isn’t that you’ve been hit with ransomware. So, it catches you off guard.
Secondly, he learnt later from the school district’s cyber insurance provider that ransomware operators often target only Windows machines in these attacks. So, he and his staff could have concentrated on the Windows machines.
Your backups may not work
Certainly, as the initial fallout of the attack unfolded Kacaroski said he and his colleague were dealing with “an incredible amount of uncertainty.” In other words, they were still in the dark about what critical services had taken a hit and testing which drives were operational by pinging them. As well as still under the assumption all the devices were under threat, not just the Windows machines.
Meanwhile Kacaroski was quietly confident, well at least at the initial stages as the school district had their proper backups in place. But he soon realised this wasn’t the case.
Kacaroski said “we have a very good backup system, or at least what we thought was an extremely solid, rock-solid backup system”. “And then we find out, at about 4 or 5 hours after the attack, that our backup system is completely gone.”
And this unfortunately happens quite frequently. For example, JBS a meat supplier still paid a whopping $11 million in ransomware to attackers. To clarify why they decided to pay the amount for the decryption key while they had a backup is because their backup didn’t get vigorous and regular testing.
The reality that the backups weren’t going to work, hit Kacaroski and his colleague pretty hard.
“The part that really, really hurt us, he said was the fact they had to rebuild 180 Windows servers…and rebuild Active Directory from scratch, with all those accounts and groups, and everything in it.”
A ransomware attack can be a months-long process
Most importantly, the attack against Northshore School District wasn’t construed by one hacker group but had a progressive turn of events according to both the FBI and the Department of Homeland Security.
Staff were told that the initial breach would have happened about six months before the final ransomware infection.
Firstly, a group of hackers wanting to gain access into Northshore servers installed Emotet. When successful, that group sold the access to another group.
After that, the second group of hackers who wanted domain credentials, installed Trickbot. Once successful that group sold the information to the third group of hackers.
Lastly, the third group are believed to have pushed the Ryuk ransomware to the school district machines. In other words, they did they final blow.
Meanwhile, none of the groups had worked together they all worked independently, leveraging off the last groups work.
Kacaroski was shocked to learn that these gangs operate like a franchise when it comes to Ryuk ransomware. “Like McDonald’s…one that does the East Coast…one that runs the West Coast…and one in between.”
There are more ransomware attacks than you’ve heard about—far more
Meanwhile, Cyber insurance providers said they’d paid for another four ransomware victims within the week since the Northshore School District was hit. In other words, that was five victims in just one week of 2019. And imagine the yearly numbers if 80% are the undisclosed victims. So, we can see the number has skyrocketed with the combination of both reported and unreported cases of a ransomware attack.
In immediate recovery, first prioritize and then look for “surprise” systems
In the initial stages of responding to an attack the organisation needs to discern what needs their attention first. What systems need to come back online first? In some cases, this is easy to discover as often a ransomware attack comes before a crucial deadline.
For instance, the ransomware attack of the Northshore came a few days before the staff pays were due. And staff need to be paid, it’s a deadline that can’t be missed, says Kacaroski.
So, Northshore decided to get the payroll up and running in a matter of days. “That was the most critical thing, he said.”
Secondly, to help ensure the school continued to run as these are used every day, they decided to get the Active Directory and the student record system back online. Kacaroski states that everyone from teachers, students and parents used the student record system so it needed to go backup online asap.
Lastly, are the “surprise” systems which are in place but may not know or understand a lot about them. That is until they’re not there anymore. Northshore soon discovered their “surprise” system was the school’s cafeteria and payments record system.
Kacaroski said we had no clue that the cafeteria provided 10,000 meals a day and 30,000 in takings in one day. And they had no idea if students owed money for the meals or not. So that one took a long to get back up and running and it didn’t have any backups.
Avoid chokepoints during a long, collaborative recovery
In responding to the Northshore School District attack there were only two sysadmins. In other words, there was not much that two people could do in such a short period of time. Kacaroski and his colleague still had to sleep, eat and take time to rest. It was impossible for them to work 24/7.
As a result, of the system admins knowing the systems so well, they become the go to point of contact for rebuilding the entire business, piece by piece. And the system admins can get bogged down with too many teams coming to them for information, sign-off and verifications, and feel overburdened.
Kacaroski advises that organisations are best to free up their system admins for the moment to help move the recovery process forward. They can do this by adding more sysadmins until all the systems are back up and running.
For Northshore School District, both methods were used as straight after the attack the school called up a local hosting firm that had done good work on small jobs in the past. So, three additional system admins were sent immediately to help share the load and clean up the problem.
Instantly, the team went from two to five fulltime experiences sysadmins.
Because there were now more sysadmin staff the team had more breathing space. Secondly, for the time being the school found a paper workaround for the cafeteria records. The children were then offered three different lunch options to keep recording on paper manageable. That bought the system admin team more time to rebuild.
Lastly, the school district decided to move its student record system, to a SaaS solution, rather than the 27 Windows servers. It was great to see a vendor drop everything and migrate them in a record 6 days, rather than in normal time frame of 6 months, thanks to a good relationship.
Kacaroski stressed the importance of having strong relationships with local vendors, other school districts, parents, and other teams inside the school district itself. This enabled Northshore to recover about 80 – 85 percent of its systems and files.
“Relationships were the most critical thing, says Kacaroski”
