January 25, 2021 | Alex Terlecky

Roxborough Water and Sanitation emerged a champion of cybersecurity.

At 4 a.m. on a Friday morning, Barbara Biggs, General Manager of Roxborough Water and Sanitation District, received word from the Director of Operations that they were in the midst of an ongoing cyberattack.

This attack had started a few hours earlier when administrators overseeing the district network received an alarm alerting them to invasive action. As the day unfolded and the scope of the attack was fully realized, it became apparent that this was no simple hack—this was a ransomware attack.

Their servers were encrypted and unusable, and the only way to get back online, it seemed, was to give in to the demands of cybercriminals.

First Steps

Initially, Barbara described her reaction to hearing the news as one of panic. Understandably, when the workday begins ahead of schedule with the discovery that water and wastewater treatment facilities, billing systems, and servers are all locked and encrypted, it is never a good thing.

Although Barbara did not know who targeted the district—and still does not—she knew she was dealing with a ransomware attack from what she found on affected computers.

The urgency was apparent in the cybercriminals’ demands, as well as in the district’s need to make sure they could still operate as normal.

As soon as they had discovered they had become the victims of an attack, the district’s operations team worked to swiftly and safely bring the water treatment plant and wastewater conveyance systems back online.

“Our number one priority was making sure we could provide safe water and efficient wastewater treatment operations for our members,” Barbara explained.

Once at the plant, the team discovered that all the tools they typically used to automate processes were affected by ransomware. As a result, they had to revert to their procedures outlining how to operate manually.

“We lost our automatic eyes to see the plant’s operations, so we kept our physical eyes on the plant,” Barbara said.

Barbara explained that she and her team were focused on ensuring that services went uninterrupted.

“We had to perform manual calculation[s] for chemical doses. We had to physically drive around service areas to check tank levels visually,” she said. “We lost alarms, so we performed 24 hour drive-bys to make sure there were no problems. It took two weeks to get all alarms back online.”

After the Dust Settled

Once Barbara and her team were able to confirm that they could still function effectively and provide uninterrupted service to customers, the staff turned their attention to dealing with the ransomware.

“My first reaction was to reach out to the [CSD] Pool and authorities,” she said.

By 10 a.m. that same Friday, Barbara was on the phone with Vicki Sullivan, the CSD Pool’s Member Relations Coordinator, as well as Sedgwick’s claims administration, and Norton Rose Fulbright US LLP, a law firm provided to the district by the CSD Pool’s free member resource, eRisk Hub, created by cybersecurity and industry experts at NetDiligence.

Through the resources in NetDiligence’s eRisk Hub, Barbara was put in contact with a team of cyber coaches, which included a forensics investigation team, ransom negotiation experts, and a data recovery team.

“[The team provided by NetDiligence’s eRisk Hub] are literally the most responsive people I’ve worked with in my life. Their whole business is negotiating with these threat actors.”

Barbara was in communication with individuals from the CSD Pool and her contacts at Norton Rose Fulbright almost two times a day all the way through the following weekend.

Shortly after contacting the CSD Pool, Barbara reported the cyberattack to the local Sheriff’s office, who sent it up the chain to the region’s Department of Homeland Security (DHS).

By 3:00 pm that same afternoon, Barbara received her first response back from DHS. The incident was also reported to the Federal Bureau of Investigation.

What Happened Next

As they waited for their network to come back online, Barbara and her staff worked hard to determine how to move forward with a compromised system.

They came together as a team, outlined next steps, and, as they learned more about what had happened in this particular instance, developed strategies for preventing future cyberattacks, all while maintaining both support and communication with their customers.

“We all pulled together and got through it; we tried to keep information flowing to the customers,” she said.

The cyber forensics investigation team determined there was no evidence that data had been stolen, just encrypted by cyberattackers, whose identities remains unknown.

In addition to the damage wreaked on their operational infrastructure, Roxborough Water and Sanitation also discovered extensive harm done to their billing system.

“All we could do in the early weeks is put [the news of this event] on our website, Facebook page, and social media,” Barbara said. “Customers have been incredibly patient with us.”

They had to rebuild their billing system which contained thousands of accounts, working with a software company that was able to take the last billing reports and reverse engineer them to create a brand new billing system.

Moving Forward

As the team at Roxborough Water and Sanitation, along with countless other government entities and utility providers have learned, cybercriminals are ramping up their attacks on critical infrastructure.

As a result, public entities are under increased pressure to prepare for and meet this threat.

“I think they target critical infrastructure because they know we have to operate. They know we have to absolutely get back online,” Barbara said. “I know they have been focusing on smaller water and wastewater utilities because we don’t have big IT departments.”

In many ways, Roxborough Water and Sanitation was maintaining adequate security procedures prior to the situation.

Firewalls were kept up to date, virus protection was available on servers and emails, and some staff had been religious in shutting down and taking their devices offline at the end of day.

Other updates that have been implemented at the district include the addition of servers to ensure redundancy of the district’s systems.

Best practices have been expanded to include a robust password policy, full and current inventory of their environment, and tighter security surrounding online bill pay. They also regularly perform hard drive and cloud backups.

“You have to have multiple backups because you don’t know how long [the cybercriminals] have been in your system,” Barbara said.

The Aftermath

Roxborough Water and Sanitation is back up and running—now stronger and more secure than ever.

But this ordeal still has residual effects on the district. Through this experience, some internal vulnerabilities came to light at the district.

This has led them to create new plans, procedures, and practices to better manage their systems going forward.

Above all, Barbara and her team at Roxborough Water and Sanitation understand firsthand how communication, education, and preparedness can make the difference when dealing with cybersecurity.

“You may have to make sure people understand how critical it is.”