BREITUNG TWP- Matt Tuchel is used to alarms and pages in the middle of the night. Besides overseeing Tower-Soudan’s water and wastewater systems, he is a member of the Tower Ambulance Service. But the alarm he got early Monday morning, July 27, was something totally new.
“The system called me at 12:23 a.m.,” Tuchel said. “It said there was a data failure at the Soudan lift station.”
When Tuchel got to the lift station, the controls were not working or showing any data. But he could see water in the wet well (which is normal) and found he could run the lift station pumps manually.
“I called our programmer, and he said, ‘oh no,’ not another one.”
It turns out the system was part of a cyberattack that hit at least 35 cities in the state over four days, according to Terah Rinerson, Minnesota Rural Water Association Wastewater Technical Advisor. Besides hitting smaller rural systems, some larger cities like Maple Plain, Plymouth, and South St. Paul also had some of their automated systems hacked, according to reporting by Minnesota Public Radio.
“It’s been quite the fiasco,” Rinerson said.
The attacks did not cause any permanent damage, she said, and only one of the impacted systems, in Braham, had any possible issues with water quality, requiring a short-term boil water advisory.
Tuchel said he had to run the system manually for about two hours, while the programmer had him change settings to get the system back online automatically. He then stayed on site another hour to make sure the automatic system was working properly. There were no impacts to water quality, he said.
“The programmer was locked out of our system,” said Tuchel, “but I wasn’t locked out and could change the systems and IP address settings manually.”
Rinerson said the cyberattack is being investigated by the FBI and Homeland Security.
“We are caught in a disconnect,” she said. “The programming being done for these water systems often have open ports that are vulnerable.”
The automatic systems, called integrators, use the internet to connect systems to their plant operators to inform them of potential problems. They are used more often in newer systems in smaller cities who cannot have operators working 24/7.
But these connections are often being done over unsecured cellular networks, she said.
As of Tuesday, the systems in Ely and Babbitt had not been hit yet, Rinerson said. But Virginia had a “small hiccup.” Other systems in the area have been put on high alert. Many smaller systems in our area do not have internet connectivity, so were not impacted.
“The attackers waited until a system opened up on an unsecured cellular connection and then it completely took it over,” she said.
“These integrators are needed,” Rinerson said. “But they need to be safer.”
Using fiber optic cables or old-fashioned phone lines would get around this problem, she said. But these options come with added costs.
Tuchel said their engineer looked at using a radio system when the new water plant was installed, but the $50,000 setup cost was a barrier.
“We weren’t hooked into the internet before the water filtration plant upgrade,” he said.
Tuchel said the integrator system has allowed him to troubleshoot issues and alarms remotely.
“It’s really nice for a smaller system,” he said. “But it also opens you up to these vulnerabilities.” The system was password protected, he said, so he wasn’t sure how the cyber attackers got past that level of security.
Tuchel said they are working with their programmer to increase security and looking at new options to make sure this wouldn’t happen again.
The Minnesota Bureau of Criminal Apprehension sent out a cyber activity alert on July 27, noting “ongoing malicious activity impacting public drinking water systems.”
The cyber threat actor has yet to be identified, it noted, but the BCA had identified the technical specifications that were vulnerable.
“Thus far, impacted facilities were able to mitigate further compromise, but a comprehensive understanding of the impact is still being assessed.” It noted that the system failures could cause a drop in water pressure in these systems, allowing unsafe backflow into drinking water systems.
Minnesota Rural Water Association has also sent out information to all water plant operators with a list of ten immediate actions that water plant operators should be taking.
“Utilities should remain vigilant and verify the security of all remote communications,” MRWA wrote, “especially systems relying on cellular connectivity. Even if no suspicious activity has been detected, this is an appropriate time to review cybersecurity settings, emergency response procedures, and incident response contacts.”