If you’ve been a long-time reader of this blog you may recall seeing here before that in the early-2000s the US government left security of critical infrastructure up to the market investors in infrastructure (mainly banks) to figure out.
It was like a “trickle-down” theory of investment bankers showering the littlest critical infrastructure projects with the kind of money they would need to make things safe — at a market-designated level.
I have done critical infrastructure security audits, as well as security strategy consulting, before and after this time. What one might imagine on the outside is very different than what I found on the inside. That is to say, I expect most people (even myself before I started going inside) expect management to be laser focused on safety of service delivery, and willing to invest even a little extra to protect people from harm (capacity and disaster planning).
Yet that hasn’t been my experience.
For example on one engagement I had a bank ask if they should put their investments towards building adjacent bitcoin mining operations in power stations to shove “excess” power into assets they would sell off to an unregulated market.
On another engagement, as I was on my way to hack into the generation and distribution networks (they were weak), management stopped me and said “wait a minute, we care not much if those go down and people are without service, as that’s routine for us; instead please focus attacks on our trading systems and financial operations around billing and pricing” (they were weak too).
To be fair they were saying they could handle dangerous life-threatening accidents because that’s what they have been planning for all along… yet when I probed deeper it was more like they knew that those accidents wouldn’t have an effect on their P&L. Really.
And these were giant even “bulk” organizations, not “small systems” that have less of a fighting chance to argue with banks that may make final decisions on risk management models:
There are over 145,000 active public water systems in the United States (including territories). Of these, 97% are considered small systems under the Safe Drinking Water Act, meaning they serve 10,000 or fewer people.
Alas, from an economics standpoint it’s easy to say “poor” American banks do not have the money to spend on public utilities. Yet a wider macro view is probably that American investors with loads of cash to invest made it a conscious market decision since at least 1998 (when I pwned 1,000s of infrastructure routers across five states using clear-text passwords) to not invest in service safety. They’re not cash strapped as much as they’re not regulated in a way that a whole history of relevant accidents and basic common sense would force a cash infusion into the areas we might expect.
Also sometimes I wonder things like why Microsoft’s billionaires even charged utilities to license software for water utilities in the first place… or why the utilities didn’t all shift to software that came without a license, avoiding built-in end-of-life (EOL) and support models wildly inconsistent with their operation plans.
Anyway, here’s the TL;DR on the most recent “news” in America that uses the headline of “cash strapped” Americans (who have been violating basically every basic principle of safe operations even as laid out by the US government for years):
- All computers used by plant personnel had remote control
- All computers connected to plant’s control system
- All computers connected directly to Internet
- Out of date OS (Win7 – EOL Jan 2020)
- All users share the same password
- No network protection (firewall)
Shocking. It doesn’t take much money to fix all of that, especially if you had done it a year ago.
And here’s a post I wrote about many of the prior warnings: Was Stuxnet the First?
And here’s a post I wrote (in 2011!) about this exact issue: Chicken LittleStux is Falling
Let me now suggest a different narrative. “Cash strapped” is a military negotiation and planning phrase despite having an enormous amount of money in its budget.
Cash-strapped US military to cut Persian Gulf fleet: USS Harry S Truman will not return to Middle East, leaving only one American carrier group near the strategic Strait of Hormuz
And now for something completely different, look at hard lessons of 1991 when a missile downed an AC-130 gunship and how the US military responded.
America decided not one more AC-130 would be lost to attack. And 30 years later it’s still true. Was it cash infusion? No.
All 14 airmen aboard were killed, but one Air Force general wrote that their sacrifice helped usher in a new era of the AC-130, one where new technology and tactics helped ensure that no gunship has been lost in combat since.
“We owe much to those who sacrificed everything aboard Spirit 03, not only because ‘they gave the last full measure of devotion’ for us, but also because they bequeathed to us, at a critical point in history, the decisive motivation to reinvent the AC-130 for a new challenge and a new century,” wrote now-retired Maj. Gen. Mark Hicks, a career gunship pilot, in the summer 2014 issue of Air Commando Journal.
The lesson from the US military success with the AC-130, however, was not an expensive reinvention of technology and newly dedicated staff as much as what Deming called the statistical control process to improve existing practices — commitment to delivering quality and identifying exposure or risks earlier.
For what it’s worth, in 1980s when “cash strapped” Ford hired Deming he improved safety, quality and changed management practices in those areas. They called it Total Quality Management and focus on lack of cash; he turned risk around so much they soon outperformed GM and became the most profitable car company.
Had Ford stuck with Total Quality Management, it might have avoided many of the problems that have plagued it recently. Instead, as the years rolled by, the concept faded into the background at Ford as its champions retired and were replaced by executives who had other priorities. “U.S. automakers had so much confidence, they felt they had achieved quality and didn’t need to focus on it anymore”…
Perhaps read that insight as Ford was no longer was “cash strapped” so their focus deteriorated and safety declined.
Cash infusions could have actually led to the wrong outcome. Again, it was focus on the wrong things that led to the AC-130 being shot down, and like Deming’s work at Ford maintaining focus on quality is what made a huge difference in safety. Spend as little as possible and no less.
Here’s the money quote from the story of how an AC-130 program now has run three decades without any attacker forcing one down.
…improved fire control and better sensors really helped, but it was a commitment to be tactically sound that really made the difference,” Hicks wrote. Walter expressed a similar view. “The fundamental lesson learned is to always expect to be fired upon when firing.”
They don’t say the fundamental lesson is a cash infusion (in fact they brush that away as “really helped, but”). They certainly spent some money and also had some accidents — but it was focus on quality that mattered most.
Although losing a brand new, low density-high demand asset like an AC-130J is bad news, this is what testing is for. Better have a permanently grounded plane than one laying on the ground burning in the enemy’s backyard.
And I wonder if we should apply the same lessons domestically. Stop making safety in critical infrastructure about cash moving hands and instead make it about being tactically sound. I don’t mean NERC’s Critical Infrastructure Protection (CIP) either as some of you may remember it was a very cynical game by utilities to avoid NIST 800-53 and pretend they needed their own set of rules so they could ignore them.
We’ve known what happened in a water system in 2021 is what we talked about in 2000 after a water system was compromised, as I said above in my links to blog posts from a decade ago. There have been many, many studies in between then and now.
However, unlike the US military resolve to care deeply about stop loss, the market-driven critical infrastructure seems to have long taken the opposite approach and push the question how many more catastrophes are allowed before they really, really have to care.
I say don’t make it about cash, because it’s always been that way. Take a look at America’s healthcare system for reference. Anyone who says government run health care would be more inefficient is willfully ignoring that the United States pays more per capita on health costs than any advanced country, yet is the only one without universal health care. Cutting out health insurance companies whose sole goal is to manage “cash strapped” issues by pushing huge amounts of money around using a market-based solution could save billions and still improve safety.
In fact, you might say the inflationary cost of security has made safety even less likely to happen because it gives bankers and easy out by claiming the risks are worth not spending on controls. So the less cash-strapped the less secure… could be a logical outcome.
Make it about quality, about tactical soundness, not about opening coffers or another form of congressional-military-industrial-complexity.
See also 2020: “What We’ve Learned from the December 1st Attack on an Israeli Water Reservoir?”
The reservoir’s HMI system was connected directly to the internet, without any security appliance defending it or limiting access to it. Furthermore, at the time of the publication, the system did not use any authentication method upon access. This gave the attackers easy access to the system and the ability to modify any value in the system, allowing them, for example, to tamper with the water pressure, change the temperature and more. All the adversaries needed was a connection to the world-wide-web, and a web browser.