As I have written here several times before, like in a post on accepting mistakes to reduce their frequency, I am a big fan of the phrase “fail faster”.

Many years ago when I was Director of IT and Security at a very large enterprise I was fond of saying “fail faster” to my staff. I wanted them to feel comfortable with the idea that they should focus on always improving. The CIO was not fond of this and constantly asked me why a Director of Security, of all people, would encourage failure?

I could give a hundred examples (sports, martial arts, arts, etc.) where a perfect score is not only unlikely but self-defeating. This was familiar to some, but others still tried to prove to me that “only first place matters” and failures always should be downplayed or obscured. My fear was that their behavior was a slippery slope to fraud. Their concern was that my behavior was demotivating.

Today a colleague read a post called “Fail a Security Audit Already — it’s Good for You” and asked me if this means QSAs are too soft on their clients. The author gives this analysis:

If the audit is a stress-test of your environment that helps you find the weaknesses before a real attack, you should be failing audit every now and then. After all, if you’re not failing any audits there are two possible explanations:

1) You have perfect security.

2) You’re not trying hard enough.

I disagree and will try to explain why it’s different in this case. The author clearly is not speaking from the auditor perspective. You don’t want to tell companies to fail a PCI DSS audit. It’s a subtractive system. A company does not get a pass until they have removed all areas of remediation or compensation and can prove that things are running smoothly on an ongoing basis. The following paragraph has a strange depiction of the audit process.

Companies should be failing audits, whether internal or external, far more often than they suffer breaches. The fact that few companies are failing any audits should be cause for concern, not celebration.

How exactly has the author concluded the “fact” that few companies are failing audits? As a long-time auditor I find companies trying to pass audits far more often than they are being breached. I would call this reviewing test results and remediation in order to pass an audit.

And what celebration is the author talking about? When an auditor leaves a passing score there typically is a sigh of relief, not celebration. I am tempted to suggest this to a restaurant. Next time the health inspector gives them a passing score I will ask them to serve free cake and champagne. Probably won’t fly. I suspect there is no evidence of celebration.

The motto of fail faster works for rapid development for improvement but “trying” to fail an audit or an exam is bad advice to give a company. It’s like saying your tachometer isn’t trying hard enough if it doesn’t fail every once and a while to tell you the correct RPM. Or that you aren’t a good driver if you aren’t trying to fail your license test. Imagine if auditors tried to fail their certification test to prove that they were really trying hard to understand the regulations.

The decision of when to try and fail is nuanced. It can be confusing, which goes back to the reason the CIO cautioned me about motivation and interpretation. There are some things you want to fail and measure frequently (e.g. practice runs, tests) and things you don’t want to fail (e.g. final exams). The CSO article does not make this important distinction, and does not mention that you should consider the consequences of failure, when it tells you to fail. When we limit our definition of an audit to something like a formal audit (the final Report on Compliance to the Payment Card Industry Security Standards Council) then it is not good advice to try and fail. You should try to pass, by failing faster.