Alarm Fatigue in the Security World

Question: What happens when you get in your car, completely preoccupied, plug your phone in its charger, arrange your coffee cup, start the engine, back out of the driveway and reach the end of your street?

Answer: Ding…Ding…Ding…

Of course, the seatbelt dinger goes off.  You pause at the stop sign, reach around your shoulder, and while simultaneously changing the radio to your favorite station, you click the belt in its proper place and silence the alarm.

Why is that alarm so effective?  Usually it is in a quiet car and the alarm rivals the sound of some of the most irritating noises known to man.  The motion of buckling your seat belt is so ingrained in your brain that as soon as the alarm goes off, the required action is immediately obvious.  There is no thinking, there is no contemplating, there is just a quick sweep of the arm and the Ding goes away.

Recently, I was pointed me to this article from a recent NPR show (http://www.wbur.org/npr/265702152/silencing-many-hospital-alarms-leads-to-better-health-care) that discusses the impact of alarm fatigue in the healthcare world.  One quote leapt out at me and resonated:  ‘…technology has gotten out of control. "We have devices that beep when they are working normally, We have devices that beep when they're not working.’

Can alarm fatigue be a problem for the security world?  You bet.  Security IS a world of beeping flashing lights.   Security teams are faced every day with the “properly working beeping devices and the improperly working beeping devices” problem.   Security technologies are supposed to produce alerts and data.  If a security technology (firewall, IDS, SEIM, vulnerability scanner, etc.)  goes dark, i.e. isn’t producing alerts, it is probably a bad thing.  Someone tripped over a power cable and unplugged the thing because we usually think if a security product isn’t spitting out loads of data then it isn’t earning its keep. 

Security teams have all of these beeps flowing at them – which are the most important?  which are just noise?  which need to be escalated?  Eventually, like the hospital workers in the report, the desensitization level against these alarms from our technologies rises and security teams start missing the most impactful alarms.  Yet technology isn’t the only factor in the Alarm fatigue facing security.  A CISO and his/her team can face ‘alarms’ from a variety of sources.  Does this log look familiar?

8:02 AM – Malware infection on 10.1.2.30

8:30 AM – Voice mail from colleague re: new hacker group

9:00 AM – Meeting with QSA re: last week’s vulnerability scan

10:30 AM – Sun just released a new patch to JRE 5.4.3.2

11:15 AM – Vulnerability scan on DMZ completed

11:30 AM – Meeting with XYZ department on new application being installed next week

12:00 PM – Company just like us announced major breach

12:02 PM – CVE-2014-123 just released

1:45 PM – Meeting with audit committee re: security risks

2:00 PM – System outage at a local branch

2:15 PM – Weird(?) network traffic reported by network team

2:30 PM – Security policy meeting on revising standards due to new regulation

2:53 PM – Malware outbreak on multiple machines

3:00 PM – New contractor onboarding

3:20 PM – Present Security awareness training to new employees

4:15 PM – Industry ISAC security conference call

4:32 PM – HR reports social engineering attempt

5:07 PM – Port scan on 192.168.3.45

5:30 PM – Multiple failed login attempts on 192.168.100.23

6:15 PM – Vulnerability scan found 142 critical vulnerabilities

Which of these are the most important?  What resources should be tasked with digging into these alarms?  Which represents the most risk to the organization – the malware on the unknown IP address, the regulatory compliance results, the unexplained network traffic, the social engineering attempt?

According to the healthcare report:  ‘Boston Medical Center is attracting national attention as a hospital that apparently has conquered alarm fatigue. Its analysis showed the vast majority of so-called "warning" alarms, indicating potential problems with such things as low heart rate, don't need an audible signal. The hospital decided it was safe to switch them off.’    So is the answer to Security Alarm fatigue turning off certain alarms?  Perhaps so but I think the answer lies in adding more dimensions to these alarms than we traditionally have.   We need more depth on alarms and we need the ability to quickly put together the picture around the alarm to understand the risk to the business.

How is alarm fatigue affecting your organization?  Are you taking steps to put in the right filters and dimensions to clarify the alarms?  The goal is not to eliminate the alarms but rather get the motion down to where the thinking and contemplating is minimal.  Like the seatbelt dinger, if the resolving action is understood and clear, the alarm may be addressed with a simple sweep of the arm.