What is Alert Deafness?

Ping! One of the CI pipelines is failing.

Ding! Critical production error incoming…

The exponential increase in data processed by organizations means a rise in errors, failures, and vulnerabilities is expected. But with pings and dings popping up over 500 times per day (according to the International Data Corporation), developers are left with a completely unmanageable environment.

With so many notifications vying for developers’ attention, it’s no surprise that we become desensitized to alarms. Roughly 30% of alerts go unnoticed due to this deafness – but it’s not ignorance; it’s exhaustion. On-call developers and “always-on” services suffer the hardest, and error handling becomes a pain rather than a priority across the board for security, IT, and DevOps teams.

In this post, we’ll review why and how alert deafness has become a menace to application development and what you can do about it.

The End of ‘Unavailable’

Has the remote working age signaled the end of out-of-hours? 60% of companies admit to using employee monitoring software, which creates a fear that setting your status to “unavailable” or “busy” will rouse suspicion. Organizations (especially “always-on” services) expect constant vigilance from development teams.

The pressure trickles down from the top. Check Point found that global cyberattacks increased by 38% in 2022, driven by agile hackers that exploited work-from-home environments. Companies in all industries are feeling the heat, and their remediation efforts are sometimes misguided as they expect developers to work more and react faster.

Creating a culture of 24-hour urgency through alert overload isn’t just unattainable; it’s also unhelpful for developers’ productivity. A 2021 article in the Journal of Alzheimer’s Disease found that a decline in cognitive function was linked to long working hours, which throws water on the idea that developers should react to every alert instantly. With a whole host of other tasks to attend to, developers can easily become too overwhelmed and burnt out to even pay attention to the notifications.

When Everything is Urgent, Nothing is

The impact of developer burnout can be felt far beyond employee satisfaction surveys. And who can blame us? If we gave in to every single ping, development velocity and efficiency would take a huge hit. This challenge is further amplified for teams using multiple security tools. In the 2023 Cloud Security Report, 17% of respondents using 1-3 security tools felt overwhelmed by alerts, and this percentage increases to 40% for respondents using 4-6 security tools.

The goal is to respond to threats, not just to see them. If you can set up processes to identify a manageable selection of actionable alerts amid the high volume of suggestions, you’ll be successful in avoiding the dreaded “everything is urgent” paradox. Organizations inadvertently push their employees away by returning to this archaic boy-who-cried-wolf belief.

Recent reports by global recruitment firm Hays found that 95% of employers are experiencing skills gaps and a shortage of tech workers, meaning they cannot mitigate risks due to staff shortages. Recruiting is just as time-consuming and expensive as security teams investigating notifications that are not relevant, so it’s in everyone’s best interest to sort out this alert problem.

The book Site Reliability Engineering: How Google Runs Production Systems suggests a hierarchical alarm system is key. An “alert” should be categorized as an event that needs urgent human attention. One step down from an alert is a “ticket,” an incident that isn’t urgent and can be remediated by a person in the near future. A “log” is the lowest level and should be used for diagnostics and incident tracking.

Evolving Complexity Requires Evolving Resilience

Attacks like social engineering, ransomware, and brute force are often financially lucrative, so it’s unlikely that we will see a sharp decline in malicious activity anytime soon. The pressure to stay on top of everything increases as the security industry and threats evolve. Similarly, the evolving complexity of your operations and applications means your security prioritization will change. The alerts that scared your small team to death can now become tickets or logs in light of your advanced operations.

Let’s face it, you’ll probably get more failures and vulnerabilities pecking away at systems this year, and streamlining your security defenses and alert posture will make your daily operations a lot easier. Security teams often agree that it’s impossible to fix what you can’t measure, which is why data is invaluable for controlling alerts.

You can use data from a centralized alert system to identify patterns, aggregate related alerts, and keep tabs on how long security teams spend on remediation. Security analysts can use this data to understand the context behind alerts and create defined threat response strategies that segment alerts based on priority. For example, critical alerts should be based on metrics that negatively impact customers, employees, and end-users – not minor issues that can be auto-remediated.

Don’t Silence the Alerts – Refine Them

There are multiple ways to wrestle back control over the tsunami of alerts flooding your inbox every day. Here are a few effective methods.

Prioritize Prioritization

What does “critical” mean to you? It’s not a trick question – it’s one that many busy organizations and security teams neglect to consider. Breaking the vicious cycle of alert deafness means distinguishing between alerts and focusing on strategic remediation. Realistically speaking, alerts should only occur if immediate action needs to be taken by a member of your security team. Anything outside this parameter can be turned into a ticket, automated, or even (gasp) deleted. You can go back to basics and define what you constitute as an error, who will handle it, and what effect it will have on the supply chain.

Filter the Notifications

We could all benefit from better organization in our lives, and the same is true for alert control. Every alert should be automatically validated and enriched with the right context to reduce the time spent (and wasted) investigating it. The tips to make this a reality are surprisingly simple:

• Use different channels for different projects.
• Color-code projects, alerts, and notifications.
• Use email filters to aggregate and filter alerts.
• Use tags to indicate different alerts.
• Ensure you know every alert’s priority level, timeframe, and remediation strategy.

Who Owns What?

When everyone on the team is busy, it’s easy to think or hope that someone else is actively checking or reacting to recent alerts. If everyone shares this mindset, notifications can go unattended. A shared responsibility model, where responsibility is divided between security, IT, and DevOps teams, is often ineffective and difficult to maintain. Imbalances between the scope of threats and the size and expertise of the team can contribute to developer burnout and employee turnover, meaning you’ll need to distribute responsibility between developers and clearly define ownership over alerts. For example, who is checking, and how often should they do so?

Invest in Training and Awareness

Deciding whether or not to investigate a security alert is a business-critical decision that should be well thought out. Security teams are the ones feeling the heat, so it can be challenging to get buy-in and recognition from stakeholders who might not see the direct negative impact of alert deafness. Here are three key pillars that should guide your stakeholder awareness strategy:

1. Define processes: Your organization should have clear strategies in place to mitigate risks. Security teams can test their capabilities and understanding with exercises like red/blue/purple team testing.
2. Implement standardization: Team training can help you take a consolidated approach to alert best practices across your organization. Creating a runbook and treating it as an ultimate alert guide is helpful. A runbook gives new and existing team members clear direction on how and when to action alerts, which helps reduce dependencies from architectural and organizational standpoints.
3. Assess periodically: Conducting regular assessments, analysis, and periodic KPI reviews of the alerts your team has received over X amount of time is a good way to raise awareness of alert fatigue amongst senior management.

Invest in the Right Security Tools

People are central to cybersecurity, but sometimes organizations must rethink how security teams spend their days. Unnecessary alerts and false positives only serve to waste time and money, which is where security tools can provide a helping hand.

Investing in security tools is a catch-22. It’s probably time to replace siloed systems, but adding too many doesn’t necessarily fix alert deafness. Instead, utilize the power of automation, AI, and ML and consolidate all tools into fewer platforms to avoid alert duplication and improve prioritization. By asking security vendors how they prioritize risks, you can better understand how the tools will perform in your workflows.

With that in mind, tools must remain reliable and trustworthy – those that generate high false positives only add to the problem, so you should choose highly accurate options that provide contextual analysis and actionable intelligence. Your organization will still need developers, and their human intervention is best used for unique and critical alerts. Anything else can be automated, which means alerts should only occur on a need-to-know basis.

Set Security Teams Free From Alert Deafness

While you can never eradicate security alerts, you can certainly reduce them to a significant degree. Security teams will thank you, and you will see the benefits across the board, including improved productivity, allocation of resources, and development velocity.

From code to cloud, Check Point CloudGuard’s CNAPP unifies cloud security, merging deeper security insights to prioritize risks and prevent critical attacks – providing more context, actionable security and smarter prevention. With Check Point CloudGuard’s Effective Risk Management engine, you can prioritize risks and receive actionable remediation guidance, enabling you to focus on the most crucial 1% of risks.

If you would like to see CloudGuard in action, please fill in the form to schedule a demo, and a cloud security expert will help to understand your needs.

If you have any other questions, please contact your local Check Point account representative or channel partner using the contact us link.