In my previous blog, I wrote about security awareness programs and provided some high-level recommendations for how you can improve their effectiveness. In this article, I’d like to share some thoughts on how to test and measure how well those programs are doing. How do you know if you are making an impact? What can you actually measure?
In answering these questions, you need to first establish your goals and metrics and determine if there is a baseline. For instance, let’s start with a key component that should be covered in every security awareness program – anti-phishing. According to Verizon’s 2018 Data Breach Investigations Report (DBIR), on average, “78 percent of people don’t click a single phish all year.” (That sounds like a passing grade.) So calculating the number of people who actually click on phishing emails could be a starting point for your organization’s anti-phishing campaign metrics, assuming you conduct click tests.
The 2018 DBIR also states that, “on average 4 percent of people in any given phishing campaign will click.” The threat actor only needs one open door, which is why phishing is still one of the most concerning threat vectors year after year. Let’s say you can show that 85 percent of the people in your organization don’t click – does that mean the training is effective? It’s a start, but there’s still much more to consider, such as how to address the other 15 percent.
Interestingly, the 2018 DBIR also looks at the behavior patterns of individuals who do click, and it turns out that they are likely to be repeat offenders. Therefore, another metric to assess is how your program changes the behaviors of known offenders over time. This may involve additional targeted training and testing for those high-risk end users.
Now, let’s look at another aspect of your training content. Are you training end users to report incidents? Are you being specific enough to be able to measure this?
For example, the 2018 DBIR revealed that the rate of end users reporting phishing campaigns is very low, with only 17 percent of incidents being reported. Consider the impacts of this finding. Your end users are your first line of defense and you need them to take two actions: don’t click the link and report it quickly.
The sooner your response team knows about a phishing campaign, the faster they can take technical actions to mitigate the incident. The data shows that on average, people who click on a phishing email will do so within an hour of receiving it. Thus, you want your savvier end users to act fast. This response time is something you can certainly measure with test campaign data, as well as real incident data, to track behavior changes over time.
I’ve focused heavily on the anti-phishing component of security awareness training and measurement in this blog. Let’s face it, we need to keep working on this because phishing continues to be a major threat vector in data breaches. However, you should still be looking to other aspects of measurement for your security awareness program. SANS suggests several categories of measurement to consider:
- Phishing Awareness
- Phishing Detection/Reporting
- Number of Infected Computers
- Awareness Survey
- Number of Updated Devices
- Number of Lost / Stolen Devices
- Secure Desktop
- Passwords
- Social Engineering
- Sensitive Data
- Data Wiping or Destruction
- Device Physical Security
- Facility Physical Security
Summary
You may or may not currently include all these topic areas in your security awareness training program, but it’s a reasonable list to take a look at and consider the usefulness of metrics.
I know it’s unlikely you spend your full work day focusing on your security awareness program. By highlighting some keys areas of proven impact, I hope you can validate your current state of program operations or gain a few new ideas to make improvements going forward.