Mimecast : Using NIST's Phish Scale to Optimize Employee Training

Running effective phishing training campaigns can be challenging. On one hand you want training to be effective and realistic, but on the other hand you don't want to overwhelm users. Fortunately, the National Institute of Standards and Technology (NIST) has developed the Phish Scale, a free, effective and easy-to-use tool to help you rate the difficulty of simulated phishing campaigns.[i] Using this tool as a guide could not only help you optimize training but also provide specific insights into user strengths and weaknesses so you can focus on the most significant performance metrics.

A Need for Better Metrics

In a Mimecast survey, 55% of security professionals said their company had received more phishing email than ever last year. The malicious links and attachments in these emails can lead to ransomware, theft of data and other crimes. Yet security departments are constantly under pressure to justify budgets to defend against them.

Senior managers want to know that budgeted funds are producing good returns on investment. This demand often manifests as pressure to demonstrate reduced click rates on simulated phishing campaigns. But while click rates are commonly used as a key performance indicator (KPI), there are other metrics to consider.

As a case in point, a sure way to reduce click rates is to repeatedly send out the same simulated phishing email. Of course, this is not effective training because the reduction in click rates simply demonstrates that users have gotten better at identifying and avoiding a familiar phishing message.

The NIST Phish Scale provides multiple KPIs that security teams can show to senior management as evidence of training gains. The scale uses two dimensions to determine how difficult it might be for an employee to identify a simulated message:

• Observable Cues: How many typos, misnomers or other false notes in the simulated email might tip off an employee that it's fake?
• Alignment: How well is the scam message aligned to employees' context, to fool them by mimicking common workplace processes or current company events?

The values for these two dimensions are combined to create an overarching level of email difficulty.[ii] To help implement this scale, NIST has provided several resources described in a guide titled "A Phish Scale: Rating Human Phishing Message Detection Difficulty."[iii]

Applying the Phish Scale

The Observable Cues dimension of the Phish Scale includes five categories:

• Error.
• Technical indicator.
• Visual presentation indicator.
• Language and content.
• Common tactics.

Examples in these categories include spelling and grammar irregularities, inconsistent email address, logo imitation, threatening language and a limited-time offer.

The Alignment dimension includes five message considerations:

• Mimicking a workplace process.
• Workplace relevance.
• Timing vis à vis other situations or events (such as a news story).
• Engendering concern for NOT clicking.
• Having been the subject of targeted training or specific warnings.

Each of these considerations is rated as "extreme" (8 points), "significant" (6 points), "moderate" (4 points), "low" (2 points) or "not applicable" (0 points).

To assess the difficulty rating, first count each instance of an Observable Cue (every individual instance of a spelling or grammar error for instance). Once you have the total, determine whether you have "few cues" (1 to 8), "some cues" (9 to 14) or "many cues" (15 or more). Next assign a corresponding number of points for each of the Alignment considerations and obtain this total to determine which category of Alignment the message belongs to: "high" (18 or above), "medium" (11 to 17) or "low" (10 or fewer). Finally, determine the difficulty level of the phishing email using Table 1, below.

	Alignment Context	Table 1: NIST Phish Scale, Email Difficulty
	High	Very Difficult	Very Difficult	Moderately Difficult
Medium	Very Difficult	Moderately Difficult	Moderately Difficult
Low	Moderately Difficult	Moderately Difficult	Least Difficult
		Few	Some	Many
	Observable Cues

Working Through an Example

Using the phish example in Figure 1 below, we can count ten Observable Cues and then assign the following points for Alignment: "mimics a workplace process" (8 points), "workplace relevance" (8 points), "aligns with situations or events" (4 points), "concern for NOT clicking" (2 points) and "subject of targeted training" (0 Points), for a total of 22 points on Alignment.

We now have a rating of "some" Observable Cues and a "high" Alignment. Referring back to Table 1 above, we see that this campaign would receive a "very difficult" rating.

Source: Canham et al[i]
[i] "Phish Derby: Shoring the Human Shield Through Gamified Phishing Attacks," Frontiers Reporting to Senior Management

Having two metrics to consider, difficulty and failure rate, means that you can demonstrate value through employee improvement, even if click rates remain stable.

Documenting employees' improvement in spotting difficult emails alongside sustained click rates is still evidence for improved performance, because users are performing equally well on more difficult emails. If click rates decline over time, you also have evidence that this represents a meaningful improvement in performance.

Better Training Results

Using the NIST Phish Scale can give you the additional bonus of fine-tuning your phishing campaigns by considering the number of Observable Cues separate from the Alignment context. Having a close hold on the difficulty of emails allows you to tailor campaigns according to ability. Increasing the difficulty of campaigns to add challenge for your cyber resilience stewards, or decreasing difficulty for your repeat clickers, creates a more productive training experience for all users regardless of ability levels.

Monitoring the Alignment dimension of the Phish Scale can also provide you with insight into which job roles or departments are most susceptible to various phishing pretexts and allow you to adjust training for these users accordingly.

The NIST Phish Scale was introduced in late 2020 as a free tool. You can also buy email security, training, reporting and testing platforms, tools and services with varying degrees of sophistication and integration.[1] For example, Mimecast's Safe Phish tool lets security teams turn real-life, "de-weaponized" phishing attacks on your company into training exercises.

The Bottom Line

Email phishing presents a persistent risk to companies, whose employees are often targeted with messages containing malicious links, infected attachments and other exploits that can lead to ransomware, data theft and other crimes. NIST's Phish Scale can help companies train their employees to avoid these traps.

[1] "Security Awareness Computer-Based Training Reviews and Ratings," Gartner

[i] "The Phish Scale: NIST-Developed Method Helps IT Staff See Why Users Click on Fraudulent Emails," National Institute of Standards and Technology

[ii] "Categorizing Human Phishing Difficulty: A Phish Scale," Journal of Cybersecurity

[iii] "A Phish Scale: Rating Human Phishing Message Detection Difficulty," National Institute of Standards and Technology