In this three-part blog series (part one, part two), the security operations experts at Arctic Wolf walk you through everything you need to know about penetration testing (“pen tests”) and the security benefits they can provide your business.
In this third and final blog post we explain what to do when you “fail” your pen test. And away we go…
What Happens When You “Fail” Your Pen Test?
You entered the test with high hopes. You’ve made real strides in your security posture every quarter, every year, and maybe you thought that your cybersecurity was better than most businesses, or at least above average.
But you failed.
Maybe you found out in the form of a late-night incident, where your existing detection tools alerted on a pen tester’s activities—but only after a compromise already occurred. Maybe the pen testers caused an outage you had to resolve (see outcome scoping in Planning Your Pen Test for more). Maybe you learned you failed from the pen tester’s after-action report, where you discovered that your systems had been compromised without your knowledge.
This outcome is disappointing, and frustrating. It can be embarrassing to report a pen test failure; it can feel like all the hard work and focus you applied to cybersecurity has been wasted. Such reactions are reasonable and understandable. You feel like you failed.
But it’s important to remember: No, you didn’t fail. A pen test that exposes vulnerabilities isn’t a “failure”—it is exactly the outcome you needed. A skilled pen tester will almost always find some vulnerability to exploit and highlight—you should be getting something for your pen test investment, after all! When your pen tester compromises your systems, they have accomplished their mission. And they’ve provided you with a rare opportunity to identify and close real gaps in your security so that you can defend the business.
Getting the Most Out of Your Pen Test
The results of your penetration test should include both areas of success and defined areas for improvement, along with a detailed account of the methods the pen tester exploited to compromise your systems. Now it’s your turn to leverage this gold mine of security intelligence. Gather your stakeholders to review the test’s outcome. As always, you’ll want to cast a wide net—effective cybersecurity is a cross-functional exercise focused on business risk, not just IT threats alone.
An important stakeholder to include in this work is your blue team, including any security partners (such as Arctic Wolf). When a “red team” and a “blue team” can collaborate on a testing exercise and on review of the resulting reports, they typically produce a more effective total understanding of your security.
Next, review your tester’s after-action report. You’re looking to understand two things:
1. What areas of security resisted your pen tester?
Even when a pen test “fails,” a tester will typically evaluate or attempt multiple methods of compromise, and report on the effort expended on these secure areas. Like physical safes, which are rated by how long a time they can resist an expert safecracker, understanding the depth of your cybersecurity will help you continue to improve it.
Reviewing these attempts and the defenses that worked will help you validate some of your security controls, strike the correct ongoing balance between security and usability, and concentrate attention on key areas of exposure.
2. Understand the kill chain.
Review how the pen tester achieved their exploit against your business systems. How did they perform reconnaissance to discover your vulnerabilities? What attack tools did they select and why? How did they access your systems and execute the attack, while evading your defenses and detection? Use the MITRE ATT&CK Framework to map out exactly how the compromise occurred.
At every stage, identify how your organization could have broken the kill chain and thwarted the attack. Be creative—the most effective defense isn’t always the most obvious. Detection and response capabilities are often a less-disruptive defense than additional levels of protection which may be cumbersome and impede usability.
For example, if the pen tester used social research to identify the CFO’s name and spoof their email in a spear-phishing malware attack, don’t just consider email security tools. Think about clearly identifying internal versus external emails; changing policies so that internal email addresses are harder to guess, and hardening procedures around file sharing and verifying attachments.
At every stage of the kill chain, you should be able to identify several areas of security improvement, from the technical to the procedural to the personal.
Once you’ve laid out areas for possible improvement, evaluate which ones you want to adopt first. Remember, a single improvement in a single stage will break the kill chain for this attack but attackers always work to reroute around defenses. Prioritize the improvements that will add the most robust defense against a range of attacks. Such improvements can span from tactical countermeasures—such as changes to configurations, permissions, rules, and procedures in existing systems—to strategic enhancements, such as new security investments, re-architecture activities, GPO changes, and more.
Make sure that your investment in time and resources is strategic. Rather than adding a single malware signature to your endpoint tools, think about how you can use threat intelligence to keep detection continuously updated. Rather than changing the alerting priority around executive email accounts, think about how your business can clear all security alerts in a timely manner. Rather than simply locking down your Remote Desktop Protocol permissions, consider whether your organization detects misconfigurations and vulnerabilities across systems on an ongoing basis.
Assign Responsibility
Once you know what security improvements you plan to make, assign the responsibility to implement them. For most items, primary responsibility will belong to IT, but expect that approximately 20% of all changes may involve other functional groups. When assigning the responsibility for these improvements, be realistic about what each team will require to get them done. Will you push out other items of the plan? Allocate additional budget or headcount? Reduce responsibility sprawl?
A pen test is a real opportunity to force meaningful change—don’t squander it with an after-action exercise where everyone simply promises to really, truly do the stuff that they had already promised to do in the past, but never did.
Some of the improvements you identify will require executing on existing plans or capabilities, e.g., installing already-purchased security tools that quickly became shelfware, or assigning clear escalation responsibilities for alerts. And some improvements you identify may require additional support from new or existing partners or vendors.
In this latter category, capabilities to implement may include:
- Single pane of glass visibility across your various systems at risk, including network, endpoint, cloud, and security tooling
- 24×7 security monitoring, with near-real-time detection, escalation, triage, and response
- Risk and vulnerability management, including guidance on which patches and fixes to prioritize
- Security operations, where a vendor’s offering provides access to dedicated security experts who understand your business’s needs and respond to incidents or pen tests in progress
- Security journey support, a relationship with security partners that facilitates ongoing hardening and posture improvement as you extend beyond the insights driven by an individual pen test
You can work with your existing security providers to understand which capabilities are in scope for them. You should also explore other options, including potential security partners better positioned to enhance your capabilities and replace partnerships that didn’t meet the security challenges set by the pen test.
Take the gaps exposed by your pen test failure seriously. Engage in a meaningful after-action evaluation, and vigorously implement the new capabilities your organization requires. Once you do so, you may find the failed pen test was the best thing to happen for your company’s security. Then, after an appropriate length of time passes, perform another pen test. Your blue team will likely be on a firmer footing—whether that means new staff, new skills, or new security partnerships—and you’ll be able to engage in a full purple team exercise.
And so next time, you might just find that:
You “Passed” Your Pen Test!
If you’ve “passed” a pen test, either on your first try or in a subsequent engagement, congratulations! This is a testament to all the hard work you’ve done to establish a meaningful security practice for your organization. Take a moment to enjoy your success.
But in the immortal words of Han Solo, “Great, kid! Don’t get cocky.” A successful pen test is not the culmination of a security journey, just another milestone along the way. A successful pen test is no guarantee that you can defeat tomorrow’s attackers—armed with new tricks, new tools, new zero-day vulnerabilities, and simply another chance.
You should still perform the same after-action exercise. Make sure that your pen tester conducted robust attempts to compromise your system and reported on those activities in detail. Understand exactly how you blocked the pen tester’s attacks; understand which defenses, if any, the pen tester was able to bypass. Your list of actions to implement may be shorter and sweeter, but it is no less serious.
Use this opportunity to also evaluate the systems and techniques out of scope of the pen test and determine if you would have protected yourself as effectively there. Take time to consider upcoming changes in your organization that will leave you exposed in areas where you were protected today. Do you have a cloud migration coming up that will expose high-value data out from under your validated on-premises security? Will you be adding new locations or systems, and are they up to your security standard?
And finally, ask yourself: How can I continue to improve my defense in depth? Can I add another layer or enhancement to my detection and risk management capabilities? Are my employees properly trained to protect the organization? And so on.
By asking these questions, you’ll give yourself the best possible opportunity to tackle the next pen test—and keep your business secure.
Thanks for joining us on this blog series. To learn more about the benefits of pen tests, read Why You Need a Pen Test, and to plan one, check out Planning Your Pen Test.
