Blue Team
Your metrics are boring and dangerous. Recycled slides with meaningless counts of alerts, incidents, true and false positives… SNOOZE. Even worse, it’s motivating your team to distort the truth and subvert progress. This talk is your wake-up call to rethink your detection and response metrics. Metrics tell a story. But before we can describe the effectiveness of our capabilities, our audience first needs to grasp what modern detection and response is and its value. So, how do we tell that story, especially to leadership with a limited amount of time? Measurements help us get results. But if you’re advocating for faster response times, you might be encouraging your team to make hasty decisions that lead to increased risk. So, how do we find a set of measurements, both qualitative and quantitative, that incentivizes progress and serves as a north star to modern detection and response? Metrics help shape decisions. But legacy methods of evaluating and reporting are preventing you from getting the support and funding you need to succeed. At the end of this talk, you’ll walk away with a practical framework for developing your own metrics, a new maturity model for measuring detection and response capabilities, data gathering techniques that tell a convincing story using micro-purple testing, and lots of visual examples of metrics that won’t put your audience to sleep.
Explain the importance of this talk/demo: This talk presents a new approach to detection and response metrics. I propose moving away from the typical approach of measuring effectiveness solely based on quantitative indicators, such as event counts, which are often used by security operation centers or legacy detection and response programs. I introduce a new maturity model for measuring detection and response capabilities. I provide a methodology for utilizing micro-purple testing – tests that validate detection logic and analysis and response processes – to measure overall visibility into threats. Finally, I walk the audience through a practical framework that will help them develop their own metrics. Key takeaways
1. A new maturity model that helps tell the story of modern detection and response, the value it provides, and how your current capabilities level against your goal state. 2. Visual examples of metrics you can use today to present across teams and leadership, along with a framework for developing your own detection and response metrics and practical advice on how to strategically move to these modern metrics when change is hard and leadership hates surprises. 3. Methods to measure and prioritize threat coverage with micro-purple testing – tests that validate detection logic and analysis and response processes. Who will enjoy this talk?
* A CISO that wants to better understand what modern detection and response metrics should look like and how to include them in their overall program metrics.
* Managers and directors that present detection and response metrics to leadership and the rest of their organization.
* Engineers and analysts that are tired of their work being misrepresented with sad, unmotivating metrics.
* Anyone interested in learning more about detection and response.
Allyn Stott is a senior staff engineer at Airbnb. He currently works on the information security technology leadership team where he spends most of his time working on threat detection and incident response. He especially enjoys building strategies for hunting down advanced threat actors. Over the past decade, he has built and run detection and response programs at companies including Delta Dental of California, MZ, and Palantir. Red team tears are his testimonials.
In the late evenings, after his toddler ceases all antics for the day, Allyn writes a semi-regular, exclusive security newsletter. This morning espresso shot can be served directly to your inbox by subscribing at meoward.co.
Allyn has previously presented at Black Hat, Kernelcon, The Diana Initiative, Texas Cyber Summit, and BSides Berlin, Singapore, Toronto, Seattle, Orlando, St Pete, San Antonio, Charleston, and Atlanta. He received his Masters in High Tech Crime Investigation from The George Washington University as part of the Department of Defense Information Assurance Scholarship Program.
We proudly present SecretCon, an entirely unparalleled conference for the state of Minnesota, built for our new digital reality. This conference is dedicated to the many specialties of our hacker, cybersecurity, and privacy community. We have taken it upon ourselves to construct a conference that not only embraces our past, but also looks to the future. Join us!
