In AI-powered security, advantage comes not from automation alone, but from clear insight into how decisions are made.
At Arctic Wolf, home to one of the world’s largest commercial security operations centers (SOC), we process over 10 trillion security events weekly.
Rather than chasing automation for its own sake, we build AI that scales human expertise – preserving judgment where it matters most.
But what is the optimal combination of humans and machines for security operations? How do you best combine AI agents with human experts to defeat evolving cyber adversaries at scale?
It’s not philosophy. These are critical engineering problems that require empirical measurement. And it parallels what Moneyball solved in baseball: determining when human expert judgment creates more value than algorithmic speed.
Speed ≠ Security
Early SOC automation optimized for velocity: alerts closed, tickets resolved, MTTR reduced. Great dashboard metrics. But not reliable indicators of actual security.
Adversaries adapt. Data shifts. Models drift. The real risk isn’t under-automation — it’s overconfidence in systems you can’t validate.
The modern SOC needs different metrics. We need to know if AI maintains precision when signals are weak or deliberately manipulated. We need to measure escalation quality when uncertainty is high, not when the answer is obvious. We need to track robustness as attack patterns evolve and validate alignment with how experienced analysts actually make decisions under pressure.
Moneyball’s Lesson: Measure What Wins
Moneyball revolutionized baseball by measuring contributions to win, not radar gun readings.
The same principle applies to SOC automation.
Don’t ask if the AI ran. Ask if it changed the investigation’s outcome. Did it surface threats humans would miss? Did it reduce analyst workload without hiding risk? Most critically – was the AI humble and properly escalated to a human when needed?
That last question matters most. At scale, effective automation depends on continuously monitoring and evaluating systems limits in a complex and evolving threat environment.
Engineering Autonomous Human Machines Teams & Systems
At 10 trillion events weekly, Arctic Wolf runs Databricks Lakeflow Jobs to orchestrate the data used for our complex security investigations. We also use Databricks Agent Bricks that extend analyst capabilities while knowing when to escalate to the right human. AI Agents automatically handle pattern recognition and initial triage. Human experts make calls that require judgment and additional context. Determining where those calls matter requires continuous measurement.
With Databricks, we’ve built an evaluation infrastructure that helps answer: When does human intervention measurably improve outcomes? How do you combine AI with security experts to optimize human-machine teams and systems? The Databricks Data Intelligence platform unlocks new data and AI use cases at the scale we need, secured through unified governance, and with quality AI agents that know our business.
The system traces decisions in real-time, capturing confidence scores and reasoning chains as they happen. We benchmark AI decisions against expert analyst responses to establish ground truth. Pattern analysis reveals which scenarios benefit most from human intervention. Drift detection catches model degradation before it degrades security posture. And we can also track divergence between human and machine decisions to understand not just where they disagree, but why.
It’s like tracking every pitch in baseball and then using that data to know when the pinch hitter changes your odds.
Human-in-the-Loop as Engineering
Human-in-the-loop gets debated as ethics. That matters. But it’s not actionable without empirical grounding.
We treat it as a measurement. Which alert scenarios produce the highest AI error rates when adversaries deliberately evade detection? Where do analysts consistently override AI recommendations, and when are they right? What confidence thresholds actually predict when human review adds material value? How does the human-machine performance balance shift as threat tactics evolve?
At our scale, effective augmentation means knowing precisely when to automate and when to escalate. That knowledge comes from measuring your operational reality, not industry assumptions.
The balance shifts constantly, based on things as varied as the threat landscape, data quality, and system performance. You find the balance through instrumentation and expert feedback.
Evidence Over Promises
Security analysts can gain trust in AI systems that can be monitored, audited, and improved from feedback. In the same way, security leaders trust automation that demonstrably reduces risk while maintaining visibility and transparency. They trust human augmentation where the human component is empirically justified.
This is the same trust challenge Billy Beane faced with Moneyball. Skepticism dissolved when data driven wins became undeniable.
The same goes when competing against cyber adversaries. Like elite athletes, they are highly adaptive competitors, meaning the challenge is no longer just about automation — it’s about improving AI precision, agility, and trust at a massive scale.
The Data-Driven Advantage
Moneyball demonstrated a fundamental principle: measurement beats intuition. Security operations are no different: Elite teams don’t debate humans versus AI — they instrument when machines reduce risk and quantify the results.
At one of the world’s largest commercial SOCs processing 10 trillion events weekly, we’ve engineered human-machine teams that operate at an unprecedented scale. These aren’t collaborative in theory but optimized through continuous measurement of human-augmented AI teams. In partnership with Databricks, optimizing human-in-the-loop becomes quantifiable: we know how to best combine human experts and machines.
The most advanced AI-driven cybersecurity doesn’t sideline humans. It makes their involvement intentional, observable, and provably valuable — showing, not assuming, when judgment matters.
The real question isn’t whether your AI routes decisions to human analysts, but whether those handoffs are governed by calibrated confidence, observable reasoning, and clear attribution.
Authors
Michael Mylrea, AI Technical Fellow & Architect, Arctic Wolf
Dan Schiappa, President, Technology and Services, Arctic Wolf
Ari Kaplan, Global Head of Evangelism, Databricks, and a leading influencer in data, AI, and sports analytics. Moneyball, the popular movie was partly based on Ari’s analytical and scouting experiences innovating Major League Baseball, and later creating the Cubs, Dodgers, and Orioles analytics departments.
