/projects/detections-under-load
Detection Under Load: Measuring Sigma Rule Robustness
An execution-based benchmark that measures published Sigma coverage for LSASS credential dumping, separates telemetry gaps from rule-logic misses, and tests how detections survive operator-controlled changes.
Overview
Detection Under Load is a measurement harness for detection rules. It replaces description-level confidence with an execution question: when the same ATT&CK technique is performed by several real tools, which published rules actually fire, why do the others miss, and which detections survive simple operator choices?
Research question
The first study focuses on LSASS credential dumping (T1003.001). Public telemetry provides seven comparable captures produced in one lab under one Sysmon configuration, with the dumping tool as the principal variable. This makes it possible to compare rule behavior without assembling unrelated recordings into one synthetic benchmark.
My role
I designed and implemented the benchmark, including corpus pinning, Sigma evaluation, miss classification, adversary-controlled mutation, rule-population analysis, independent engine cross-checking, result generation, and the detection content written in response to the findings.
Reproducible corpus and rule inputs
The manifest pins source repositories by commit and capture archives by SHA-256. Corpus preparation separates attack captures from eligible benign recordings and excludes unlabelled or sibling-technique captures rather than assuming they are clean. A rerun therefore consumes the same bytes or fails explicitly.
Native Sigma evaluation
The evaluator parses rules with pySigma and compiles their condition trees into predicates over event dictionaries. Log-source pipelines are applied according to each rule's requested telemetry. This avoids treating a backend conversion gap as a failure in the rule being measured.
Explaining a miss
Every rule and capture pair is classified as detected, missing telemetry, out of scope, or a logic miss. The classifier distinguishes absent event fields from tool-specific selectors and from cases where all required evidence exists but the condition still fails. That separation is essential: a coverage number without a reason can misdiagnose the collection layer as a detection defect.
Robustness under operator control
The mutation ladder changes only artifacts the operator controls: names, locations, version resources, and recorded fingerprints. It does not rewrite operating-system-reported behavior. A control mutation changes a field no selected rule reads; any resulting coverage change invalidates the run.
Measured findings
The benchmark evaluates 80 published T1003.001 rules against seven dumping tools in 354,229 Windows events. Each tool triggers between three and eight rules, with a median of five. Renaming operator-supplied artifacts removes 12 of the 35 baseline detections, and relocation into commonly excluded directories removes eight more. Nanodump falls from three published detections to zero on rename alone.
The findings expose three recurring failure modes: selectors tied to operator-chosen strings, access masks omitted because of noise trade-offs, and broad directory filters that remove the activity they are meant to protect.
Detection content and delivery controls
Six response rules cover credential access, process injection, privilege activation, suspicious execution, and focused dumping behaviors. The repository validates Sigma syntax, verifies KQL and SPL conversion, and checks committed benchmark records in CI. Results are generated from machine-readable records instead of maintained as disconnected prose.
Independent validation
A Zircolite cross-check runs the same process-access population through an independently implemented engine. Across the seven campaigns and 23 process-access rules, both evaluators agree on which rules fire and on event counts.
Limits
The benchmark deeply measures one technique on one public corpus. Mutation tiers model operator choices rather than replaying newly recorded intrusions, and benign-corpus rates are not deployment-specific false-positive guarantees. The cross-check covers the process-access subset, not every Sigma construct.