Pablo Zavala · AI Safety Evaluation · Research Engineering

Cybersecurity Anomaly Detection

A Carnegie Mellon security-analytics project that fits an Isolation Forest on benign-only Linux kernel traffic from the BETH honeypot dataset, then grades it under a frozen threshold. On an 18,000-event holdout at simulated 1% attack prevalence, the detector catches 90% of intrusions (162 of 180) at a 1.94% false positive rate, with ROC AUC 0.9714. The entry audits its own headline: precision sits at 0.3189, so roughly one alert in three flags a real attack, and three dominant features hand an attacker a spoofing playbook, which keeps the detector one layer in defense in depth.

Claim precision

What the evidence demonstrates: Frozen-threshold holdout: recall 0.9000 (162 of 180 attacks; 95% CI 0.85 to 0.94) at FPR 0.0194, ROC AUC 0.9714, from an Isolation Forest fit on benign-only traffic
Capability and evidence frontier: Precision at the chosen threshold is 0.3189 at simulated 1% prevalence, the malicious class holds a single botnet-staging campaign on one honeypot host, and three dominant features leave the detector spoofable.

Public reproducible repo + executed notebook

Role: Sole author: feature engineering, separability analysis, benign-only detector fit, threshold governance, error analysis, and the public reproduction package.

Evaluation Card

Detection performance

Sample: 18,000-event holdout at simulated 1% prevalence, threshold frozen after validation
Evaluator: Recall, FPR, precision, ROC AUC, and PR AUC, with bootstrap 95% CIs from 400 holdout resamples
Result: Recall 0.9000 (162 of 180 attacks; CI 0.85 to 0.94) at FPR 0.0194 (346 of 17,820 benign events); ROC AUC 0.9714; PR AUC 0.2972; precision 0.3189 (CI 0.28 to 0.36); alert budget 2,822 per 100k events.
Verification scope: One attack campaign and one honeypot environment; the metrics grade this dataset rather than production traffic.

Error analysis

Sample: The 346 false positives the frozen threshold flags on the holdout
Evaluator: Process and event taxonomy over the flagged benign traffic
Result: systemd processes carry 86% of false positive volume, and getdents64 directory enumeration fires at 37x its benign baseline, so an alert on getdents64 from systemd deserves a fast-path review rule before anyone pages a human.
Verification scope: Triage guidance derives from one environment's benign baseline.

Evidence level

Sample: Public repository: executed notebook with embedded outputs, nine committed figures, pinned requirements, and a data rebuild script
Evaluator: RANDOM_STATE=42 re-execution plus count assertions in the rebuild script
Result: Re-execution reproduced every headline metric exactly, with 14 of 15 code cells byte-identical (one cosmetic pandas dtype-label diff); the rebuild script asserts the three defining row counts; MIT covers the code and the dataset carries CC0 1.0.
Verification scope: The 95 MB subset stays outside version control, so a full rebuild needs a Kaggle account; a 500-row stratified sample ships for schema inspection.

Affiliation

Sample: Carnegie Mellon University coursework, fall 2024, revised February 2025
Evaluator: Repository README attribution
Result: Solo security-analytics coursework, upgraded to a public reproducible repository.
Verification scope: Course handouts, prompts, and grading context stay private.

Page updated

Sample: July 2026
Evaluator: Site content ledger
Result: Entry drafted July 10, 2026 from the public repository: its README, data documentation, and executed notebook outputs.
Verification scope: Numbers quote the committed executed notebook and its documented re-execution rather than a rerun on fresh traffic.

Evaluation axes with sample size, evaluator, result, and verification scope.
Axis	Sample	Evaluator	Result	Verification scope
Detection performance	18,000-event holdout at simulated 1% prevalence, threshold frozen after validation	Recall, FPR, precision, ROC AUC, and PR AUC, with bootstrap 95% CIs from 400 holdout resamples	Recall 0.9000 (162 of 180 attacks; CI 0.85 to 0.94) at FPR 0.0194 (346 of 17,820 benign events); ROC AUC 0.9714; PR AUC 0.2972; precision 0.3189 (CI 0.28 to 0.36); alert budget 2,822 per 100k events.	One attack campaign and one honeypot environment; the metrics grade this dataset rather than production traffic.
Error analysis	The 346 false positives the frozen threshold flags on the holdout	Process and event taxonomy over the flagged benign traffic	systemd processes carry 86% of false positive volume, and getdents64 directory enumeration fires at 37x its benign baseline, so an alert on getdents64 from systemd deserves a fast-path review rule before anyone pages a human.	Triage guidance derives from one environment's benign baseline.
Evidence level	Public repository: executed notebook with embedded outputs, nine committed figures, pinned requirements, and a data rebuild script	RANDOM_STATE=42 re-execution plus count assertions in the rebuild script	Re-execution reproduced every headline metric exactly, with 14 of 15 code cells byte-identical (one cosmetic pandas dtype-label diff); the rebuild script asserts the three defining row counts; MIT covers the code and the dataset carries CC0 1.0.	The 95 MB subset stays outside version control, so a full rebuild needs a Kaggle account; a 500-row stratified sample ships for schema inspection.
Affiliation	Carnegie Mellon University coursework, fall 2024, revised February 2025	Repository README attribution	Solo security-analytics coursework, upgraded to a public reproducible repository.	Course handouts, prompts, and grading context stay private.
Page updated	July 2026	Site content ledger	Entry drafted July 10, 2026 from the public repository: its README, data documentation, and executed notebook outputs.	Numbers quote the committed executed notebook and its documented re-execution rather than a rerun on fresh traffic.

How to Inspect This Work

Threshold governance

Every headline number comes from one evaluation design: a 60,000-event sample at simulated 1% attack prevalence splits 52.5/17.5/30 into fit, validation, and holdout, stratified on the label, and the alert threshold (0.6892, the F2 optimum on the validation precision-recall curve) freezes before the 18,000-event holdout gets scored. The reported recall and false positive rate therefore measure a governed decision rule rather than a curve searched after the fact.

What the labels touch

The forest's fit call receives features only, and its 31,185-event training pool contains zero attack rows. Labels enter at exactly three points: filtering the fit split to benign traffic, choosing the validation threshold, and grading the frozen holdout, the standard shape of a novelty-detection deployment.

Reader check

The repository reruns end to end: the data rebuild script asserts the three defining row counts, requirements stay pinned, and a RANDOM_STATE=42 re-execution reproduced every headline metric exactly, with 14 of 15 code cells byte-identical to the original and the remaining diff a cosmetic pandas dtype label.

Case Study

Problem

Kernel-level security telemetry arrives faster than anyone can label it, so a deployable detector must learn benign behavior alone and still catch attacks it has never seen; the evaluation must also keep the headline honest at realistic attack rates.

Setup

The subset holds 347,399 kernel-process events with 16 fields across 5 honeypot hosts, drawn from the 8,004,918-event BETH corpus (Highnam et al. 2021): the full labelled validation split plus one attacked host's 158,432 confirmed-malicious events, which enriches the malicious rate to 45.6% for exploratory analysis. Both labels come manually annotated by the dataset authors, and the detection models resample to a realistic 1% prevalence.

Method

Sixteen raw fields distill to 7 engineered features following the BETH paper's guidance, a UMAP projection confirms the engineered space separates the classes before any model trains, and an Isolation Forest with 300 trees fits on 31,185 benign-only events. A 60,000-event sample at simulated 1% prevalence splits 52.5/17.5/30 into fit, validation, and holdout; the alert threshold maximizes F2 on validation and freezes before holdout scoring, with bootstrap CIs from 400 resamples.

Result

The frozen threshold catches 162 of 180 holdout attacks (recall 0.9000; 95% CI 0.85 to 0.94) while flagging 346 of 17,820 benign events (FPR 0.0194), with ROC AUC 0.9714, PR AUC 0.2972, and an alert budget of 2,822 per 100k events. Precision lands at 0.3189: at 1% prevalence, roughly one alert in three flags a real attack, the deliberate price of weighting recall over precision.

Verification scope

The malicious class holds a single botnet-staging campaign on one host, all traffic comes from one honeypot environment, and the subset enriches malice to 45.6% while only the resampled 1% prevalence mirrors production; the notebook's early in-sample demonstration (recall 1.0 at precision 0.034, at the default decision boundary) illustrates score behavior only and stays out of the headline.

Evidence

The public repository carries the executed notebook with embedded outputs, nine committed figures, pinned requirements, a data rebuild script that asserts the defining row counts, and a documented RANDOM_STATE=42 re-execution that reproduced every headline metric exactly.

Key Outcomes

Recall 0.9000 (162 of 180 attacks; 95% CI 0.85 to 0.94) at a 1.94% false positive rate on the frozen-threshold holdout, with ROC AUC 0.9714
The Isolation Forest fits on 31,185 benign-only events, and labels enter at exactly three points: the benign fit filter, the validation threshold choice, and holdout grading
Precision 0.3189 on the frozen-threshold holdout at simulated 1% prevalence and an alert budget of 2,822 per 100k events: roughly one alert in three flags a real attack, stated on the page
Error taxonomy turns false positives into triage guidance: systemd carries 86% of volume and getdents64 fires at 37x its benign baseline
A RANDOM_STATE=42 re-execution reproduced every headline metric exactly, with 14 of 15 code cells byte-identical to the original

Methods

Isolation Forest
UMAP
Threshold governance
Bootstrap confidence intervals
Unsupervised anomaly detection