Security researcher and threat intelligence practitioner. Former Security Research Engineer at Cisco ASIG. Founder of Southwest Valley Research Group. CVE co-author. GXPN · GCPN · CRTO · OSCP.
Building a Large-Scale Breach Intelligence Pipeline: Collection, Ingestion, and ClickHouse Analytics
Several years of operational threat intelligence work made the limitations of ad-hoc tooling obvious. Manual searches across scattered breach dumps, inconsistent parsing, and slow aggregation queries limited my ability to perform timely identity correlation, password reuse analysis, and actor profiling. This post documents the full pipeline I built, from primary-source collection through normalization and loading into a purpose-built analytical backend, and the reasoning behind key technical decisions. Requirements The system was designed around four core needs: ...
Building a Threat Intelligence Research Platform: Architecture and Design Decisions
Several years of running threat intelligence and investigative operations on informal infrastructure made the gaps obvious. Breach data analysis, Telegram collection, case investigation tooling, and security research were sharing the same environment with no isolation between operational contexts, no network visibility, and no detection capability pointed inward. A malformed archive during ingestion could affect active case data. A noisy scan could contaminate telemetry. This post documents the architecture built to address those problems and the reasoning behind each design decision. ...