Python
Docker
Git
SQLite
CythonWhat it is
A large-scale network traffic (NetFlow dump) processing pipeline, used as input for a DDoS detection and mitigation system. For each traffic record, the system resolves source and destination IP geolocation and delivers clean, structured data for downstream consumption.
Challenge
Process millions of rows of traffic data within a 5-minute window, under strict time and memory constraints. Beyond the volume, each row required an IP geolocation lookup, which made pure Python processing unfeasible within the required SLA.
Solution
- Hybrid architecture with Python as the orchestrator (flow control, aggregation, and output formatting) and Cython to accelerate the critical paths — both in NetFlow dump parsing and geolocation lookups.
- Used the MaxMind GeoIP database to resolve source and destination IP geolocation at scale.
- Parallel processing to handle the data volume within the available time window while keeping memory usage under control.
- Cleaned and standardized output data, producing structured datasets later used to feed presets for the anti-DDoS system.
Result
A pipeline capable of processing large volumes of network traffic within the required time SLA, delivering clean, geolocated data as the foundation for automated attack mitigation decisions.
Tech Stack
- Python
- Cython
- MaxMind GeoIP
- SQLite
- Parallel processing / multiprocessing
Technical Highlights
- Optimized critical code paths with Cython to reach near-C performance in Python
- IP geolocation lookups at scale (millions of records) within a tight time window
- Parallel processing to balance speed and memory usage
- Data cleaning and standardization pipeline for downstream security systems