Self-configuring NetFlow anomaly detection using cluster density analysis

The growing number of malicious network attacks has resulted in the need for a fast, reliable method to identify possible malicious activity. For any organization, it is critical that confidential and proprietary data is sufficiently secured to address both legal and contractual obligations. The changing nature of security attacks has caused a surge of interest in anomaly detection mechanisms. Such mechanisms are suitable as they can dynamically adapt to changed network conditions and threats without security personnel intervention. While anomaly detection mechanisms have significant potential, they are technically limited. Many anomaly detection approaches are unsuitable for real time environments. The approaches also typically operate based on 'what is common is normal'. Mechanisms are typically singular in focus analysing data on one specific type. This paper proposes a novel framework to detect anomalies previously hidden within current detection techniques. The approach is easily extensible taking input from many security assessment applications; network traffic, asset criticality. Using time based correlations with historic data; a method for generating a normalized view of activity on the network is achieved. Once normality has been established for specific time intervals an extensible environment is implemented which allows for the active monitoring of anomalies in real-time. Anomalies which had sufficient 'commonality' to remain undetected by other mechanisms are identified and analysed. The proposed solution is completely autonomous, capable of acting independently with no previous knowledge required. The presented results describe NetFlow activity of the NPD Groups' network over a 24-hour period and outline real world anomalies that were detected.