Network Traffic Analysis with ntopng and Suricata on Ubuntu: A Practitioner's Guide to Network-Level Threat Detection
Complete guide to deploying ntopng and Suricata on Ubuntu for enterprise network security monitoring. Covers installation, integration, rule management, threat intelligence, performance tuning, and troubleshooting for comprehensive threat detection.
Network Traffic Analysis with ntopng and Suricata on Ubuntu: A Practitioner's Guide to Network-Level Threat Detection
Understanding the landscape of network security monitoring
After spending years in the trenches of network security, I've learned that the most effective threat detection comes from combining multiple perspectives on network traffic. While signature-based detection catches known threats and behavioral analysis identifies anomalies, the real magic happens when you bring these approaches together. That's exactly what we're going to build today – a comprehensive network monitoring solution that pairs Suricata's powerful intrusion detection capabilities with ntopng's deep packet inspection and visualization prowess.
The integration we're building serves as your network's early warning system, capable of detecting everything from basic port scans to sophisticated data exfiltration attempts. What makes this combination particularly powerful is how Suricata's rule-based detection complements ntopng's protocol analysis and behavioral monitoring. While Suricata excels at identifying specific attack patterns through its extensive rule sets, ntopng provides the context and visualization that transforms raw alerts into actionable intelligence. Together, they create a monitoring ecosystem that's greater than the sum of its parts.
This guide takes you through a production-ready deployment on Ubuntu, specifically targeting the latest LTS releases. We'll start with the fundamentals and progressively build toward an enterprise-grade implementation that can handle multi-gigabit traffic loads while maintaining the kind of visibility that security teams dream about. Whether you're protecting a small business network or securing enterprise infrastructure, the principles and configurations we'll explore scale accordingly.
Prerequisites and initial system preparation
Before diving into the installation, let's ensure your Ubuntu system is properly prepared for what's ahead. I've found that spending a few extra minutes on system preparation saves hours of troubleshooting later. Start with a fresh Ubuntu 22.04 or 24.04 LTS installation – while older versions might work, the latest LTS releases offer better performance and security features that we'll leverage throughout this deployment.
Your hardware requirements will vary based on network size and traffic volume, but for a production deployment monitoring a medium-sized network, I recommend at least 8 CPU cores, 16GB of RAM, and 500GB of fast storage. The CPU cores are particularly important because we'll be distributing packet processing across multiple threads, and having dedicated cores for this work prevents the kind of resource contention that leads to dropped packets and missed detections.
# Update your system and install essential dependencies
sudo apt update && sudo apt upgrade -y
sudo apt install -y software-properties-common wget curl \
build-essential git python3-pip jq net-tools ethtool \
redis-server mariadb-server mariadb-client
Network interface configuration deserves special attention here. Modern network cards come with various offloading features that, while great for general performance, can interfere with packet analysis. I've learned the hard way that these features can cause Suricata to miss packets or miscalculate checksums, leading to false negatives that defeat the entire purpose of our monitoring system.
# Disable problematic NIC offloading features
sudo ethtool -K eth0 gro off lro off tso off gso off
sudo ethtool -K eth0 rx-checksumming off tx-checksumming off
# Increase ring buffer sizes for better packet capture performance
sudo ethtool -G eth0 rx 4096 tx 4096
# Make these changes persistent across reboots
sudo nano /etc/network/interfaces
# Add these lines under your interface configuration:
# up ethtool -K eth0 gro off lro off tso off gso off
# up ethtool -G eth0 rx 4096 tx 4096
Database preparation is another crucial step that's often overlooked. Both tools can generate substantial amounts of data, and having properly configured storage backends ensures you won't lose valuable forensic information when you need it most. Redis serves as ntopng's high-speed cache for real-time data, while MariaDB handles long-term storage of flow records and historical data.
# Secure MariaDB installation
sudo mysql_secure_installation
# Create database and user for ntopng
sudo mysql -u root -p
CREATE DATABASE ntopng;
CREATE USER 'ntopng'@'localhost' IDENTIFIED BY 'StrongPasswordHere123!';
GRANT ALL PRIVILEGES ON ntopng.* TO 'ntopng'@'localhost';
FLUSH PRIVILEGES;
EXIT;
Installing and configuring Suricata IDS/IPS on Ubuntu
The foundation of our threat detection system begins with Suricata installation. While Ubuntu's default repositories include Suricata, I strongly recommend using the official PPA to get the latest stable version. The performance improvements and security fixes in newer versions are substantial, and you'll want access to the latest detection capabilities.
# Add the official PPA for the latest stable Suricata release
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt update
# Install Suricata with required tools
sudo apt install -y suricata suricata-update
# Verify installation
sudo suricata --build-info
With Suricata installed, the real work begins with configuration. The main configuration file at /etc/suricata/suricata.yaml
is where we'll define how Suricata monitors your network. One of the most critical decisions you'll make is defining your HOME_NET variable – this tells Suricata which IP addresses belong to your internal network, allowing it to properly contextualize traffic and reduce false positives.
sudo nano /etc/suricata/suricata.yaml
The network variables section needs to accurately reflect your network topology. I've seen many deployments fail because administrators used the default settings without customization. Take the time to map out your network segments and configure these variables accordingly. If you have DMZ servers, guest networks, or other segmented areas, include them all in your HOME_NET definition.
vars:
address-groups:
HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
EXTERNAL_NET: "!$HOME_NET"
HTTP_SERVERS: "$HOME_NET"
SMTP_SERVERS: "$HOME_NET"
SQL_SERVERS: "$HOME_NET"
DNS_SERVERS: "$HOME_NET"
TELNET_SERVERS: "$HOME_NET"
AIM_SERVERS: "any"
port-groups:
HTTP_PORTS: "80,81,311,383,591,593,901,1220,1414,1741,1830,2301,2381,2809,3037,3128,3702,4343,4848,5250,6988,7000,7001,7144,7145,7510,7777,7779,8000,8008,8014,8028,8080,8085,8088,8090,8118,8123,8180,8181,8243,8280,8300,8800,8888,8899,9000,9060,9080,9090,9091,9443,9999,11371,34443,34444,41080,50002,55555"
SHELLCODE_PORTS: "!80"
ORACLE_PORTS: 1521
SSH_PORTS: 22
DNP3_PORTS: 20000
MODBUS_PORTS: 502
FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]"
FTP_PORTS: 21
GENEVE_PORTS: 6081
VXLAN_PORTS: 4789
TEREDO_PORTS: 3544
Interface configuration determines how Suricata captures packets from your network. For modern Linux systems, AF_PACKET provides the best balance of performance and compatibility. The configuration below sets up multi-threaded packet processing with optimizations that I've refined through numerous production deployments.
af-packet:
- interface: eth0
threads: auto
cluster-id: 99
cluster-type: cluster_flow
defrag: yes
tpacket-v3: yes
ring-size: 100000
buffer-size: 64535
use-emergency-flush: yes
Output configuration is where you define how Suricata logs its findings. The EVE JSON format has become the de facto standard for modern security tooling, and we'll configure it to capture comprehensive event data while maintaining performance. The key here is balancing detail with storage requirements – you want enough information for thorough investigations without overwhelming your storage capacity.
outputs:
- eve-log:
enabled: yes
filetype: syslog
facility: local5
level: Info
community-id: true
types:
- alert:
payload: yes
payload-buffer-size: 4kb
packet: yes
metadata:
app-layer: true
flow: true
rule:
metadata: true
raw: true
- http:
extended: yes
custom: [accept, accept-language, authorization]
- dns:
requests: yes
responses: yes
- tls:
extended: yes
ja3: yes
- files:
force-magic: yes
force-hash: [md5, sha256]
- smtp:
extended: yes
- ssh
- flow
- netflow
- stats:
totals: yes
threads: no
deltas: yes
Rule management represents the heart of Suricata's detection capabilities. The suricata-update tool streamlines the process of maintaining current rule sets, automatically downloading and managing rules from multiple sources. I recommend starting with the Emerging Threats Open rules as your baseline, then adding specialized rule sets based on your specific threat landscape.
# Update rules to latest version
sudo suricata-update
# List available rule sources
sudo suricata-update list-sources
# Enable Emerging Threats Open rules (free, comprehensive)
sudo suricata-update enable-source et/open
# Enable abuse.ch SSL blacklist
sudo suricata-update enable-source abuse.ch/sslbl
# Enable abuse.ch Feodo Tracker botnet C2 indicators
sudo suricata-update enable-source abuse.ch/feodotracker
# Apply the rule updates
sudo suricata-update
# Reload Suricata to apply new rules without downtime
sudo kill -USR2 $(pidof suricata)
Starting Suricata as a system service ensures it begins monitoring immediately after system boot and recovers automatically from any crashes. The systemd configuration below includes optimizations for production environments, including proper CPU affinity settings and memory limits.
# Configure Suricata startup options
sudo nano /etc/default/suricata
# Add these configuration lines:
# LISTENMODE=af-packet
# IFACE=eth0
# Enable and start Suricata service
sudo systemctl enable suricata
sudo systemctl start suricata
# Verify Suricata is running properly
sudo systemctl status suricata
sudo tail -f /var/log/suricata/suricata.log
Testing your Suricata installation confirms everything is working correctly. The EICAR test string provides a safe way to verify detection capabilities without introducing actual malware into your environment. When you see the alert in your logs, you'll know your detection pipeline is functioning properly.
# Test with a harmless but detectable pattern
curl -A "BlackSun" http://www.google.com/
# Check if alert was generated
sudo grep -i "BlackSun" /var/log/suricata/fast.log
# Monitor real-time alerts
sudo tail -f /var/log/suricata/fast.log
Setting up ntopng for network monitoring and visualization
Installing ntopng requires adding the official repository to ensure you get the latest features and security updates. The development team maintains packages specifically optimized for Ubuntu, and these include performance enhancements that aren't available in the distribution's default repositories.
# Add ntop repository for Ubuntu 22.04
wget https://packages.ntop.org/apt-stable/22.04/all/apt-ntop-stable.deb
sudo apt install ./apt-ntop-stable.deb
# For Ubuntu 24.04, use:
# wget https://packages.ntop.org/apt-stable/24.04/all/apt-ntop-stable.deb
# sudo apt install ./apt-ntop-stable.deb
# Update package lists and install ntopng
sudo apt update
sudo apt install -y ntopng nprobe
# Install additional dependencies for full functionality
sudo apt install -y pfring-dkms
Redis configuration needs attention before starting ntopng, as it serves as the primary data store for real-time metrics. The default Redis configuration works for small deployments, but production environments benefit from tuning that optimizes for ntopng's access patterns. These settings ensure data persistence while maintaining the performance needed for real-time analysis.
sudo nano /etc/redis/redis.conf
Configure Redis with these optimizations that I've found work well for most deployments. The key is balancing memory usage with persistence requirements – you want enough memory for active data while ensuring important information survives system restarts.
# Bind to localhost only for security
bind 127.0.0.1 ::1
# Disable protected mode since we're binding to localhost
protected-mode no
# Set a strong password
requirepass YourVeryStrongRedisPasswordHere2024!
# Memory management for ntopng workload
maxmemory 2gb
maxmemory-policy allkeys-lru
# Persistence settings optimized for ntopng
save 900 1
save 300 10
save 60 10000
# Enable AOF for better durability
appendonly yes
appendfsync everysec
# Restart Redis with new configuration
sudo systemctl restart redis-server
sudo systemctl enable redis-server
Creating ntopng's main configuration file establishes how it monitors your network and presents data. Unlike Suricata's monolithic configuration file, ntopng uses a simpler format that's easier to manage. The configuration below provides a solid foundation that you can expand based on your specific requirements.
sudo mkdir -p /etc/ntopng
sudo nano /etc/ntopng/ntopng.conf
Each line in the configuration represents a command-line parameter that ntopng would normally receive. This approach makes it easy to test changes interactively before making them permanent. The configuration below sets up ntopng with HTTPS enabled, proper network segmentation, and integration with our Redis instance.
# Network interface to monitor (adjust to your interface name)
-i=eth0
# Additional interface for Suricata syslog data
-i=syslog://127.0.0.1:5140
# Web interface ports
-w=3000
-W=3443
# Data directory for persistent storage
-d=/var/lib/ntopng
# PID file location
-G=/var/run/ntopng.pid
# Local networks (must match Suricata's HOME_NET)
-m="192.168.0.0/16,10.0.0.0/8,172.16.0.0/12"
# Redis configuration with password
-r=127.0.0.1:6379:YourVeryStrongRedisPasswordHere2024!@0
# MySQL flow export for long-term storage
-F="mysql;localhost;ntopng;flows;ntopng;StrongPasswordHere123!"
# DNS resolution mode (1 = resolve host names)
-n=1
# Disable localhost traffic analysis
-x=0
# Community ID for flow correlation with Suricata
--community-id
# Enable geolocation services
--disable-login=1
SSL certificate generation adds essential security to ntopng's web interface. While self-signed certificates work for internal deployments, I recommend using Let's Encrypt for any system that might be accessed from outside your immediate network. The process below creates a self-signed certificate suitable for initial testing.
# Generate self-signed certificate for HTTPS
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout /etc/ssl/private/ntopng.key \
-out /etc/ssl/certs/ntopng.crt \
-subj "/C=US/ST=State/L=City/O=Organization/CN=ntopng.local"
# Set proper permissions
sudo chmod 400 /etc/ssl/private/ntopng.key
sudo chmod 444 /etc/ssl/certs/ntopng.crt
# Update ntopng configuration to use certificates
echo "-c=/etc/ssl/certs/ntopng.crt" | sudo tee -a /etc/ntopng/ntopng.conf
echo "-k=/etc/ssl/private/ntopng.key" | sudo tee -a /etc/ntopng/ntopng.conf
Starting ntopng and configuring it as a system service ensures continuous monitoring. The systemd unit file manages the service lifecycle, automatically restarting it if it crashes and ensuring it starts at boot time.
# Start ntopng service
sudo systemctl start ntopng
sudo systemctl enable ntopng
# Verify ntopng is running
sudo systemctl status ntopng
sudo netstat -tlnp | grep -E "3000|3443"
# Check logs for any issues
sudo journalctl -u ntopng -f
Initial web interface configuration happens when you first access ntopng through your browser. Navigate to https://your-server-ip:3443
and log in with the default credentials (admin/admin). The system will immediately prompt you to change this password – choose something strong and unique, as this interface provides deep visibility into your network traffic.
The first configuration steps in the web interface establish how ntopng processes and presents data. Navigate to Settings > Preferences and configure your time zone, data retention policies, and alerting preferences. Pay particular attention to the "Local Networks" configuration – this should match your Suricata HOME_NET definition exactly to ensure consistent traffic classification across both tools.
Integrating Suricata with ntopng for comprehensive threat detection
The integration between Suricata and ntopng transforms two powerful tools into a unified threat detection platform. The key to successful integration lies in configuring proper data flow between the systems while maintaining the performance characteristics needed for real-time analysis. Through years of deployment experience, I've found that syslog-based integration provides the best balance of reliability and performance for most environments.
Configuring Suricata to send EVE logs via syslog requires modifying the output configuration we established earlier. The syslog facility provides a standardized method for inter-process communication that both tools understand natively, eliminating the need for custom parsing logic or intermediate processing steps.
# Configure rsyslog to forward Suricata events
sudo nano /etc/rsyslog.d/30-suricata.conf
Add these rsyslog rules to establish the forwarding pipeline. The configuration below captures Suricata's EVE output and forwards it to ntopng's syslog collector while maintaining a local copy for backup and troubleshooting purposes.
# Load required modules
module(load="imuxsock")
module(load="imfile")
# Monitor Suricata EVE JSON log if using file output
input(type="imfile"
File="/var/log/suricata/eve.json"
Tag="suricata"
Severity="info"
Facility="local5"
PersistStateInterval="1"
freshStartTail="on")
# Forward Suricata events to ntopng
if $syslogfacility-text == 'local5' then {
*.* @@127.0.0.1:5140
stop
}
# Also keep a local copy for debugging
local5.* /var/log/suricata-syslog.log
# Restart rsyslog to apply changes
sudo systemctl restart rsyslog
# Verify syslog forwarding is working
logger -p local5.info "Test Suricata message"
sudo tcpdump -i lo -n port 5140 -A
Configuring ntopng to receive syslog data completes the integration pipeline. The companion interface feature in ntopng allows it to correlate Suricata alerts with network flows, providing context that neither tool could achieve alone. This correlation happens automatically when both tools process the same traffic with properly synchronized timestamps.
# Update ntopng configuration to ensure syslog interface is configured
sudo nano /etc/ntopng/ntopng.conf
# Verify these lines are present:
# -i=syslog://127.0.0.1:5140
# --community-id
The Community ID feature deserves special attention as it enables flow correlation between tools. This standardized flow hashing algorithm ensures that both Suricata and ntopng generate identical identifiers for the same network flows, enabling automatic correlation of alerts with traffic patterns. When configured correctly, clicking on a Suricata alert in ntopng immediately shows the associated network flows, packet details, and behavioral analysis.
Testing the integration confirms data is flowing correctly between systems. Generate some test traffic that triggers Suricata rules, then verify the alerts appear in ntopng's interface with proper correlation to flow data.
# Generate test alert traffic
curl -A "TestSuricataAgent" http://testmynids.org/uid/index.html
# Verify alert appears in Suricata
sudo tail -f /var/log/suricata/fast.log
# Check ntopng is receiving events (via web interface)
# Navigate to Alerts > Alert Explorer in ntopng
# You should see Suricata alerts appearing in real-time
Fine-tuning the integration involves adjusting various parameters to optimize for your specific environment. The correlation window determines how long ntopng waits to match alerts with flows – too short and you'll miss correlations, too long and you'll consume excessive memory. Through experimentation, I've found that a 60-second window works well for most networks, though high-volume environments might need shorter windows to manage memory usage.
Advanced integration features unlock additional capabilities that enhance your threat detection arsenal. For instance, ntopng can enrich Suricata alerts with geographic information, reputation scores, and behavioral anomaly indicators. This enrichment happens automatically once you've configured the appropriate data sources in ntopng.
Configuring rule sets and threat intelligence feeds
Effective threat detection relies on comprehensive and current rule sets that reflect the evolving threat landscape. While Suricata ships with basic rules, the real power comes from leveraging community and commercial rule sources that security researchers worldwide continuously update. Managing these rules effectively requires a systematic approach that balances detection coverage with performance impact.
The Emerging Threats (ET) Open ruleset provides an excellent foundation with over 50,000 rules covering everything from malware communications to vulnerability exploitation attempts. These rules undergo constant refinement based on real-world threat intelligence, making them invaluable for detecting current attack campaigns. The suricata-update tool handles the complexity of downloading, merging, and managing these rules automatically.
# Configure suricata-update with additional sources
sudo suricata-update enable-source et/open
sudo suricata-update enable-source oisf/trafficid
sudo suricata-update enable-source abuse.ch/sslbl
sudo suricata-update enable-source abuse.ch/feodotracker
sudo suricata-update enable-source tgreen/hunting
# Update all enabled sources
sudo suricata-update
# Configure automatic rule updates via cron
sudo crontab -e
# Add this line for daily updates at 2 AM:
# 0 2 * * * /usr/bin/suricata-update && systemctl reload suricata
Custom rule creation addresses threats specific to your environment that generic rules might miss. Writing effective rules requires understanding your network's normal behavior patterns and the specific threats you face. The example below demonstrates creating rules for detecting suspicious activities unique to your infrastructure.
# Create custom rules file
sudo nano /var/lib/suricata/rules/local.rules
Add custom rules tailored to your environment. These examples show different rule types that detect various suspicious activities while maintaining reasonable performance overhead.
# Detect potential data exfiltration via DNS
alert dns any any -> any any (msg:"CUSTOM Possible DNS Exfiltration - Large Query"; dns.query; content:"|00 01|"; depth:2; offset:2; byte_test:1,>,30,0,relative; threshold:type limit, track by_src, count 1, seconds 60; classtype:policy-violation; sid:1000001; rev:1;)
# Detect suspicious PowerShell downloads
alert http any any -> $HOME_NET any (msg:"CUSTOM Suspicious PowerShell Download"; flow:established,to_client; content:"powershell"; http_uri; content:"IEX"; http_response_body; content:"WebClient"; http_response_body; classtype:trojan-activity; sid:1000002; rev:1;)
# Detect repeated failed SSH attempts
alert tcp any any -> $HOME_NET 22 (msg:"CUSTOM SSH Brute Force Attempt"; flow:to_server,established; content:"SSH-"; depth:4; detection_filter:track by_src, count 5, seconds 60; classtype:attempted-admin; sid:1000003; rev:1;)
# Detect cryptocurrency mining pools
alert tls any any -> any any (msg:"CUSTOM Possible Crypto Mining Pool Connection"; tls.sni; content:"pool."; content:"mining"; distance:0; within:20; classtype:policy-violation; sid:1000004; rev:1;)
Managing rule performance requires careful attention to which rules are active and how they're configured. Not all rules are suitable for every environment – a rule detecting industrial control system attacks provides no value in a standard corporate network while consuming processing resources. The process below helps identify and disable unnecessary rules.
# Analyze rule performance impact
sudo suricata --engine-analysis
# Review rules by performance impact
sudo grep "Rule matches" /var/log/suricata/suricata.log | sort -k6 -n
# Disable specific problematic rules
sudo suricata-update --disable-sid 2027758
# Disable entire rule categories
sudo suricata-update --disable-conf /etc/suricata/disable.conf
Create a disable configuration file for systematic rule management:
sudo nano /etc/suricata/disable.conf
Add rules or groups to disable based on your environment:
# Disable specific SIDs that cause false positives
2027758
2027759
# Disable entire groups not relevant to our environment
group:emerging-scada.rules
group:emerging-games.rules
# Disable all rules with specific metadata
metadata:"performance_impact" "significant"
Integrating threat intelligence feeds into ntopng complements Suricata's rule-based detection with reputation-based blocking. These feeds provide constantly updated lists of known malicious IPs, domains, and URLs that ntopng can use for immediate detection and optional blocking.
# Configure ntopng threat intelligence feeds via the web interface
# Navigate to Settings > Categories
# Add custom category lists:
# Malware domains
# URL: https://urlhaus.abuse.ch/downloads/hostfile/
# Category: Malware
# Update frequency: Daily
# Botnet C2 servers
# URL: https://feodotracker.abuse.ch/downloads/ipblocklist.txt
# Category: Botnet
# Update frequency: Hourly
# Configure automatic updates
# Settings > Preferences > Updates
# Enable "Automatically update lists"
# Set update frequency to "Every 6 hours"
Correlation between rule-based and reputation-based detection multiplies the effectiveness of both approaches. When Suricata detects suspicious behavior to a destination that ntopng identifies as malicious based on threat intelligence, the combined signal provides high confidence that you're observing actual malicious activity rather than a false positive.
Setting up logging and alerting mechanisms
Comprehensive logging forms the foundation of effective incident response and threat hunting. Both Suricata and ntopng generate extensive logs, but without proper configuration and management, these logs can quickly become overwhelming or, worse, disappear when you need them most. The logging architecture we'll implement provides both real-time visibility and long-term retention while maintaining system performance.
Configuring Suricata's EVE logging for optimal detail requires balancing completeness with performance. The configuration below captures essential security events while avoiding the overhead of logging every packet that crosses your network. Each log type serves a specific purpose in your security operations workflow.
sudo nano /etc/suricata/suricata.yaml
Update the outputs section with comprehensive logging that captures security-relevant events:
outputs:
- eve-log:
enabled: yes
filetype: regular
filename: eve.json
rotate-interval: day
community-id: true
types:
- alert:
payload: yes
payload-buffer-size: 4kb
packet: yes
http-body: yes
tagged-packets: yes
metadata:
app-layer: true
flow: true
- anomaly:
enabled: yes
types:
decode: yes
stream: yes
applayer: yes
- http:
extended: yes
custom: [authorization, server, set-cookie, warning, connection]
dump-all-headers: both
- dns:
enabled: yes
requests: yes
responses: yes
formats: [detailed, grouped]
types: [query, answer]
- tls:
extended: yes
ja3: yes
ja3s: yes
session-resumption: yes
custom: [subject, issuer, serial, fingerprint, sni, certificate, chain]
- files:
force-magic: yes
force-hash: [md5, sha1, sha256]
- smtp:
extended: yes
custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
- ssh:
enabled: yes
- nfs:
enabled: yes
- smb:
enabled: yes
- rdp:
enabled: yes
- flow:
enabled: yes
Log rotation prevents disk space exhaustion while maintaining forensic capabilities. The configuration below implements daily rotation with compression, keeping 90 days of logs for investigation purposes. This retention period aligns with most compliance requirements while remaining manageable from a storage perspective.
# Configure logrotate for Suricata
sudo nano /etc/logrotate.d/suricata
/var/log/suricata/*.log /var/log/suricata/*.json {
daily
rotate 90
compress
delaycompress
missingok
notifempty
create 640 suricata suricata
sharedscripts
postrotate
systemctl reload suricata 2>/dev/null || true
endscript
}
Setting up centralized syslog collection enables integration with existing SIEM platforms while maintaining local copies for immediate analysis. The configuration below sends critical alerts to a central collector while keeping detailed logs locally.
# Configure rsyslog for remote forwarding
sudo nano /etc/rsyslog.d/40-suricata-forward.conf
# Forward critical alerts to SIEM
if $programname == 'suricata' and $msg contains 'priority:1' then {
@@siem.company.local:514
}
# Forward all security events to backup collector
if $programname == 'suricata' then {
@@backup-log.company.local:514
}
Configuring ntopng alerting ensures you're notified of critical security events in real-time. The web interface provides extensive options for alert configuration, but the key is focusing on high-value alerts that require immediate attention rather than drowning in noise.
Navigate to the ntopng web interface and configure alerts through Settings > Alerts. Focus on these critical alert categories that typically indicate real security incidents:
-- Custom alert script for ntopng (place in /usr/share/ntopng/scripts/lua/modules/alerts/)
-- This example detects potential data exfiltration
local alerts_api = require("alerts_api")
local alert_consts = require("alert_consts")
local script = {
default_enabled = true,
default_value = {
threshold = 1073741824, -- 1GB threshold
time_window = 3600 -- 1 hour window
}
}
function script.hooks.min(params)
local host = params.entity_info
local bytes_sent = host["bytes.sent"]
if bytes_sent > params.threshold then
alerts_api.trigger(params.alert_entity, {
alert_type = alert_consts.alert_types.alert_data_exfiltration,
alert_severity = alert_consts.alert_severities.error,
alert_json = {
bytes_sent = bytes_sent,
threshold = params.threshold
}
})
end
end
return script
Email notification configuration ensures critical alerts reach the right people immediately. Configure SMTP settings in ntopng through Settings > Preferences > Alerts, providing your mail server details and recipient addresses. Test the configuration thoroughly to ensure alerts arrive reliably during actual incidents.
# Test email alerting from command line
echo "Test alert from ntopng" | mail -s "ntopng Test Alert" security-team@company.com
# Monitor ntopng alert processing
sudo journalctl -u ntopng -f | grep -i alert
Webhook integration enables automated response to security events. Modern security operations rely on automation to handle the volume of events, and webhooks provide the integration point for security orchestration platforms. Configure webhooks in ntopng through Settings > Notifications > Endpoints.
{
"webhook_url": "https://your-soar-platform.com/api/webhook",
"method": "POST",
"headers": {
"Authorization": "Bearer YOUR_API_TOKEN",
"Content-Type": "application/json"
},
"alert_format": {
"timestamp": "@timestamp",
"severity": "@alert_severity",
"type": "@alert_type",
"host": "@host",
"description": "@description",
"json": "@json"
}
}
Advanced configuration for enhanced detection
Performance tuning transforms a basic installation into a production-ready security sensor capable of handling enterprise traffic loads. The optimizations we'll implement focus on maximizing packet processing capacity while maintaining detection accuracy. Through careful tuning, I've seen deployments scale from handling hundreds of megabits to multiple gigabits of traffic on the same hardware.
CPU affinity configuration ensures packet processing threads aren't interrupted by system tasks, dramatically reducing packet drops under load. The configuration below dedicates specific CPU cores to packet processing while reserving others for management tasks. This separation prevents resource contention that leads to missed detections.
# Identify CPU topology and NUMA nodes
lscpu --extended
lstopo --logical
# Configure CPU isolation at boot
sudo nano /etc/default/grub
# Add to GRUB_CMDLINE_LINUX:
# isolcpus=2-7,10-15 nohz_full=2-7,10-15 rcu_nocbs=2-7,10-15
Update Suricata's threading configuration to leverage isolated CPUs:
threading:
set-cpu-affinity: yes
cpu-affinity:
- management-cpu-set:
cpu: [ 0, 1, 8, 9 ]
- worker-cpu-set:
cpu: [ 2-7, 10-15 ]
mode: "exclusive"
prio:
low: [ 2, 3, 10, 11 ]
medium: [ 4, 5, 12, 13 ]
high: [ 6, 7, 14, 15 ]
default: "high"
detect-thread-ratio: 1.0
Memory pool optimization reduces allocation overhead during packet processing. Pre-allocating memory pools eliminates the performance penalty of dynamic allocation during high-traffic periods. The settings below establish memory pools sized for typical enterprise traffic patterns.
# Optimize memory pools in suricata.yaml
stream:
memcap: 8gb
checksum-validation: no
prealloc-sessions: 500000
midstream: false
bypass: yes
flow:
memcap: 1gb
hash-size: 1048576
prealloc: 500000
emergency-recovery: 30
defrag:
memcap: 512mb
hash-size: 65536
prealloc: yes
Custom detection rules for your environment address threats that generic rules miss. Every network has unique characteristics – whether it's custom applications, specific user behaviors, or industry-specific threats. The rules below demonstrate patterns I've developed for common enterprise scenarios.
# Create environment-specific rules
sudo nano /var/lib/suricata/rules/custom-enterprise.rules
# Detect lateral movement via SMB
alert smb any any -> $HOME_NET 445 (msg:"CUSTOM Potential Lateral Movement - Multiple SMB Connections"; flow:to_server,established; detection_filter:track by_src, count 10, seconds 60; classtype:suspicious-activity; sid:5000001; rev:1;)
# Detect suspicious database queries
alert tcp $HOME_NET any -> $SQL_SERVERS 1433 (msg:"CUSTOM Suspicious SQL Query - Bulk Data Request"; flow:to_server,established; content:"|01 01|"; offset:0; depth:2; pcre:"/SELECT.*FROM.*WHERE.*1\s*=\s*1/i"; classtype:attempted-recon; sid:5000002; rev:1;)
# Detect unauthorized cloud storage usage
alert tls $HOME_NET any -> any 443 (msg:"CUSTOM Unauthorized Cloud Storage Access"; tls.sni; pcre:"/\.(dropbox|mega|mediafire|wetransfer)\.com$/"; classtype:policy-violation; sid:5000003; rev:1;)
# Detect potential ransomware behavior
alert smb any any -> $HOME_NET any (msg:"CUSTOM Possible Ransomware - Multiple File Modifications"; flow:established; smb.ntlmssp_domain; content:!"WORKGROUP"; detection_filter:track by_src, count 100, seconds 10; classtype:trojan-activity; priority:1; sid:5000004; rev:1;)
Behavioral analysis configuration in ntopng complements signature-based detection with anomaly identification. The machine learning algorithms in ntopng establish baselines of normal behavior, then alert when significant deviations occur. This capability catches zero-day attacks and insider threats that bypass traditional signatures.
Configure behavioral checks through the ntopng web interface under Settings > Preferences > Alerts. Enable these critical behavioral checks that have proven effective in production environments:
- Flows flood detector: Detects scanning and DoS attempts
- TCP SYN flood detector: Identifies SYN flood attacks
- Flows anomaly detector: Uses statistical analysis to identify unusual traffic patterns
- Blacklisted flows: Alerts on communication with known malicious hosts
- Device protocol anomaly: Detects when devices communicate using unexpected protocols
Advanced ntopng scripting enables custom detection logic tailored to your environment. The Lua scripting interface provides access to flow data and host information, enabling complex detection scenarios that would be impossible with static rules alone.
-- Custom ntopng detection script for suspicious encryption
-- Place in /usr/share/ntopng/scripts/lua/modules/flow_checks/
local checks = require("checks")
local script = {
default_enabled = true,
l7_proto = {"TLS", "QUIC"},
gui = {
i18n_title = "flow_checks.suspicious_entropy",
i18n_description = "Detects high-entropy traffic indicating encryption or compression"
}
}
function script.hooks.protocolDetected(now, flow)
local payload_ratio = flow["payload_ratio"]
local bytes_ratio = flow["bytes_ratio"]
if payload_ratio > 0.95 and bytes_ratio > 100 then
local alert_severity = alerts_api.alert_severities.warning
local alert_type = alerts_api.alert_types.suspicious_entropy
alerts_api.trigger(alert_type, alert_severity, {
flow = flow,
payload_ratio = payload_ratio,
bytes_ratio = bytes_ratio
})
end
end
return script
Monitoring and maintenance procedures
Ongoing monitoring ensures your security infrastructure continues operating at peak effectiveness. The metrics and procedures below establish a maintenance rhythm that keeps your detection capabilities sharp while preventing system degradation. I've learned that proactive maintenance prevents most operational issues before they impact security visibility.
Performance monitoring focuses on key metrics that indicate system health and detection effectiveness. These metrics serve as early warning signs of problems that could compromise your security posture. Regular review of these indicators helps identify trends before they become critical issues.
# Create monitoring script for automated checks
sudo nano /usr/local/bin/security-monitor.sh
#!/bin/bash
# Security monitoring health check script
LOG_FILE="/var/log/security-monitor.log"
ALERT_EMAIL="security-team@company.com"
# Function to check Suricata health
check_suricata() {
local drops=$(tail -n 100 /var/log/suricata/stats.log | \
jq -r '.stats.capture.kernel_drops' | \
awk '{sum+=$1} END {print sum}')
local packets=$(tail -n 100 /var/log/suricata/stats.log | \
jq -r '.stats.capture.kernel_packets' | \
awk '{sum+=$1} END {print sum}')
if [ "$packets" -gt 0 ]; then
local drop_rate=$(echo "scale=2; $drops * 100 / $packets" | bc)
if (( $(echo "$drop_rate > 1.0" | bc -l) )); then
echo "[WARNING] High packet drop rate: ${drop_rate}%" | tee -a "$LOG_FILE"
echo "Packet drops detected on $(hostname): ${drop_rate}%" | \
mail -s "Suricata Performance Alert" "$ALERT_EMAIL"
fi
fi
}
# Function to check ntopng health
check_ntopng() {
local response=$(curl -s -o /dev/null -w "%{http_code}" https://localhost:3443/lua/rest/v2/get/ntopng/health.lua)
if [ "$response" != "200" ]; then
echo "[ERROR] ntopng health check failed: HTTP $response" | tee -a "$LOG_FILE"
systemctl restart ntopng
fi
}
# Function to check disk space
check_disk_space() {
local usage=$(df /var/log | tail -1 | awk '{print $5}' | sed 's/%//')
if [ "$usage" -gt 80 ]; then
echo "[WARNING] Log partition usage high: ${usage}%" | tee -a "$LOG_FILE"
# Compress old logs
find /var/log/suricata -name "*.json" -mtime +7 -exec gzip {} \;
fi
}
# Main execution
echo "[$(date)] Starting security monitoring check" | tee -a "$LOG_FILE"
check_suricata
check_ntopng
check_disk_space
echo "[$(date)] Security monitoring check complete" | tee -a "$LOG_FILE"
# Make script executable and schedule it
sudo chmod +x /usr/local/bin/security-monitor.sh
# Add to crontab for every 5 minutes execution
sudo crontab -e
# */5 * * * * /usr/local/bin/security-monitor.sh
Rule update automation keeps your detection capabilities current with emerging threats. The security landscape changes daily, and manual rule updates inevitably fall behind. The automation below ensures your rules stay current while maintaining stability through testing and validation.
# Create comprehensive rule update script
sudo nano /usr/local/bin/update-security-rules.sh
#!/bin/bash
# Automated rule update with validation
BACKUP_DIR="/backup/suricata/rules-$(date +%Y%m%d)"
LOG_FILE="/var/log/rule-updates.log"
# Backup current rules
backup_rules() {
echo "[$(date)] Backing up current rules" >> "$LOG_FILE"
mkdir -p "$BACKUP_DIR"
cp -r /var/lib/suricata/rules "$BACKUP_DIR/"
if [ $? -eq 0 ]; then
echo "[$(date)] Backup successful" >> "$LOG_FILE"
return 0
else
echo "[$(date)] Backup failed - aborting update" >> "$LOG_FILE"
return 1
fi
}
# Update rules
update_rules() {
echo "[$(date)] Updating rules" >> "$LOG_FILE"
suricata-update \
--suricata /usr/bin/suricata \
--suricata-conf /etc/suricata/suricata.yaml \
--no-test 2>&1 | tee -a "$LOG_FILE"
return $?
}
# Test new rules
test_rules() {
echo "[$(date)] Testing new rules" >> "$LOG_FILE"
suricata -T -c /etc/suricata/suricata.yaml -S /var/lib/suricata/rules/suricata.rules
if [ $? -eq 0 ]; then
echo "[$(date)] Rule test passed" >> "$LOG_FILE"
return 0
else
echo "[$(date)] Rule test failed - reverting" >> "$LOG_FILE"
cp -r "$BACKUP_DIR/rules" /var/lib/suricata/
return 1
fi
}
# Apply new rules
apply_rules() {
echo "[$(date)] Applying new rules" >> "$LOG_FILE"
systemctl reload suricata
sleep 10
if systemctl is-active --quiet suricata; then
echo "[$(date)] Rules successfully applied" >> "$LOG_FILE"
# Clean old backups (keep last 7)
find /backup/suricata -name "rules-*" -mtime +7 -exec rm -rf {} \;
return 0
else
echo "[$(date)] Suricata failed after rule update - reverting" >> "$LOG_FILE"
cp -r "$BACKUP_DIR/rules" /var/lib/suricata/
systemctl restart suricata
return 1
fi
}
# Main execution
if backup_rules; then
if update_rules; then
if test_rules; then
apply_rules
fi
fi
fi
Database maintenance prevents performance degradation as historical data accumulates. Both Suricata and ntopng generate substantial amounts of data, and without proper maintenance, query performance degrades to the point where investigations become impractical.
-- MySQL/MariaDB maintenance for ntopng flow data
-- Run weekly to maintain performance
-- Analyze table statistics
ANALYZE TABLE ntopng.flows;
-- Optimize table structure
OPTIMIZE TABLE ntopng.flows;
-- Archive old flow data (keep 90 days)
CREATE TABLE IF NOT EXISTS ntopng.flows_archive LIKE ntopng.flows;
INSERT INTO ntopng.flows_archive
SELECT * FROM ntopng.flows
WHERE last_switched < DATE_SUB(NOW(), INTERVAL 90 DAY);
DELETE FROM ntopng.flows
WHERE last_switched < DATE_SUB(NOW(), INTERVAL 90 DAY);
-- Update indexes for better query performance
ALTER TABLE ntopng.flows
ADD INDEX idx_time_src_dst (first_switched, src_addr, dst_addr),
ADD INDEX idx_bytes (in_bytes, out_bytes);
Certificate renewal automation ensures HTTPS communications remain secure. Expired certificates not only trigger security warnings but can break integrations between systems. The automation below handles certificate renewal for both self-signed and Let's Encrypt certificates.
# Automated certificate renewal script
sudo nano /usr/local/bin/renew-certificates.sh
#!/bin/bash
# Certificate renewal automation
CERT_DIR="/etc/ssl/certs"
KEY_DIR="/etc/ssl/private"
DAYS_BEFORE_EXPIRY=30
check_cert_expiry() {
local cert_file=$1
local days_left=$(openssl x509 -in "$cert_file" -noout -dates | \
grep notAfter | \
cut -d= -f2 | \
xargs -I {} date -d {} +%s | \
awk '{print int(($1 - systime()) / 86400)}')
if [ "$days_left" -lt "$DAYS_BEFORE_EXPIRY" ]; then
return 0 # Certificate needs renewal
else
return 1 # Certificate still valid
fi
}
renew_self_signed() {
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout "$KEY_DIR/ntopng-new.key" \
-out "$CERT_DIR/ntopng-new.crt" \
-subj "/C=US/ST=State/L=City/O=Organization/CN=ntopng.local"
if [ $? -eq 0 ]; then
mv "$KEY_DIR/ntopng-new.key" "$KEY_DIR/ntopng.key"
mv "$CERT_DIR/ntopng-new.crt" "$CERT_DIR/ntopng.crt"
systemctl restart ntopng
echo "[$(date)] Certificate renewed successfully"
fi
}
# Check and renew if necessary
if check_cert_expiry "$CERT_DIR/ntopng.crt"; then
echo "[$(date)] Certificate expiring soon, renewing..."
renew_self_signed
fi
Troubleshooting common integration issues
Even well-planned deployments encounter issues, and knowing how to diagnose and resolve them quickly minimizes security gaps. The troubleshooting procedures below address the most common problems I've encountered across numerous deployments. Understanding these patterns helps you recognize and resolve issues before they impact your security posture.
Packet drop issues manifest as missed alerts and incomplete flow records. When Suricata or ntopng drops packets, you lose visibility into potential threats. The diagnostic process below helps identify the root cause and implement appropriate fixes.
# Diagnose packet drops
sudo suricata-sc -c "dump-counters" | grep -E "kernel_drops|errors"
# Check interface statistics
ethtool -S eth0 | grep -E "drop|error|overflow"
# Monitor ring buffer usage
cat /proc/net/pf_ring/info
# Analyze CPU utilization by thread
top -H -p $(pgrep suricata)
Common solutions for packet drops include increasing ring buffer sizes, optimizing CPU affinity, and reducing rule complexity. The approach depends on where bottlenecks occur in your processing pipeline.
# Increase ring buffers
sudo ethtool -G eth0 rx 4096
# Update AF_PACKET ring size in suricata.yaml
# ring-size: 500000
# Disable expensive rules temporarily to isolate issues
sudo suricata-update --disable-sid 2027758,2027759
sudo systemctl reload suricata
Memory exhaustion causes both tools to become unresponsive or crash. Identifying memory leaks and optimizing memory usage prevents these critical failures. The diagnostic steps below help pinpoint memory issues before they cause outages.
# Monitor memory usage over time
while true; do
date
ps aux | grep -E "suricata|ntopng" | grep -v grep
free -h
sleep 60
done | tee memory-monitor.log
# Check for memory leaks
valgrind --leak-check=full --show-leak-kinds=all \
suricata -c /etc/suricata/suricata.yaml -i eth0
# Analyze memory allocation patterns
pmap -x $(pgrep suricata)
Integration connectivity problems prevent Suricata alerts from appearing in ntopng. When this integration breaks, you lose the correlation capabilities that make the combined system powerful. The troubleshooting process below systematically identifies and resolves connectivity issues.
# Verify syslog forwarding
tcpdump -i lo -n port 5140 -c 10
# Check rsyslog processing
journalctl -u rsyslog -f
# Test manual syslog injection
logger -p local5.info "Test Suricata Alert"
# Verify ntopng is listening
netstat -anup | grep 5140
False positive storms can overwhelm your security team and hide real threats. While some false positives are inevitable, excessive rates indicate configuration issues that need addressing. The process below helps identify and suppress problematic rules while maintaining security coverage.
# Identify top false positive generators
awk '/alert/ {print $3}' /var/log/suricata/fast.log | \
sort | uniq -c | sort -rn | head -20
# Analyze specific rule for false positives
grep "sid:2019401" /var/log/suricata/fast.log | \
awk '{print $1, $2}' | sort | uniq -c
# Create threshold configuration to reduce noise
sudo nano /etc/suricata/threshold.config
Add threshold rules to manage noisy alerts:
# Suppress specific rule for internal scanner
suppress gen_id 1, sig_id 2019401, track by_src, ip 192.168.100.50
# Rate limit noisy rules
threshold gen_id 1, sig_id 2027758, type limit, track by_dst, count 1, seconds 3600
# Threshold for services generating expected traffic
threshold gen_id 1, sig_id 2100498, type threshold, track by_src, count 10, seconds 60
Performance degradation over time affects many deployments as data accumulates and rules multiply. Regular performance audits identify degradation before it impacts security operations. The assessment below provides a comprehensive health check.
# Comprehensive performance audit script
sudo nano /usr/local/bin/security-performance-audit.sh
#!/bin/bash
echo "Security Stack Performance Audit - $(date)"
echo "========================================="
# Suricata performance metrics
echo -e "\nSuricata Performance:"
echo "---------------------"
tail -n 1 /var/log/suricata/stats.log | jq -r '
"Packet Rate: \(.capture.kernel_packets / 60) pps",
"Drop Rate: \((.capture.kernel_drops / .capture.kernel_packets) * 100)%",
"Decoder Events: \(.decoder.invalid)",
"Flow Memory: \(.flow.memuse / 1048576) MB"'
# ntopng performance metrics
echo -e "\nntopng Performance:"
echo "-------------------"
curl -s -u admin:admin "https://localhost:3443/lua/rest/v2/get/system/stats.lua" | jq -r '
"Active Hosts: \(.num_hosts)",
"Active Flows: \(.num_flows)",
"Packets Processed: \(.packets)",
"Throughput: \(.throughput_bps) bps"'
# System resource usage
echo -e "\nSystem Resources:"
echo "-----------------"
top -bn1 | grep -E "Cpu|Mem|suricata|ntopng"
# Disk I/O statistics
echo -e "\nDisk I/O:"
echo "---------"
iostat -x 1 2 | tail -n +4
Best practices and conclusion
After years of deploying and maintaining network security monitoring infrastructure, certain patterns consistently lead to success. The practices below represent hard-won lessons that transform good deployments into great ones. Following these guidelines helps you avoid common pitfalls while maximizing the value of your security investment.
Start with comprehensive documentation of your deployment. Document not just the configuration but the reasoning behind each decision. When you need to troubleshoot issues at 2 AM or onboard new team members, this documentation becomes invaluable. Include network diagrams showing tap points, processing flows, and integration points between systems.
Implement gradual rollouts for all changes. Whether adding new rules, updating software, or modifying configurations, test changes in a controlled manner. Start with passive monitoring before enabling blocking, test new rules on subset of traffic before full deployment, and always maintain rollback procedures. This approach prevents minor issues from becoming major outages.
Maintain separate development and production environments when possible. A development environment allows you to test updates, new rules, and configuration changes without risking production stability. Even a small virtual environment provides valuable testing capabilities that prevent production issues.
Regular training and knowledge sharing within your team multiplies the effectiveness of your tools. Schedule monthly review sessions where team members share interesting findings, discuss false positives, and refine detection strategies. This collaborative approach improves detection quality while building team expertise.
Establish clear escalation procedures for security events. Define what constitutes a critical alert requiring immediate response versus events that can wait for normal business hours. Document response procedures for common scenarios, including who to contact, what data to collect, and how to contain threats.
Performance baselines deserve continuous attention. Regularly review system performance metrics and update your baselines as network patterns evolve. What worked for last year's traffic patterns might not suit today's cloud-heavy, encrypted traffic flows. Use these baselines to identify gradual degradation before it becomes critical.
Integration with existing security tools multiplies the value of your deployment. Whether feeding alerts to a SIEM, enriching events with threat intelligence, or triggering automated responses through SOAR platforms, these integrations transform raw alerts into actionable intelligence. Plan these integrations from the beginning rather than treating them as afterthoughts.
The combination of Suricata and ntopng creates a security monitoring platform that provides both broad coverage and deep inspection capabilities. Suricata's signature-based detection catches known threats with high precision, while ntopng's behavioral analysis and visualization capabilities provide the context needed to understand and respond to complex attacks. Together, they form a detection system that adapts to evolving threats while maintaining the performance needed for modern network speeds.
Success with this deployment comes from treating it as a living system rather than a static installation. Networks evolve, threats change, and your monitoring must adapt accordingly. Regular maintenance, continuous tuning, and constant learning keep your detection capabilities sharp. The investment you make in properly deploying and maintaining these tools pays dividends through improved security posture, faster incident response, and deeper understanding of your network's behavior.
Remember that tools alone don't provide security – they enable security when properly deployed, configured, and operated by skilled professionals. The configurations and procedures in this guide provide a solid foundation, but your unique environment will require customization and continuous refinement. Embrace this iterative process, learn from both successes and failures, and build a security monitoring capability that grows stronger over time.