Using VPC Flow Logs to Monitor and Optimize Your Network Traffic

Amazon Virtual Private Cloud (VPC) Flow Logs is a feature provided by AWS (Amazon Web Services) that allows you to capture information about the IP traffic going to and from network interfaces in your VPC. It provides detailed information about the traffic within your VPC, including information such as source and destination IP addresses, ports, protocol, and the amount of data transferred.

VPC Flow logs capture information that provide visibility into network traffic for monitoring, troubleshooting security and performance, and optimizing resource utilization. The information captured includes: 

• Source and destination IP addresses and ports
• Protocol (TCP, UDP, ICMP)
• Number of packets and bytes
• Start and end time of flows
• Whether the traffic was accepted or rejected

VPC Flow logs do not capture payload data or real-time traffic; data is aggregated over 10 seconds or longer intervals before being published to CloudWatch Logs, S3, or Kinesis Firehose.

Decoding and Understanding AWS VPC Flow Logs

It’s essential to understand what each field captured in a VPC flow log means, in order to analyze and troubleshoot your VPC network traffic effectively. Here’s a quick overview:

Default Fields

VPC flow logs contain the following default fields:

• version: The VPC flow logs version used
• account-id: The AWS account ID that owns the interface
• interface-id: The ID of the elastic network interface (ENI) where traffic is recorded
• srcaddr: The source IP address
• dstaddr: The destination IP address
• srcport: The source port number
• dstport: The destination port number
• protocol: The protocol (TCP, UDP, ICMP, and etc.)
• packets: The number of packets in the flow
• bytes: The number of bytes in the flow
• start: The time the flow started
• end: The time the flow ended
• action: Whether traffic was accepted or rejected
• log-status: Logging completion status

Additional Fields

You can enable further metadata like:

• VPC ID: ID of the VPC containing the interface
• Subnet ID: ID of the subnet containing the interface
• Instance ID: ID of the EC2 instance containing the interface
• TCP flags: TCP flags seen on the flow (SYN, ACK, etc.)
• Type: Type of flow (IPv4 or IPv6)
• Packet loss: Percentage of lost packets

These fields can provide more context for the recorded flows.

Analyzing the comprehensive flow log data enables deep network visibility for monitoring, troubleshooting, and optimization.

Key Benefits of Enabling VPC Flow Logs

VPC Flow Logs can help with: 

• Network monitoring: Analyze traffic patterns, identify anomalous flows, and optimize performance.
• Troubleshooting: Diagnose connectivity and latency issues and ACL misconfigurations.
• Security analysis: Detect suspicious traffic, unauthorized access attempts, and DDoS patterns. 
• Cost optimization: Identify unnecessary traffic across regions/VPCs to reduce data transfer costs.

Enabling VPC Flow Logs

You can enable flow logs at the VPC, subnet, or network interface level. Flow logs can be published to CloudWatch Logs, S3, or Kinesis Firehose.

To enable, you specify the resources to monitor and the destination via AWS Console, CLI, or API calls. You can also configure sampling rate, aggregation interval, and filters.

Using the AWS Management Console

• Open the Amazon VPC Console and navigate to “Flow Logs” in the left sidebar
• Click “Create Flow Log”
• Specify the resources to monitor by choosing VPCs, Subnets, or Network Interfaces
• Select the destination for logs – CloudWatch Logs, S3 bucket, or Kinesis Data Firehose
• Set additional parameters like log format, filters, or aggregation interval
• Click “Create”

Using AWS CLI

Below is a sample CLI command to create a flow log:

aws ec2 create-flow-logs –resource-ids vpc-111bbb222 –resource-type VPC –traffic-type ALL –log-destination-type cloud-watch-logs

Using CloudFormation Template

You can specify a flow log resource in your AWS CloudFormation template as below:

Resources:

  VPCFlowLog: 

    Type: AWS::EC2::FlowLog

    Properties:

      ResourceType: VPC

      ResourceId: vpc-12abc34df5

      TrafficType: ALL

      LogDestinationType: cloud-watch-logs

      LogDestination: !Ref LogsGroup

      LogFormat: ${version} ${vpc-id}

This allows flow log creation to be automated as part of stack deployments.

The key is choosing the right resources for monitoring, log destinations, and traffic filters based on your specific use cases.

Enhanced Visibility with Coralogix

Sending your VPC Flow Logs to Coralogix provides greater visibility with:

• Log parsing and indexing to make logs searchable and easier to analyze
• Anomaly detection to spot abnormal traffic patterns 
• Root cause analysis to correlate network issues with other log types inside a VPC
• Traffic pattern analysis over time to understand growth and changes

Coralogix integrates seamlessly with AWS services to ingest flow logs data and provides a powerful query language and visualization tools for insights.

You can read more about how to set up VPC flow logs delivery to S3 for streaming via lambda to Coralogix here.

Coralogix security offering provides a comprehensive set of out-of-the-box detections & alerts for AWS VPC Flow Logs. Some of the notable ones include:

Alerts

Here are some examples of the out-of-the-box alerts customers can deploy to their platform with 1-click.

• Non-Standard DNS Protocol Activity
  • Scope: This alert triggers when DNS queries from internal hosts use TCP port 53 instead of the standard UDP. While TCP over port 53 can be legitimate for zone transfers or to accommodate large packets, it can also indicate suspicious activities such as tunneling or attempts to bypass security controls.
  • Potential Impact: Non-standard DNS protocol activity could indicate malicious behavior, such as data exfiltration or command and control communication. It may lead to unauthorized access to resources, data breaches, or the introduction of malware into the network.
  • Remediation Recommendation:
    • Investigate: Immediately investigate the source and destination of the DNS traffic using TCP port 53. Determine if it aligns with legitimate activities such as zone transfers or if it indicates malicious behavior.
    • Monitor: Continuously monitor DNS traffic for similar anomalies. Implement alerting mechanisms to promptly identify and respond to any deviations from normal behavior.
    • Restrict Traffic: Enforce strict egress filtering rules to only allow DNS traffic over UDP port 53. Consider implementing DNS security solutions that can detect and mitigate suspicious DNS activities.
    • Educate Users: Educate internal users about the risks associated with non-standard DNS protocol activities and the importance of adhering to security policies and best practices.

• Incoming Connections Over Remote Service Ports
  • Scope: This alert triggers when an incoming SSH (port 22) or RDP (port 3389) connection is accepted. These ports enable remote command line and desktop access, posing a security risk if unauthorized.
  • Potential Impact: Unauthorized incoming SSH or RDP connections can lead to unauthorized access to sensitive systems, data breaches, or the execution of malicious activities such as privilege escalation or data theft.
  • Remediation Recommendation:
    • Review Access Controls: Review and strengthen access controls to restrict SSH and RDP access only to authorized users and IP addresses.
    • Implement Multi-Factor Authentication (MFA): Require the use of MFA for SSH and RDP authentication to add an extra layer of security.
    • Monitor and Audit: Continuously monitor incoming SSH and RDP connections for unauthorized access attempts. Enable detailed logging and auditing to track user activities and detect suspicious behavior.
    • Network Segmentation: Implement network segmentation to isolate critical systems from less secure environments, reducing the impact of potential security breaches.
    • Update and Patch Systems: Ensure that SSH and RDP servers are regularly updated with the latest security patches to mitigate known vulnerabilities.

• Excessive Inbound ICMP Traffic
  • Scope: This alert triggers whenever an IP address sends many ICMP ping requests to hosts within a VPC within a short interval.
  • Potential Impact: Excessive inbound ICMP traffic can potentially be used in ICMP flood attacks, causing network congestion, service disruption, or denial of service (DoS) to the affected hosts.
  • Remediation Recommendation:
    • Implement Rate Limiting: Configure rate-limiting mechanisms to limit the number of ICMP requests allowed per second, mitigating the impact of ICMP flood attacks.
    • Block Suspicious IPs: Identify and block IP addresses generating excessive ICMP traffic, especially those exhibiting patterns indicative of malicious intent.
    • Network Intrusion Detection/Prevention Systems (NIDS/NIPS): Deploy NIDS/NIPS solutions to detect and block ICMP flood attacks in real-time.
    • Traffic Shaping: Implement traffic shaping techniques to prioritize legitimate traffic and mitigate the impact of ICMP floods on network performance.
    • Update Security Policies: Regularly review and update security policies to adapt to emerging threats and vulnerabilities, including ICMP flood attacks.

Dashboards

Using Coralogix Security, out-of-box insights can be deployed to understand various metrics. Some of the examples include – 

• Total Accepted/Rejected traffic
• Total Egress traffic
• Distribution of top source countries over time
• Top source IPs
• Top destination IPs etc.

Optimization Based on Insights

By leveraging the deeper visibility provided by VPC Flow Logs analytics with Coralogix, you can:  

• Detect and troubleshoot sub-optimal network routes faster
• Identify unnecessary cross-region/VPC traffic to minimize costs
• Inform security group and NACL rules optimization
• Plan for scaling needs proactively based on growth trends

VPC Flow Logs + Coralogix provides the comprehensive visibility and analytics capabilities needed to master observability across cloud-native applications and dynamic AWS network environments.

Conclusion

As we have seen, VPC flow logs are essential to the Security Monitoring arsenal. By sending AWS VPC flow logs to Coralogix, organizations can unlock further value through integrated log analytics and security capabilities. Coralogix enables faster searching through parsing and indexing and customizable dashboards and alerts tailored to the environment that provide security teams early warning across operations, security, and business metrics extracted from the flow logs.

Learn more about Coralogix security offerings today.