Geneva, 24 March 2011

Cisco experiences of IP traffic flow measurement and billing with NetFlow

Benoit Claise,Distinguished Engineer, Cisco

ITU-T Workshop onIP Traffic Flow Measurement

(Geneva, Switzerland, 24 March 2011)

What is NetFlow?

Cache

CollectorNetFlow Records export

Over UDP or SCTP

Traffic

What is NetFlow?

NetFlow is used for traffic monitoring, security analysis, capacity planning and billing

Billing is just a few % of our customers, mainly for charge back within enterprise network (not between service providers)

NetFlow = a exporting protocol: NetFlow v5, 7, 8, 9 (RFC3954), and IPFIX (RFC5101/RFC5102)

NetFlow v9 and IPFIX work with a template based mechanismAdvantage: extensibility, just need to add new Information Element

NetFlow = a metering process: Flexible NetFlowAdvantages: cache and export content flexibility

User selection of flow keysUser definition of the records

Flexible NetFlow: Potential Key FieldsIPv4IP (Source or Destination)

Payload Size

Prefix (Source or Destination)

Packet Section (Header)

Mask (Source or Destination)

Packet Section (Payload)

Minimum-Mask (Source or Destination)

TTL

ProtocolOptions bitmap

Fragmentation Flags

Version

Fragmentation Offset

Precedence

Identification DSCP

Header Length TOS

Total Length

Interface Input

Output

Flow Sampler ID

Direction

Source MAC address

Destination MAC address

Dot1q VLAN

Source VLAN

Layer 2

IPv6

IP (Source or Destination)

Payload Size

Prefix (Source or Destination)

Packet Section (Header)

Mask (Source or Destination)

Packet Section (Payload)

Minimum-Mask (Source or Destination)

DSCP

Protocol Extension Headers

Traffic Class Hop-Limit

Flow Label Length

Option Header Next-header

Header Length Version

Payload Length

Dest VLAN

Dot1q priority

MulticastReplication Factor*

RPF Check Drop*

Is-Multicast

Flexible NetFlow: Potential Key Fields

Input VRF Name

BGP Next Hop

IGP Next Hop

src or dest AS

Peer AS

Traffic Index

Forwarding Status

Routing TransportDestination Port TCP Flag: ACK

Source Port TCP Flag: CWR

ICMP Code TCP Flag: ECE

ICMP Type TCP Flag: FIN

IGMP Type* TCP Flag: PSH

TCP ACK Number TCP Flag: RST

TCP Header Length TCP Flag: SYN

TCP Sequence Number TCP Flag: URG

TCP Window-Size UDP Message Length

TCP Source Port UDP Source Port

TCP Destination Port UDP Destination Port

TCP Urgent Pointer

Application

Application ID*

*: IPv4 Flow only

Flexible NetFlow: Potential Non-Key Fields

Plus any of the potential “key” fields: will be the value from the first packet in the flow

Counters

Bytes

Bytes Long

Bytes Square Sum

Bytes Square Sum Long

Packets

Packets Long

Timestamp

sysUpTime First Packet

sysUpTime First Packet

IPv4

Total Length Minimum (*)

Total Length Maximum (*)

TTL Minimum

TTL Maximum

(*) IPV4_TOTAL_LEN_MIN, IPV4_TOTAL_LEN_MAX (**)IP_LENGTH_TOTAL_MIN, IP_LENGTH_TOTAL_MAX

IPv4 and IPv6

Total Length Minimum (**)

Total Length Maximum (**)

Performance

Limited Resources in RouterDon’t enable all flow keysThe routers still have to route packets

NetFlow for Billing: Experience

Packet Size Standard Deviation σ f

Mean Packet Size µf

#P

ackets

Nf

Estimation Accuracy (PLT_NZIX1, S24D00, Cisco, f=5%

Issue: Can we use Sampled NetFlow for billing?

Huge amount of data, must sometimes deal with sampled NetFlow, i.e. 1 out of N packets, depending on the platformPacket Sampling for Flow Accounting: Challenges and Limitations, Tanja Zseby, Thomas Hirsch, Benoit Claise, PAM 2008

Issue: Can we use Sampled NetFlow for billing?

Square sum of bytes available in Flexible NetFlowNot used in practice, not even by the collectors!Customers afraid of legal issues with sampling along with a billing service

AS=196 E-BGP

ISP 1$5.00 per 100 MB

traffic index = 1traffic index = 1PrefixPrefix Traffic-indexTraffic-index

Forwarding Information Base

prefix twoprefix two traffic index = 2 traffic index = 2 prefix oneprefix one traffic index = 1 traffic index = 1

Destination Sensitive Billing Proposal(many years ago)

AS=193

Customer

E-BGP

AS 192

ISP 2$7.00 per 100 MB

1. BGP routing updates

2. Go through a table-map statement

3. table-map calls a route-map

4. route-map’s criteria: if criteria 1 -> traffic-index = 1

if criteria 2 -> traffic-index = 2

prefix oneprefix one traffic index = 1 traffic index = 1 Accounting

I-BGP

BGP Policy Accounting Principles

Allows to classify packets based onIP access lists, BGP community list

to characterize the exit points, where each exit point would set an specific community

BGP AS paths

The ISP

The Customer

Issue: What about the Returning Packets?

ISP 1$5.00 per 100 MB

ISP 2$7.00 per 100 MB

FTP Request

100 MB back100 MB back

Who should pay for the 100 MB back?

Destination Sensitive Billing requires also source lookup (Source Sensitive Billing)

Who should pay for the 100 MB back?

Destination Sensitive Billing requires also source lookup (Source Sensitive Billing)

The ISP

The Customer

Issue: What about the Returning Packets?

ISP 1$5.00 per 100 MB

ISP 2$7.00 per 100 MB

FTP Request

100 MB back100 MB back

Lookup:• On the outgoing packets

(on the packets coming back)

• On the source • Same selection criteria

Lookup:• On the outgoing packets

(on the packets coming back)

• On the source • Same selection criteria

The ISP

The Customerin Europe

Issue: BGP Asymmetry Problem

ISP 1 in Asia ISP 2 in US

FTP Request

100 MB back

Will charge the 10 Meg as if they were directly coming from the US!!!

Issue: BGP Asymmetry Problem

The source lookup is based on the route the router would take to reach the source!

Too Many Issues

Destination Sensitive Billing requires Source Sensitive BillingBGP asymmetry problemOnly the traffic following the BGP routes will be accounted

What if local policies outside of BGP?

Limited amount of buckets in the Destination Sensitive Billing

Doesn’t scale: too many entries

Performance issuesEntire NMS solution to be put in place

Destination Sensitive Billing

Conclusion/feedback from customers: too many issuesnot realistically deployable -> back to some sort of flat rate

Benoit’s concern:If we bill per AS-PATH and each AS get a piece of the pie, people will create new AS and try to attract trafficBad for the internet performance

Geneva, 24 March 2011

Cisco experiences of IP traffic flow measurement and billing with NetFlow

Benoit Claise,Distinguished Engineer, Cisco

ITU-T Workshop onIP Traffic Flow Measurement

(Geneva, Switzerland, 24 March 2011)