ciscolive brkapp-3003
DESCRIPTION
Troubleshooting SCETRANSCRIPT
BRKAPP-3003
Troubleshooting the Application ControlEngine (ACE)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 2
Core Message
To understand the architecture and flow management is to understanding how to troubleshoot the Application Control Engine
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 3
Session ObjectiveAt the End of the Session, You Will Be Able To: ACE Architecture
Understand the ACE architecture and connectivity through ACE
Verify software images, licenses and image recovery
Use the real-time “TCP-DUMP” command
Implement management traffic protection
Understand access-list list on ACE
Flow Management Understand the difference between “L4” and “L7” processing
Check for possible asymmetric flows
Understand high availability from the show commands
Provide layer 7 troubleshooting
Ability to monitor performance and troubleshoot resources
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 4
ACE ArchitectureDiscuss the Architecture
Functions of control plane and data plan
Common debugging commands
Packet Capturing on and logging
Traffic Forwarding on ACE
Management Traffic Protection
Flow ManagementConnection Handling on ACE
Health Monitoring on ACE
High Availability on ACE
Layer 7 Troubleshooting and Performance
Session Agenda
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 5
ACE Architecture
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 6
ACE Module Hardware Architecture
SwitchFabricInterface
16G
DaughterCard 1
DaughterCard 2
8G
8G
SSLCrypto
10G
NetworkProcessor 1
NetworkProcessor 2
10G10G
2GClassificationDistributionEngine(CDE)
Consoleport
SupConnect
100M
ControlPlane
Data Plane
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 7
Network Processer Micro-Engines
Receive + Fastpath (+ Transmit)
IP Reassembly + Timers + Syslog
Inbound Connection Manager
Outbound Connection Manager
Connection Close Management
TCP
HTTP
Application fixups
SSL Record Layer
Static and user-configurable REGEX
TCP Normalization + FixUps
Rx FastPath
FastPath
FastPath
FastPath
FastPath
IP FragTimers
ICM
OCM CCM TCP HTTP
HTTP SSLRecord
RegEx FixUpsTCP Norm.
CPU “Xscale”
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 8
Separation of Data and Management Traffic
Control-PathDevice controlConfiguration manager (CLI, XML API, SSH, …)Server health monitoring (native probes, TCL scripts)SYSLOGs, SNMP, …ARP, DHCP relayHigh-Availability
Control path and data path run on separate processors
Data-PathConnection management
TCP termination
Access lists
SSL Offload
Regular expression matching
Load Balancing & forwarding
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 9
Traffic to the ACE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 10
Traffic Flow to the CDE
The ACE has no native ports. The Switch Fabric Interface forwards packets to the CDE
A packet comes in over the Switch Fabric Interface marked with the VLAN and the L2 information
This is the TenGigabit Ethernet link (Te?/1, where ? is the slot number)
Packets entering/leaving the ACE will traverse this link, using VLAN tagging to indicate the VLAN
The CDE (Classification and Distribution Engine) fills out the IMPH header and forwards traffic to the appropriate blade subsystem (e.g., CP, NP1, NP2…)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 11
Traffic Flow to the CDE - Continue
The CDE hashes incoming packets to be forwarded to either NP1 or NP2 based on the following:
TCP/UDP – hash of source/destination port
Non TCP/UDP IP – hash on source/destination IP address
NonIP – hash on source/destination L2 MAC
All forwarding is done on the NPs. These constitute two parallel forwarding paths which maintain independent connection state and forward independently
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 12
Traffic to the ACE – Control Plane
Traffic directed to the ACE itself is received on the Control Plan. Useful statistics are:
“Show netio stats” and “show fifo stats” counting traffic into/out of the CP
“Show netio clients” show applications which have registered to receive traffic from the CDE
There are a number of useful context-specific commands These are for ACE terminated traffic, and do not measure traffic forwarded by the ACE!! show ip traffic
show [protocol] statisticsprotocol can be arp, udp, tcp, icmp
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 13
ACE in a Nutshell
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 14
ACE in a Nutshell Cisco ACE provides many advanced load balancing features
Features consisting of interface and application security, server offload, and application load balancing
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 15
Virtual Context Setup
Virtual contexts are virtualized ACEs. Each virtual context has independent configuration and dedicated resources assigned. One context can pull resources from another
Every ACE device contains a special virtual context called "Admin“. It is recommended that you create separate virtual contexts for load balancing
The capacity of each ACE virtual context is determined by its resource class
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 16
Common Debugging
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 17
Common Debugging VIP is not responding
when trying to connect
If you try ping the VIP you must configure loadbalance vip icmp-reply
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 18
Common Debugging
Show command on the Catalyst 6500 Supervisor
show versionshow clockshow moduleshow powershow asic slot <n>show interface TenGigabitEthernet <n>/1show interface TenGigabitEthernet <n>/1 trunkshow svclc vlan-group[no] power enable <module>
Make sure the module status is OK
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 19
Common Debugging Show command available on the ACE
show versionshow clockshow ft group statusshow ip int brshow int vlan <n>show arpshow service-policyshow serverfarmshow rservershow probeshow connshow statshow ip trafficshow resource usageshow np 1 me-stats “-s norm”show np 1 me-stats “-s norm –M1”
System Information
L2, L3
Performance,ResourcesDebuggingFlows
L4, L7
This provides the DELTA
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 20
Looking at the Normalization counters
Shows the DROP counters in Fast Path and TCPswitch/Admin# show np 1 me-stats "-s norm" | i Drop[Drops] L2 invalid DA mac: 0[Drops] L4 port is zero 0[Drops] TCP invalid conn miss flags: 0[Drops] TCP invalid flags: 0[Drops] TCP urgent pointer denied: 0[Drops] TCP non-zero reserved field: 0[Drops] TCP syn data denied: 0[Drops] TCP non-syn options on syn: 0[Drops] TCP syn options on non-syn: 0[Drops] TCP no of denied options: 0[Drops] TCP option length wrong: 0[Drops] fp TCP invalid ack in syn-ack: 0[Drops] fp TCP invalid ack for syn-ack: 0[Drops] fp TCP ack past seq: 0[Drops] fp TCP window left edge: 0[Drops] fp TCP window right edge: 0[Drops] fp TCP data past FIN: 0[Drops] fp TCP FIN has wrong seq: 0[Drops] fp TCP RST has wrong seq: 0[Drops] fp TCP RST has wrong ack: 0[Drops] fp TCP ack > FIN_ACK exp: 0[Drops] fp TCP exceeded MSS: 18
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 21
Show Module from the Catalyst 6500 Supervisor
cat6k#show modMod Ports Card Type Model Serial No.--- ----- -------------------------------------- ------------------ -----------1 1 Application Control Engine 10G Module ACE10-6500-K9 SAD093508042 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45 SAD04450L445 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD08300D5L
Mod MAC addresses Hw Fw Sw Status--- ---------------------------------- ------ ------------ ------------ -------1 0001.0002.0003 to 0001.0002.000a 0.504 8.6(0.252-En 3.0(0)A1(2) Ok2 00d0.d32e.1b42 to 00d0.d32e.1b71 1.5 5.4(2) 8.5(0.46)RFW Ok5 000f.f7be.b17c to 000f.f7be.b17f 4.0 8.1(3) 12.2(PP_R31_ Ok
Mod Sub-Module Model Serial Hw Status---- --------------------------- ------------------ ----------- ------- -------5 Policy Feature Card 3 WS-F6K-PFC3BXL SAD083006N2 1.3 Ok5 MSFC3 Daughterboard WS-SUP720 SAD082905VE 2.1 Ok
Mod Online Diag Status---- -------------------1 Pass2 Pass5 Pass
Module status shows OK
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 22
Verifying Version and Licenses
switch/Admin# show versionCisco Application Control Software (ACSW)Softwareloader: Version 12.2[118]system: Version A2(1.0) [build 3.0(0)A2(1.0)system image file: [LCP] disk0:c6ace-t1k9-mz.A2_1.bininstalled license: ACE-08G-LIC ACE-VIRT-250 ACE-SSL-15K-K9
HardwareCisco ACE (slot: 1)cpu info:number of cpu(s): 2cpu type: SiBytecpu: 0, model: SiByte SB1 V0.2, speed: 700 MHzcpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz
Installed Licenses
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 23
Available System Memory and Uptime
switch/Admin# show version – Continuation of output[...]
memory info:total: 958004 kB, free: 335372 kBshared: 0 kB, buffers: 3540 kB, cached 0 kB
cf info:filesystem: /dev/cftotal: 499744 kB, used: 447136 kB, available: 52608 kB
last boot reason: reload command by adminconfiguration register: 0x1ACE kernel uptime is 7 days 23 hours 42 minute(s) 25 second(s)
Displays ACE module uptime
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 24
What Licenses Are Installed
View the current licenses installedswitch/Admin# show licenseACE-250CTX-08G-SSL-15K.lic:SERVER this_host ANYVENDOR ciscoINCREMENT ACE-08G-LIC cisco 1.0 permanent 1 \
VENDOR_STRING=<count>1</count> HOSTID=ANY \
NOTICE="<LicFileID>20060523161924670</LicFileID><LicLineID>1</LicLineID> <PAK></PAK>" SIGN=76DA7526434AINCREMENT ACE-SSL-15K-K9 cisco 1.0 permanent 1 \
VENDOR_STRING=<count>1</count> HOSTID=ANY \
NOTICE="<LicFileID>20060523161924670</LicFileID><LicLineID>7</LicLineID>
<PAK></PAK>" SIGN=1077701CF92CINCREMENT ACE-VIRT-250 cisco 1.0 permanent 1 \
VENDOR_STRING=<count>1</count> HOSTID=ANY \
Shows the license file installed
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 25
Installing New Licenses on ACE
Copy license file to disk0: on the ACEswitch/Admin# dir disk0:
636 Apr 17 16:04:04 2007 ACE-250CTX-08G-SSL-20K.lic236 Apr 17 16:06:54 2007 ACE-16G-LIC.lic
switch/Admin# license ?install Install the licenseuninstall Uninstall the licenseupdate Update existing license
License commands available on the ACE
License commands available on the ACE. Reload only required when increase throughput on the ACE10switch/Admin# license install disk0:ACE-16G-LIC.licInstalling license... done
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 26
ACE File System
Use the dir command to view directory listing for filesswitch/Admin# dir ?core: Directory or filenamedisk0: Directory or filenameimage: Directory or filenameprobe: Directory or filenamevolatile: Directory or filename
The internal File system is mapped as below/mnt/cf - Image:
Also the following compressed file systems are used/TN-HOME = disk0:/TN-CONFIG = Startup config /TN-LOGFILE = Internal Storage for audit logs/TN-CERTKEY-STORAGE : internal storage for Cert and Keys/TN-COREFILE = core:
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 27
ACE File System
Load debug plug-in to access ACE file systemStartup configuration located at /mnt/cf/TN-CONFIG
ACE will generate / fix any missing or corrupted file systems during boot
When to use the format command?
If you receive the following error
Warning!! This will erase everything in the compact flash including startup configs for all the contexts and reboot the system!!
switch/Admin# write memory ERROR!config filesystem is not mounted on compact flash
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 28
Working with Core Files
If ACE creates a core file you can locate the files from the core directory
All cores files are stored in dir core: (core names are self explanatory)
switch/Admin# dir core:99756 Apr 5 17:57:05 2007 ixp2_crash.txt13047 Apr 5 17:56:59 2007 loadBalance_core_log.tar.g
Ixpx_crash.txt will have some details on the core dump
If it is a kernel crash , then a file named crashinfo wil be available in core:
Show version wil show last reload reason
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 29
Invoke Context
To display the context running configuration information from the Admin context, use the invoke context command
invoke context context_name show running-config
switch/Admin# invoke context BreakingPoint show running-config write memory Generating configuration....
switch/Admin# invoke context Exchange2010 show running-config | include 192.168.1.1Generating configuration....ip address 192.168.1.11ip address 192.168.1.12
alias 192.168.1.1 255.255.255.0Sandbox-Pod2-ACE20-1/Admin#
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 30
System Logging
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 31
Logging Features Each virtual context generates logs independently and
send to specified destinationsSyslog server, console, telnet/ssh, buffer, flash, supervisor, SNMP, NAT
Rate limiting of syslog messages is recommended. Never log to the console using level 7
ACE will log connection setup/teardown at the connection speed
Access-List deny entries are logged
Use the terminal monitor command to display log message when not using console
Useful commands to troubleshoot syslogging:show logging statistics
show logging queue | last
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 32
Basic Configuration to Enable Logging Enable logging on the ACE
logging enablelogging monitor 7no logging message 111008no logging message 111009logging timestampdo terminal monitor
It is recommended to disable or changing the severity level of some syslog messages. Use logging message syslog_id [level severity_level] command
To enable the logging of connection setup and teardown messages, use the logging fastpath command. Use the logging rate-limit to limit the rate at which the ACE generates messages in the syslog
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 33
Real-Time “TCP Dump”
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 34
Real-Time “TCP Dump”
Supportability and analysis of load balanced traffic is a major requirement in today's load balanced environment
ACE can capture real-time packet information for the network traffic that passes through the ACE
The attributes of the packet capture are defined byan ACL
The ACE buffers the captured packets, and you can copy the buffered contents to a file in flash memory on the ACE or to a remote server
User can also display the captured packet information on your console or terminal; capture can also be exported to capture to Ethereal
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 35
Real-Time “TCP Dump”
To enable the packet capture on ACE use the capturecommand
capture c1 interface vlan 211 access-list FILTER bufsize 64
Buffer in Kbytes(can be circular)
Pre-defined ACL toidentify relevant traffic
Interface to applycapture
One capture session per context
Capture triggered at flow setup
Capture configured on client interface where flow is received
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 36
Real-Time “TCP Dump” ACE can capture traffic based on a configured access-list
and interface
Follow the following procedure to capture traffic on ACE:1. Specify an ACL
2. Capture on an interface or globally
access-list FILTER line 10 extended permit tcp any any eq wwwcapture c1 interface vlan 211 access-list FILTER
Show capture status show status and buffer size
switch/Admin# show capture c1 status Capture session : c1 Buffer size : 64 KCircular : no Buffer usage : 1.00%Status : stopped
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 37
Real-Time “TCP Dump” Start the capture on the ACEswitch/Admin# capture c1 start23:40:37.236868 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 58: 172.16.11.190.443 > 209.165.201.11.1180: S 1389739009:1389739009(0) ack 617249474 win 17408 <mss 1460> (ttl 255, id 2401, len 44, bad cksum 0!)23:40:37.239102 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 54: 172.16.11.190.443 > 209.165.201.11.1180: . ack 71 win 17408 (ttl 255, id 2402, len 40, bad cksum 0!)switch/Admin# capture c1 stop
To copy the packet capture to disk0: use the copy capture
switch/Admin# copy capture c1 disk0: c1
Maximum buffer size is 5MB of data
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 38
Traffic Forwarding on ACE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 39
ACE Load Balancer Policy Lookup Order
There can be many features applied on a given interface, so feature lookup ordering is important
The feature lookup order followed by datapath in ACE is as follows:1. Access-control (permit or deny a
packet)
2. Management traffic
3. TCP normalization/connection parameters
4. Server load balancing
5. Fix-ups/application inspection
6. Source NAT
7. Destination NAT
The policy lookup order is implicit, irrespective of the order in which the user configures policies on the interface
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 40
ACE in Router Mode
IP subnets cannot overlap within a context but can across two contexts
Non-Load balanced traffic is routed. ACE needs to ARP for destination before forwarding packet
Client MAC ACE MAC
Client IP VIP
Random Port VIP Port
ACE MAC SelectedServer MAC
Client IP Server IP
Random Port Server Port
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 41
ACE in Bridge Mode
Non-Load balanced connection are bridged from client to server vlan
Client MAC ACE MAC
Client IP VIP
Random Port VIP Port
Client MAC SelectedServer MAC
Client IP Server IP
Random Port Server Port
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 42
Checking VLAN Configuration Show interface provides you with valuable information switch/Admin# show interface vlan 211vlan210 is up
Hardware type is VLANMAC address is 00:16:36:fc:b3:36Virtual MAC address is 00:0b:fc:fe:1b:02Mode : routedIP address is 172.16.10.21 netmask is 255.255.255.0FT status is activeDescription:WAN SideMTU: 1500 bytesLast cleared: neverAlias IP address is 172.16.10.23 netmask is 255.255.255.0Peer IP address is 172.16.10.22 Peer IP netmask is 255.255.255.0Assigned on the physical port, up on the physical port
499707 unicast packets input, 155702918 bytes1485258 multicast, 5407 broadcast0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops497610 unicast packets output, 46804782 bytes6 multicast, 8201 broadcast0 output errors, 0 ignored
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 43
MAC Addresses
Virtual MAC (VMAC) is used for the alias IP, VIP address Alias IP and Virtual IP (VIP) are associated with a
VMAC only if high availability is configured Active context responds to ARPs for alias IP with
VMAC One unique VMAC per FT Group 00:0b:fc:fe:1b:XX
(XX=FT group number in hex) Packets destined to the VMAC are blocked on standby
context
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 44
MAC Addresses
The VMAC is a function of ft-group-id. Therefore different cards must have different ft-group-ids Use the show interface internal iftable to locate the
VMAC Each ACE supports 1,024 shared VLANs, and uses
only one bank of MAC addresses randomly selected at boot time ACEs may select the same address bank so avoid this
conflict use the shared-vlan-hostid command
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 45
Key Things to Know About ARP on ACE
For unicast packets, if the destination MAC is unknown ACE will drop the packet, instead of flooding it
So IP-address-to-MAC mapping and outgoing interface needs to happen first
ARP entries are populated as follows:With ARP requests
Learning through incoming ARP requests
Gratuitous ARP packets
Layer 2 mode:No MAC learning
So ARP is the way to learn IP to MAC and interface mapping
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 46
How to Read the ARP Table
Each virtual context maintains its own ARP tableswitch/Admin# show arpContext Exchange=======================================================================IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status=======================================================================172.16.11.1 00.00.0c.07.ac.0a vlan211 GATEWAY 226 87 sec up172.16.11.19 00.12.43.dc.83.bb vlan211 INTERFACE LOCAL _ up172.16.11.190 00.12.43.dc.83.bb vlan211 VSERVER LOCAL _ up192.168.1.1 00.0a.b8.66.60.85 vlan411 INTERFACE LOCAL _ up192.168.1.11 00.50.56.12.11.01 vlan411 RSERVER 230 87 sec up192.168.1.12 00.50.56.12.11.01 vlan411 RSERVER 229 87 sec up192.168.20.254 00.0a.b8.66.60.85 bvi2 INTERFACE LOCAL _ up==================================================================Total arp entries 11
ARP table shows the type of ARP entry from Gateway, Interface, VSERVER, RSERVER
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 47
Admin Context Resource Reservation
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 48
Admin Context Resource Reservation
If Admin context is not configured correctly admin could be starved of all resources
When configuring resource allocations in ACE, it is possible to allocate 100% of resources to non-Admin contexts, so that the Admin context is no longer reachable via ICMP, telnet, SNMP, etc
It also appears that in some cases this will cause FT between a pair of HA ACE modules to fail, and create an active/active situation
Highly recommended to put some safe guard in place to ensure that the Admin context always receives at least a small percentage of resources
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 49
Admin Context Resource Reservation Shows starved admin context
switch/Admin# show arpContext Admin =============================================================================IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status ============================================================================= 10.87.102.225 00.00.00.00.00.00 vlan621 GATEWAY - * 2 req up10.87.102.229 00.0b.fc.fe.1b.01 vlan621 ALIAS LOCAL _ up10.87.102.230 00.0a.b8.71.2f.ef vlan621 INTERFACE LOCAL _ up172.16.0.1 00.0a.b8.71.2f.ef vlan999 INTERFACE LOCAL _ up172.16.0.2 00.05.9a.3b.92.e9 vlan999 LEARNED 18 * 2 req up============================================================================= Total arp entries 5
switch/Admin# ping 10.87.102.225 Pinging 10.87.102.225 with timeout = 2, count = 5, size = 100 .... No response received from 10.87.102.225 within last 2 sec No response received from 10.87.102.225 within last 2 sec 2 packet sent, 0 responses received, 100% packet loss
Unable to reach ACEs default gateway
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 50
Admin Context Resource Reservation Shows starved resources and drops for throughputswitch/Admin# show resource usage context Admin
Allocation Resource Current Peak Min Max Denied -------------------------------------------------------------------------------Context: Admin conc-connections 9 9 0 0 0 mgmt-connections 2 12 0 0 0 proxy-connections 0 0 0 0 0 xlates 0 0 0 0 0 bandwidth 0 4715 0 0 3704068
throughput 0 4247 0 0 3704068 mgmt-traffic rate 0 468 0 125000000 0
connection rate 0 7 0 0 8 ssl-connections rate 0 0 0 0 0 mac-miss rate 0 1 0 0 0 inspect-conn rate 0 0 0 0 0 acl-memory 26816 26880 0 0 0 sticky 0 0 0 0 0 regexp 0 0 0 0 0 syslog buffer 1024 4096 0 1024 0 syslog rate 0 7 0 0 118
No resources reserved
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 51
Admin Context Resource Reservation
Shows heartbeats missed increasing. Heartbeats are not reaching the peer. Possibility for both Aces to go Active/Active
switch/Admin# sh ft stats HA Heartbeat Statistics------------------------Number of Heartbeats Sent : 1095573 Number of Heartbeats Received : 1095239 Number of Heartbeats Missed : 2987 Number of Unidirectional HB's Received : 2640 Number of HB Timeout Mismatches : 0 Num of Peer Up Events Sent : 1 Num of Peer Down Events Sent : 1 Successive HB's miss Intervals counter : 0 Successive Uni HB's recv counter : 0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 52
Admin Context Resource Reservation
Below shows the problem why ACE is starved of all resources
resource-class admin limit-resource all minimum 0.10 maximum equal-to-min
Suggest the following reserved resources for Adminresource-class Admin
limit-resource conc-connections min 5.00 max equal-to-min
limit-resource mgmt-connections min 5.00 max equal-to-min
limit-resource rate bandwidth min 5.00 max equal-to-min
limit-resource rate ssl-connections min 5.00 max equal-to-min
limit-resource rate mgmt-traffic min 5.00 max equal-to-min
limit-resource rate conc-connections min 5.00 max equal-to-min
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 53
Access-Control Lists
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 54
ACL Merge Process and Enhancements New ACL merge enhancements added to the ACE
ACL merge is responsible for merging all the features and generating a single merged list for an given interface. ACL compiler is responsible for programming the merged list into MTrie data structure – “Fast retrieval of data”
ACL memory usage has been optimized to better support incremental changes
The new implementation provides a consistent ACL memory usage during system bootup time and during incremental changes after the system comes up
This feature also provides an early detection of failure if the configuration needs more ACL resources than what system can support
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 55
View Total Action Nodes
Use the show np 1 access-list resource to view action nodes
switch/Admin# show np 1 access-list resource ACL Tree Statistics for Context ID: 3=======================================ACL memory max-limit: NoneACL memory guarantee: 0.00 %MTrie nodes(used/guaranteed/max-limit):
6 / 0 / 262143 (compressed) 2 / 0 / 21999 (uncompressed)
Leaf Head nodes (used/guaranteed/max-limit): 3 / 0 / 262143
Leaf Parameter nodes (used/guaranteed/max-limit): 7 / 0 / 524288
Policy action nodes used: 4memory consumed: 4696 bytes resource-limited 128 bytes other 4824 bytes total.min-guarantee: 0 bytes total.max-limit: 78610432 bytes total, 0 % consumed
The total policy action nodes counts for ACE:
ACE Module - 200k
ACE 4710 Appliance - 40k
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 56
Troubleshooting Secure Socket Layer (SSL)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 57
Troubleshooting SSL
Configuration of SSL on ACE is relatively simply. However if you experience issue how do you troubleshoot?
Make sure the certificate and key used in ssl-proxy are valid. Use the crypto verify commandswitch/Admin# crypto verify RSA2048.key RSA2048.certKeypair in RSA2048.key matches certificate in RSA2048.cert
Check the size and location of the key. Use the show crypto key commandswitch/Admin# show crypt key allFilename Bit Size Type-------- -------- ----RSA2048.key 2048 RSA
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 58
Troubleshooting SSL
Looking at the certificate details. Use the show crypto certificate command switch/Admin# show crypto certificate cisco-sample-certCertificate:
Data:Version: 3 (0x2)Serial Number:
ad:e4:e2:f1:50:b7:ce:bdSignature Algorithm: sha1WithRSAEncryptionIssuer: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TESTValidity
Not Before: Apr 3 09:50:55 2009 GMTNot After : Apr 1 09:50:55 2019 GMT
Subject: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TESTSubject Public Key Info:
Public Key Algorithm: rsaEncryptionRSA Public Key: (1024 bit)
Modulus (1024 bit):00:cf:a2:60:66:5b:ce:b6:38:6f:94:df:0d:1c:61:26:af:7a:05:49:ed:8d:93:3b
Exponent: 65537 (0x10001)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 59
Troubleshooting SSL – CRL Download Check to make sure you can download the CRLswitch/Admin(config-ssl-proxy)# do show crypto crl test2 detailtest2:URL: http://119.60.60.23/test.crlLast Downloaded (Cached): Sat Aug 8 16:14:24 2009 UTCTotal Number Of Download Attempts: 1Failed Download Attempts: 0Successful Loads: 1 Failed Loads: 0Hours since Last Load: 0 No IP Addr Resolutions: 0Host Timeouts: 0 Next Update Invalid: 0Next Update Expired: 0 Bad Signature: 0CRL Found-Failed to load: 0 File Not Found: 0Memory Outage failures: 0 Cache Limit failures: 0Conn failures: 0 Internal failures: 0Not Eligible for download: 3 HTTP Read failures: 0HTTP Write failures: 0
Looking for all best-effort CRLs in the system and their download status. Use the show crypto crl best-effort command
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 60
Advanced SSL Debugging This command provides the current crypto statistics
switch/Admin# sh np 1 me-stats "-s cryptoCrypto Statistics: (Current)------------------ARC4 operations: 376572 0TCP msgs received: 285260 0APP msgs received: 235151 0Nitrox messages forwarded to XScale: 381041 0SSL ctx allocated: 47758 0SSL ctx freed: 47758 0SSL received bytes: 61070430 0SSL transmitted bytes: 283256220 0SSL received application bytes: 7679113 0SSL transmitted application bytes: 275120867 0SSL received non-application bytes: 53391317 0SSL transmitted non-application bytes: 3292887 0Bulk flush operations: 95037 0ME records sent to XScale: 285808 0ME records received from XScale: 47723 0ME hw responses: 471516 0First segments received: 47400 0Handshake failure alert: 94 0CM close: 446 0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 61
Advanced SSL Debugging
The show stats crypto server command provides statistics of the SSL handshake
switch/Admin# show stats crypto server+---- Crypto server termination statistics -----++------- Crypto server alert statistics --------++--- Crypto server authentication statistics ---++------- Crypto server cipher statistics -------++------ Crypto server redirect statistics ------++---- Crypto server header insert statistics ---+
These statistics provide details of the SSL packets for example; which version client interacted with ACE, which cipher is used, whether re-handshake happened, whether session id reuse happened and which SSLalerts are received or sent by ACE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 62
Connection Handling on ACE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 63
Flow ManagementLevel of Flow Processing Type of Processing Feature of Function
Layer 3 and Layer 4 Balance of first packet Basic Load Balancing
Applies to TCP/UDP for layer 4 rules Source IP Sticky
Applies to all other IP protocols TCP/IP Normalization
Select server or farm based on source IP
Layer 7 TCP Splicing Terminate TCP Connection HTTP Layer 7 rules of first request (URL LB)
Buffer request, inspect, LB Cookie Sticky (Persistence)
Create Hardware Shortcut Generic TCP Payload Parsing
Layer 7 Re-proxy TCP Splicing + ability to parse subsequent HTTP requests within the same TCP
HTTP Layer 7 rules with HTTP 1.1 connections keepalive(“persistence rebalance”)
Layer 7 Full-Proxy Fully terminate clients connection SSL Offload
TCP re-use
HTTP 1.1 Pipelining
Protocol Inspection (FTP,SIP)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 64
Internal Mapping of TCP/UDP Flows
TCP and UDP Flows = 2 X Internal Half Flows
switch/Admin# show conn
conn-id np dir proto vlan source destination stat-------------+--+----+--------+-----+--------------------------+-------------------------------+---------+
9 1 In TCP 211 209.165.201.11:1867 172.16.11.190:80 ESTAB
6 1 Out TCP 411 192.168.1.11:80 209.165.201.11:1867 ESTAB
Client IP:port VIP Address
Server IP Returning half flow automatically created for both TCP and UDP flows
INIT, SYNACK,ESTAB, CLOSED
SYN_SEEN, SYN_SEEN,ESTAB, CLOSED
Non TCP shows as “--”
Use conn-id to track flow through ACE
Check the Network Processor
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 65
Troubleshooting Connections
Use the show stats connection command to show connections statistics.
Use the clear stats connection command to clear these counters
switch/Admin# show stats connection+------------------------------------------++------- Connection statistics ------------++------------------------------------------+Total Connections Created : 288232Total Connections Current : 2Total Connections Destroyed: 283404Total Connections Timed-out: 892Total Connections Failed : 3934
Note: ACE does not destroy connection. These are connections closed correctly!!!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 66
Troubleshooting Connections Use the show stats loadbalance command to view the
loadbalance statistics
To clear the load balance statistical information stored in the ACE buffer, use the clear stats loadbalance command switch/Admin# show stats loadbalance+------------------------------------------------------------++------- Loadbalance statistics ----------------------++------------------------------------------------------------+Total version mismatch : 0Total Layer4 decisions : 0Total Layer4 rejections : 0Total Layer7 decisions : 24Total Layer7 rejections : 0Total Layer4 LB policy misses : 0Total Layer7 LB policy misses : 0Total times rserver was unavailable : 0Total ACL denied : 0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 67
Troubleshooting VIPswitch/Admin# show service-policy client-vips detailStatus : ACTIVEDescription: ------------------------------------------Interface: vlan 211 service-policy: client-vipsclass: VIP-HTTPSVIP Address: Protocol: Port:172.16.11.190 tcp eq 443 loadbalance:L7 loadbalance policy: HTTPS-POLICYVIP Route Metric : 77VIP Route Advertise : DISABLEDVIP ICMP Reply : ENABLED-WHEN-ACTIVEVIP State: INSERVICEcurr conns : 22 , hit count : 22dropped conns : 0 client pkt count : 0 , client byte count: 0 server pkt count : 0 , server byte count: 0 max-conn-limit : 0 , drop-count : 0 conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 L7 Loadbalance policy : HTTPS-POLICYclass/match : class-defaultLB action :
primary serverfarm: backend-sslbackup serverfarm : -
hit count : 22dropped conns : 0
First command you show use view connection to VIP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 68
Troubleshooting Serverfarmswitch/Admin# show serverfarm HTTPS-FARM detailserverfarm : HTTPS-FARM, type: HOSTtotal rservers : 4active rservers: 4description : -state : ACTIVEpredictor : ROUNDROBINfailaction : -back-inservice : 0partial-threshold : 0num times failover : 0num times back inservice : 0total conn-dropcount : 0
----------connections-----------real weight state current total failures
---+---------------------+--------+---------------------+-----------+------rserver: linux-1
192.168.1.11:0 8 OPERATIONAL 0 0 0max-conns : - , out-of-rotation count : -min-conns : -conn-rate-limit : - , out-of-rotation count : -bandwidth-rate-limit : - , out-of-rotation count : -retcode out-of-rotation count : -
Best command for checking server status and load
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 69
Layer 7 Troubleshooting
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 70
Layer 7 Policy Hits
Expanding the show service-policy using the detail option to provide hit count for layer 7 matches
switch/Admin# show service-policy client-vips detail
Status : ACTIVEDescription: ------------------------------------------Interface: vlan 211
service-policy: client-vipsL7 Loadbalance policy : pslb
class-map : curl1 LB action :
serverfarm: s1 hit count : 3dropped conns : 0
class-map : curl2 LB action :
serverfarm: s2 hit count : 0 dropped conns : 0
Shows hit count for layer 7 load balanced policy
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 71
Match URL Hit Count
Expanding the show service-policy using the url-summary option to provide visibility on which match http url are getting hit
switch/Admin# show service-policy url-summaryService-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: A1
match http url /ECBACCOUNTINQUIRY_V5/.* hit: 42
Service-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: A2match http url /AADSLICER/.* hit: 93match http url /ANALYSISHELP/.* hit: 102match http url /BOXIR2/.* hit: 67match http url /BUSINESSOBJECTS/.* hit: 78match http url /DSWSBOBJE/.* hit: 84
Use the show service-policy <service-policy-name> class-map <L3-class map-name> url-summary to provide better granularity
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 72
Troubleshooting HTTP Statistics
To effectively troubleshoot HTTP use the show stat http commandswitch/Admin# show stats http
+------------------------------------------++-------------- HTTP statistics -----------++------------------------------------------+LB parse result msgs sent : 6288 , TCP data msgs sent : 9143 Inspect parse result msgs : 0 , SSL data msgs sent : 6041
TCP fin/rst msgs sent : 135 , Bounced fin/rst msgs sent: 19 SSL fin/rst msgs sent : 13 , Unproxy msgs sent : 0 Drain msgs sent : 3107 , Particles read : 37917 Reuse msgs sent : 1539 , HTTP requests : 3145 Reproxied requests : 0 , Headers removed : 1549 Headers inserted : 1598 , HTTP redirects : 2 HTTP chunks : 0 , Pipelined requests : 0 HTTP unproxy conns : 0 , Pipeline flushes : 0 Whitespace appends : 0 , Second pass parsing : 0 Response entries recycled : 3032 , Analysis errors : 0 Header insert errors : 1509 , Max parselen errors : 0 Static parse errors : 9 , Resource errors : 0 Invalid path errors : 0 , Bad HTTP version errors : 0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 73
Troubleshooting HTTP Cookies ACE parses HTTP requests for cookies with the name
given in the configuration and can skip a certain number of bytes and look for another specific number of bytes.
If the cookie is not found, then the ACE looks for a string in the URL, starting with one of the characters /?&#+ and followed by a "=", then parses that value.
If no cookie or HTTP URL cookie exists ACE defaults to the predictor for that farm
ACE can parse up to HTTP headers (includes cookies) up to 64kB (default header max parse length is 2048k)
Make sure that the sticky timeout matches the session timeout on the application
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 74
Troubleshooting TCP Connection Re-Use When using TCP conn re-use,"Connection: keep-alive" is
inserted and "Connection: close" is removed from the clients HTTP request, to avoid closing the server connection early
User needs to create a layer 7 class-map and configure Source Nat when using TCP conn re-use
class-map type http loadbalance match-any L7-RE-USE2 match http url .*
Use the show stats http | include Reuse counters to check if see if TCP Re-uses is getting used
switch/Admin# show stats http | include ReuseReuse msgs sent : 1 , HTTP requests : 4
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 75
Health Monitoring on ACE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 76
Fundamentals for ACE probing ACE probes are fundamental to the system. It is key to not
oversubscribe the ACE health monitoring system
Use the show resource internal socket to determine how many open sockets the ACE has open. This is a Admin commandswitch/Admin# show resource internal socketApplication MaxLimit Current Creates Frees--------------------------------------------------------------SYSTEM 4000 0 0 0CRITICAL 50 0 0 0AAA 256 0 0 0MGMT 256 0 0 0XINETD 512 1 12 11HEALTH_MON 2500 532 193494 192962USER_TCL 200 0 0 0SYSLOG 256 10 14 4VSH 256 0 0 0OverAll - 650 194812 194162Non Reg App Usage: 107
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 77
Health Monitoring Process
If you see probing issues check the health monitoring process. The show proc cpu command provide very useful information
switch/Admin# show proc cpuCPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process972 1072965 613352 1749 35.9 18.5% 21.67% 20.90% arp_mgr
HM process is only consuming 1.40%. Why is the control plane CPU running at 30%? Check what process is running at 30%
switch/Admin# show proc cpuCPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process987 90257 57805 1561 0.0 1.40% 1.46% 1.43% hm988 90198 58952 1530 0.0 1.49% 1.49% 1.44% hm989 851 2947 288 0.0 0.0 % 0.1 % 0.0 % hm990 0 2 56 0.0 0.0 % 0.0 % 0.0 % hm
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 78
Health Monitoring on ACE
Use the show probe detail command to determine the status of the probe or possible last failure
switch/Admin# show probe detail – Cut output
--------------------- probe results --------------------probe association probed-address probes failed passed health------------------- ---------------+----------+----------+----------+-------rserver : CAS1
10.7.53.55 24 24 0 FAILED
Socket state : CLOSEDNo. Passed states : 0 No. Failed states : 1No. Probes skipped : 0 Last status code : 403No. Out of Sockets : 0 No. Internal error: 0Last disconnect err : Received invalid status codeLast probe time : Wed Nov 25 18:48:16 2009Last fail time : Wed Nov 25 18:25:16 2009Last active time : Never
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 79
High Availability on ACE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 80
High Availability Basic building blocks
FT PEEROnly one FT peer per ACE device
1:1 peer relationship
FT GROUPOne FT group per ACE virtual context
FT VLANDesignated VLAN between the redundant peers
All HA related traffic sent over this VLAN
FT VLAN can be trunked between two Catalyst 6500 Chassis
Cannot be used for normal traffic
Admin Context
Context A
Context B
Context A
Context B
ACE2 (FT PEER)
FT VLAN
FT Group 2
FT Group 3
ACE1 (FT PEER)
FT Group 1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 81
High Availability Control Traffic
TCP Connection between FT PeersState Machine (Election, Preempt, Relinquish)
Configuration sync
State Sync for ARP
HA KeepAlives
Heartbeats between FT peersHeartbeats are sent over UDP
Monitors the health of the peer
Heartbeat interval and count are configurable
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 82
ACE High Availability State Machine
Active Standby Election (assuming both peers are initialized at same time)
Based on a priority scheme Member with highest priority becomes ACTIVEOther member enters the STANDBY_CONFIG state If priorities are equal, member with the higher IP address wins
STANDBY_CONFIG StateStartup Configuration Sync from Active to StandbyRunning Configuration Sync from Active to StandbyKnob to turn on/off
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 83
ACE High Availability State Machine
STANDBY_BULK StateARP Sync (knob to turn on/off)
Connection Table Sync
Sticky Database Sync (knob to turn on/off)
STANDBY_HOT StateStandby FT group member is ready to take over
Incremental Configuration Sync from Active to Standy
Incremental State Sync from Active to Standby
STANDBY_COLD StateDue to error during Config Sync or Incremental Config Sync
No Config or State Sync happens from Active to Standby
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 84
ACE High Availability State Machine
Mismatch in software versionFT Peer may become INCOMPATIBLE (SRG Check)
ACTIVE ACTIVE state on both FT group members
Mismatch in Virtual Context LicensesConfiguration Sync (all types) for Admin context is disabled
State Sync for Admin context will continue to happen
For matching user contexts – Configuration State Sync will work
Mismatch in Other LicensesConfiguration and State Sync will work
After switchover, new Active will handle traffic as per its licenses
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 85
ACE Redundancy Query VLAN Query VLAN can be configured as an alternate path for
pinging the peer when no heat beat is being received from redundant peer
If configured, upon receiving a PEER_DOWN message from the heat beat process, the ACE data-plane tries to do a ping to the destination via the Query VLAN
If Ping fails, the Standby will transition to the ACTIVE state
If Ping succeeds, the Standby will transition to a STANDBY_COLD state
To configure a query interface, enter the following:
switch/Admin(config-ft-peer)# query-interface vlan 110
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 86
Common Debugging - Concussion
This session should provide you with some directions on where to start troubleshooting ACE!!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 87
Recommended Reading
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 88
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Cisco Preferred Access points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 90
Appendix
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 91
Layer 4 Flow Setup
SYN
SYN_ACK
ShortcutACK
ShortcutData
ShortcutData
Shortcut
Matches Existing FlowRewrites L2/L3/L4
Matches VIPSelects ServerRewrites L2/L3/L4
Basic Load BalancingSource IP stickyTCP/IP Normalization
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 92
Layer 7 Flow SetupClient Connects to “L7” VIP
SYN
Starts Buffering
ACK
Data
ACK’s Client PacketsKeeps Buffering
Matches VIP w/L7 logicChooses SEQ #Replies w/SYN_ACK
HTTP L7 rules on first request(cookie sticky, URL parsing, …)Generic TCP payload parsing
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 93
Layer 7 Flow Setup—ContinueACE Establishes Connection to Server
Data
SYN_ACK
Empties BufferSends Data to Server
Acts as ClientDoes Not Forward SYN_ACK
Parses the DataSelects ServerInitiates TCP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 94
Layer 7 Flow Setup—Continue ACE Splices the Flows (UNPROXY)
ACK
Data
ShortcutACK
ShortcutData
Shortcut
Matches Existing FlowRewrites L2/L3/L4and SEQ/ACK
Does Not Forward ACKReady to Splice the Flows
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 95
Layer 7 Flow SetupACE Reproxies the Connection
ACK
Data
ACKData
Shortcut
…ACK…
Shortcut
Shortcut
ShortcutData
REPROXYACK’s GET & Buffer…
HTTP L7 rules with HTTP 1.1connection keepalive(“persistence rebalance”)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 96
Layer 7 Flow SetupACE Acts as a Full Proxy
Fu
ll Pro
xyIn
dep
end
ent clien
t &
server con
nectio
ns
SYNSYN_ACK
ACKData
GET/HTTP 1.1 ACK SYN
SYN_ACKACK
Data—GET
ACK
ACKData
DataHTTP/1.1 200 OKHTTP/1.1 200 OK
Client connection Server connection
… …
SSL offloadTCP re-useProtocol inspectionsHTTP 1.1 pipelining