Application Application Performance Performance

Metrics Metrics APM BOFAPM BOFJuly 25, 2007July 25, 2007

Alan ClarkAlan Clark

Al MortonAl Morton

IETF 69 – Chicago – July 2007

AgendaAgenda

Note-Taker(s), Jabber, IPR (Yellow Note-Taker(s), Jabber, IPR (Yellow Sheet), Blue SheetsSheet), Blue Sheets

• Problem Statement and GoalsProblem Statement and Goals• Potential Solutions – WG or Potential Solutions – WG or

Directorate or…Directorate or…• Gap Analysis & Existing DraftsGap Analysis & Existing Drafts• Pro’s and Con’s / DiscussionPro’s and Con’s / Discussion• Consensus?Consensus?

Problem Statement and Problem Statement and GoalGoal

Application performance measurement/ Application performance measurement/ metrics is a specialized topicmetrics is a specialized topic

WG’s may have application but not WG’s may have application but not performance measurement expertiseperformance measurement expertise

Performance metric drafts tend to get less Performance metric drafts tend to get less attention than application oriented draftsattention than application oriented drafts

No WG or Directorate focused on No WG or Directorate focused on Application Performance MeasurementApplication Performance Measurement

GOAL: explore the need for a new WG GOAL: explore the need for a new WG and/or Directorate with the IETF and/or Directorate with the IETF community and capture the consensus community and capture the consensus (and solution direction) (and solution direction)

Proposed WG and or Proposed WG and or DirectorateDirectorate

APM DirectorateAPM Directorate Directorate Role: Advise and Review Directorate Role: Advise and Review

work in other WGs and write BCPwork in other WGs and write BCP APM Working GroupAPM Working Group

WG Role: Prepare RFCs in Coordination WG Role: Prepare RFCs in Coordination with other WG and write BCPwith other WG and write BCP

Short-lived APM WG – possibly Short-lived APM WG – possibly evolving to one of aboveevolving to one of above Writes a BCP or Framework RFCWrites a BCP or Framework RFC

ConstraintsConstraints

DoDo Develop in cooperation with relevant Develop in cooperation with relevant

WG’sWG’s Drafts related to IP based applications, with Drafts related to IP based applications, with

particular focus on IETF applicationsparticular focus on IETF applications Drafts related to transport protocol Drafts related to transport protocol

performance, with particular focus on performance, with particular focus on reliable transportreliable transport

Cooperate with other standards Cooperate with other standards organizations working in related areasorganizations working in related areas

ConstraintsConstraints

Don’tDon’t Develop metrics in areas where other Develop metrics in areas where other

standards organizations have standards organizations have established expertiseestablished expertise Examples: Examples:

Voice, Audio, Video perceptual quality (expertise Voice, Audio, Video perceptual quality (expertise of ITU-T SG12, SG9, VQEG)of ITU-T SG12, SG9, VQEG)

IP performance metrics (expertise of IPPM and IP performance metrics (expertise of IPPM and ITU-T SG12)ITU-T SG12)

Existing draftsExisting drafts draft-malas-performance-metrics-07.txtdraft-malas-performance-metrics-07.txt draft-venna-ippm-app-loss-metrics-00.txtdraft-venna-ippm-app-loss-metrics-00.txt draft-ietf-avt-rtcpxr-video-01.txtdraft-ietf-avt-rtcpxr-video-01.txt draft-ietf-avt-rtcpxr-audio-00.txtdraft-ietf-avt-rtcpxr-audio-00.txt draft-ietf-avt-rtcpxr-mpts-00.txtdraft-ietf-avt-rtcpxr-mpts-00.txt draft-ietf-avt-rtcpxr-transport-00.txtdraft-ietf-avt-rtcpxr-transport-00.txt draft-ietf-avt-rtcphr-01.txtdraft-ietf-avt-rtcphr-01.txt draft-xie-ccamp-lsp-dppm-01.txtdraft-xie-ccamp-lsp-dppm-01.txt draft-kikuchi-passive-measure-00.txtdraft-kikuchi-passive-measure-00.txt Others?Others?

How well do current How well do current WG’s handle WG’s handle

Performance Metrics?Performance Metrics? Feedback from authors on progress Feedback from authors on progress

made within current WG’s?made within current WG’s? Feedback from WG Chairs / A-D’s on Feedback from WG Chairs / A-D’s on

good/ difficult aspects of progressing good/ difficult aspects of progressing performance measurement drafts performance measurement drafts within existing WG’s?within existing WG’s?

Any lessons to learn?Any lessons to learn?

SIP End-to-EndSIP End-to-EndPerformance Performance

MetricsMetrics6969thth IETF – Chicago IETF – Chicago

(APM BoF)(APM BoF)

Daryl MalasDaryl Malas

Problem StatementProblem Statement

With widespread implementation of SIP With widespread implementation of SIP the following problems have surfaced:the following problems have surfaced: No standard method for measuring SIP No standard method for measuring SIP

performanceperformance Industry confusion on “How”Industry confusion on “How” Industry confusion on “Where” to measure Industry confusion on “Where” to measure Current reliance on PSTN (and other) metricsCurrent reliance on PSTN (and other) metrics

Draft defines the “How” and the “Where” Draft defines the “How” and the “Where” for common metrics applicable to ALL SIP for common metrics applicable to ALL SIP applications.applications.

MetricsMetrics Registration Request Delay (RRD)Registration Request Delay (RRD) Session Request Delay (SRD)Session Request Delay (SRD) Session Disconnect Delay (SDD)Session Disconnect Delay (SDD) Session Duration Time (SDT)Session Duration Time (SDT) Average Hops per Request (AHR)Average Hops per Request (AHR) Session Establishment Rate (SER)Session Establishment Rate (SER) Session Establishment Efficiency Rate (SEER)Session Establishment Efficiency Rate (SEER) Session Defects (SD)Session Defects (SD) Ineffective Session Attempts (ISA)Ineffective Session Attempts (ISA) Session Disconnect Failures (SDF)Session Disconnect Failures (SDF) Session Completion Rate (SCR)Session Completion Rate (SCR) Session Success Rate (SSR)Session Success Rate (SSR)

RTCP XR Video MetricsRTCP XR Video Metrics

RTCP XR – Video MetricsRTCP XR – Video Metrics RTCP XR – Audio MetricsRTCP XR – Audio Metrics RTCP XR – MPEG Transport MetricsRTCP XR – MPEG Transport Metrics RTCP XR – Transport MetricsRTCP XR – Transport Metrics Set of RTCP XR report blocks for Set of RTCP XR report blocks for

IPTV (and potentially other IP Video) IPTV (and potentially other IP Video) performance reportingperformance reporting

Incorporates packet, transport, Incorporates packet, transport, application metricsapplication metrics

RTCP HRRTCP HR

RTCP HR – intended use – carrier RTCP HR – intended use – carrier backbone VoIP servicesbackbone VoIP services

Useful in VoIP services with multiple Useful in VoIP services with multiple interconnected segments (e.g. interconnected segments (e.g. transcoding gateways)transcoding gateways)

Reports range of packet loss Reports range of packet loss distribution metrics, signal related distribution metrics, signal related metrics etc.metrics etc.

Strong ties to work in ITU-T .. H.248Strong ties to work in ITU-T .. H.248

Perf. Metric ProposalsPerf. Metric Proposals

Application Loss Pattern MetricsApplication Loss Pattern Metrics Loss Pattern metrics to infer App. Loss Pattern metrics to infer App.

Performance and derive Errored Seconds.Performance and derive Errored Seconds. LSP Dynamical Provisioning LSP Dynamical Provisioning

Performance Metrics in GMPLSPerformance Metrics in GMPLS LSP Setup (Uni/Bi-directional) and LSP Setup (Uni/Bi-directional) and

Graceful Rel.Graceful Rel. Passive Measurement of e2e QualityPassive Measurement of e2e Quality

Exploits a Seq. Num. Field to measure Exploits a Seq. Num. Field to measure loss, duplicate, and reordering. loss, duplicate, and reordering.

Other related activitiesOther related activities BMWG – Defines Performance Metrics related BMWG – Defines Performance Metrics related

to IP-based networking up and down the stack to IP-based networking up and down the stack Restricted to Lab CharacterizationRestricted to Lab Characterization

IPPM – Active Performance Characterization IPPM – Active Performance Characterization of Live Networks at IP layer (TCP also of Live Networks at IP layer (TCP also addressed)addressed) No Passive Traffic MonitoringNo Passive Traffic Monitoring

OPSAWG - BCP guidelines for OPSAWG - BCP guidelines for authors/reviewers of new IETF protocols for authors/reviewers of new IETF protocols for operational and manageability requirements operational and manageability requirements + other small OPS projects+ other small OPS projects

RMON – concluded, with many metrics RMON – concluded, with many metrics referenced for remote monitoring purposesreferenced for remote monitoring purposes

Proposal 1 – new APM Proposal 1 – new APM DirectorateDirectorate

Responsibilities: Advise protocol WGs Responsibilities: Advise protocol WGs initiating and developing APMs initiating and developing APMs correctness of metrics definitionscorrectness of metrics definitions Measurement and Reporting MethodologiesMeasurement and Reporting Methodologies Assist with Coordination with other Stds OrgsAssist with Coordination with other Stds Orgs

Mode of operationMode of operation Consult, review, and provide point of reference Consult, review, and provide point of reference Drafts would be chartered in the protocol WG.Drafts would be chartered in the protocol WG.

Directorate would prepare a Directorate would prepare a BCP/framework RFCBCP/framework RFC

Proposal 2 – new APM Proposal 2 – new APM WGWG

Responsibilities: Responsibilities: Development of RFCs that characterize the Development of RFCs that characterize the

performance of Layers above IP, especially those performance of Layers above IP, especially those utilizing IETF protocolsutilizing IETF protocols

Advance those RFCs along the Standards Track.Advance those RFCs along the Standards Track. Coordination with other Stds OrgsCoordination with other Stds Orgs Prepare a BCP/Framework RFCPrepare a BCP/Framework RFC

Mode of Operation: Mode of Operation: Partner with a specific protocol development Partner with a specific protocol development

WG whenever possible.WG whenever possible. Take on work ONLY with agreement from Take on work ONLY with agreement from

relevant protocol WG and IESG.relevant protocol WG and IESG.

Proposal 3 – Short-Lived Proposal 3 – Short-Lived APM WGAPM WG

ResponsibilitiesResponsibilities Prepare a BCP/Framework RFCPrepare a BCP/Framework RFC Act in the Role of the Directorate while Act in the Role of the Directorate while

in existencein existence Then, evolve to Directorate or WG after Then, evolve to Directorate or WG after

completing the charter.completing the charter.

Pro’s and Con’s Pro’s and Con’s

APM DirectorateAPM Directorate ProPro ConCon

APM WGAPM WG ProPro ConCon

BCPBCP ProPro ConCon

Key questions – 1Key questions – 1

Should IETF commit its resources to Should IETF commit its resources to the formal development of the formal development of application performance metrics in application performance metrics in one or more key areas?one or more key areas?

Key questions – Preferred Key questions – Preferred DirectionDirection

1.1. Protocol Development WGs with advice Protocol Development WGs with advice from an APM Directoratefrom an APM Directorate

2.2. APM WG with participation by experts APM WG with participation by experts from the relevant Protocol WG.from the relevant Protocol WG.

3.3. Short-lived WG to prepare Framework Short-lived WG to prepare Framework RFC/BCP, evolving to Dir or WG…RFC/BCP, evolving to Dir or WG…

4.4. Neither an APM Directorate or an APM Neither an APM Directorate or an APM WG are neededWG are needed

Key questions – 3Key questions – 3

If the APM Directorate approach is If the APM Directorate approach is desirable, is the proposed method of desirable, is the proposed method of working acceptable, or are working acceptable, or are modifications needed? modifications needed?

Key questions – 4Key questions – 4

If the (short-lived) APM WG approach If the (short-lived) APM WG approach is desirable, is the method of working is desirable, is the method of working acceptable, or are modifications acceptable, or are modifications needed?needed?

SummarySummary

Conclusions from this BOF?Conclusions from this BOF? Next stepsNext steps

Contact infoContact info

Alan Clark – Alan Clark – alan@telchemy.comalan@telchemy.com

Al Morton – Al Morton – acmorton@att.comacmorton@att.com