Lync 2013 – TroubleshootingMartin Rinas

Support Architect

Global Business Support Strategy

Microsoft Corporation

Agenda

Lync Performance Monitoring

Centralized Logging Service (CLS)

Lync Media on WiFi

Troubleshooting Tools

Deployment completed, SMEs are gone, the customer is left to support an ever-changing software update cycle.

Customers are often not equipped to manage the complexity of Lync. Many proactive steps can prevent the most common scenarios that generate support calls and generally can potentially leave Lync functionality crippled and unreliable.

Lync Performance Monitoring

Lync Performance MonitoringSystem Center Operations Manager (SCOM): SCOM is an alerting system providing data on server status

Performance Counters: Feed into SCOM and for general server performance monitoring. Includes active connections, processing of messages, failures raised by server, latency

Event Logs: Used to report to SCOM, configuration state on server, security policy update, service availability

Synthetic Transactions: Automated tests to detect outages in service features (e.g. , Instant Messaging [IM], registration, presence)

Call Detail Records (CDR): CDR provides telemetry on usage patterns (e.g., call volume), call establishment (e.g., conference join)

QoE Metrics: Media, network, endpoint and connection metrics

collected on endpoint

UFD: Actionable notifications displayed to user

Network Bars: Indicator providing users with information when

network performance is causing media quality issues

QoE Metrics: media, network, endpoint

and connection metrics collected on

endpoint

QoE Metrics: Media, network, endpoint and connection metrics

collected on endpoint

UFD: Actionable notifications displayed to user.

Network Bars: Indicator providing users with information when

network performance is causing media quality issuesCDR/QoE

SQL Database

QoE Data CollectionFront End Server

Lync Storage Service

Unified Contacts

Replication for

HA

Data Collection

Archival Processing

(IM, WebConf)

Monitoring Processing

(CDR/QoE)

SQL DB

Queue DB

What Factors Reduce Media Quality?Audio and video coding

Network performance• Latency (delay and jitter)

• Packet loss

• Low and/or variable throughput

Device• Low-resolution screen

• Low processing capability

Environment• Noise

• Lighting

Strong suite of metrics covering• Endpoint spec

OS, CPU, cores, processor speed, media capture/render device

• Call identification (ID)

Session Initiation Protocol (SIP) URI, conference bridge URI, and media bypass ID

• Network performance

TCP/UDP, network connectivity, PLR, IPv4 address, jitter, burstiness, and latency

• Audio quality

Healer behavior, FEC, sampling rate, RTT, MOS

• Video quality

Resolution, bitrate, FPS, PLR, freezes, video bandwidth, and rate matching

Lync 2010 Metrics

Application Sharing metrics• Latency, burst loss characteristics,

jitter

• RDP Tile Processing Latency

Audio metrics• Stereo audio usage

• Per channel signal and noise level metrics for stereo

• Improved audio healer metrics (FEC usage, packet drop ratio)

• Extended audio jitter metric to capture variability in jitter

Video metrics for multiview and single-view video

• Codec specific performance reports

• Metrics available for each multiview video channel

• Aggregated metrics provide general video performance

• Per channel metrics provide detailed video performance data

• Lync 2013 will add video metrics for identifying bad/poor video experience

• FEC usage for video (including success ratio)

QoE Metrics Added in Lync 2013

Summary

TroubleshootingIn Lync 2013, improved video metrics are aligned to the new video feature setReports will have both audio and video media performance analysisNew QoE will enable administrators to better identify problems with both audio and video

PlanningQoE provides information on

Network performance and problem identificationAudio performance issuesVideo usage and performance issuesQoE data assists in

Network planning (e.g., wired and wireless access requirements)Server and general infrastructure procurement decisions

Centralized Logging Service

• The challenge• 1 Pool, 6 Front End Servers

• Investigating intermittent client disconnects

• Prior to Lync 2013

• 6x OCSLogger, one on every server

• 6x Log files, all in different locations, and not related

• Where do I start to analyze?

• Am I looking at the right server?

• Lync 2013 - Centralized Logging Service• New in Lync 2013

• Replaces OCSLogger & OCSTracer tools

• Start/Stop and Flush trace logging on ALL SERVERS in a deployment from ONE PLACE

Centralized Logging Service - CLS

• CLS Agent• Runs on every Lync Server

• Controlled by CLS controller

• Manages log files

• CLS Controller• Controls the agents (start, stop, flush)

• Aggregates search results from all agents

• Available on every Lync Server

• Managed through Lync Management Shell

CLS - Architecture

Start, Stop, Search commands

Logging is based on Scenario’s• A scenario specifies a group of components and log levels to start/stop together• Lync 2013 comes with Pre-defined scenario’s:

AlwaysOn scenario• Designed to be Always On• Logs many common components on a INFO level• Can be run together with other scenario’s• Not enabled by Default

CLS -Scenario based logging

AddressBook

AlwaysOn

ApplicationSharing

AudioVideoConferencingIssue

CAA

CLS

CPS

DeviceUpdate

HostedMigration

HybridVoice

IMAndPresence

IncomingAndOutgoingCall

LILRLegacy

LILRLYSS

LYSSAndUCS

MediaConnectivity

MeetingJoin

MonitoringAndArchiving

RGS

SP

UserReplicator

VoiceMail

WAC

XMPP

Scenario DiscoveryLike in OCSLogger, Components/Logging Levels & Flags

CLS – Scenario’s cont’d

PS C:\> Get-CsClsScenario global/MeetingJoin| Select -ExpandProperty Provider

| Format-Table Name,Level,Flags -a

Name Level Flags

---- ----- -----

Collaboration Info TF_COMPONENT,TF_PROTOCOL,TF_CONNECTION,TF_DIAG

S4 Info TF_COMPONENT,TF_PROTOCOL

UserServices Verbose TF_COMPONENT,TF_CM,TF_RTCHTTP,TF_PROTOCOL

McuInfra Info TF_COMPONENT,TF_PROTOCOL,TF_CONNECTION

JoinLauncher Verbose TF_COMPONENT

WebInfrastructure Verbose All

Infrastructure Verbose All

InternalCommon Verbose All

UCWA Info TF_COMPONENT

WebRelay Info All

Changing / Adding scenario’sCustom Scenario’s can be created if needed

Existing Scenario’s can be updated & changed

CLS – Custom Scenario’s

$provider1 = New-CsClsProvider -Name "Sipstack" -Type "WPP" -Level "All" -Flags "All" -Role Edge

$provider2 = New-CsClsProvider -Name "ExUmRouting" -Type "WPP" -Level "All" -Flags "All"

$provider3 = New-CsClsProvider -Name "InboundRouting" -Type "WPP" -Level "All" -Flags "All"

Set-CsClsScenario -Identity "global/VoicemailVerbose" -Provider @{Add=$provider1,$provider2,$provider3}

$provider = New-CsClsProvider -Name "Sipstack" -Type "WPP" -Level "All" -Flags "All" -Role FE

New-CsClsScenario -Identity "global/VoicemailVerbose"-Provider $provider

Turn on a logging scenarioStart-CsClsLogging

–Scenario AlwaysOn –Pools pool.contoso.com

Start-CsClsLogging

–Scenario IncomingAndOutgoingCall

-Pools pool.contoso.com

Controlling Centralized LoggingPS C:\> Start-CsClsLogging -scenario "IncomingAndOutgoingCall"

-Pools "pool1.tailspin.local" -Duration 5:00

Success Code - 0, Successful on 2 agents

Tracing Status:

pool1.tailspin.local (pool1 v5.0.8308.0)

(AlwaysOn=No,Scenario=IncomingAndOutgoingCall,Started=6/24/201

3 5:27:17 AM,By=TAILSPIN\administrator,Duration=0.05:00)

fe02.tailspin.local (fe02 v5.0.8308.0) (Same as pool)

fe01.tailspin.local (fe01 v5.0.8308.0) (Same as pool)

PS C:\> Show-CsClsLogging

Success Code - 0, Successful on 5 agents

Tracing Status:

edge01.tailspin.local (edge01 v5.0.8308.0) (AlwaysOn=No)

edge01.tailspin.local (edge01 v5.0.8308.0) (Same as pool)

pool1.tailspin.local (pool1 v5.0.8308.0) (AlwaysOn=Yes)

fe02.tailspin.local (fe02 v5.0.8308.0) (Same as pool)

fe01.tailspin.local (fe01 v5.0.8308.0) (Same as pool)

pool2.tailspin.local (pool2 v5.0.8308.0) (AlwaysOn=No)

fe03.tailspin.local (fe03 v5.0.8308.0) (Same as pool)

Searching the log files

Full dump of all log files:Search-CsClsLogging

-OutputFilePath “c:\Logfiles\log.txt”

Filtered logfiles:Search-CsClsLogging

-Components “SIPStack”,”S4”,”UserServices”

-OutputFilePath “c:\Logfiles\log.txt”

-Uri "sip:alice@tailspin.com" -MatchAll

Use Snooper to analyze the log

In memory vs. On DiskLogging data is stored in memory and periodically write that data to disk.Use Sync-CsClsLogging cmdlet to "flush" the data cache. Writes logging data from memory to disk & clears cache

CLS - SearchingPS C:\> Search-CsClsLogging -OutputFilePath "c:\SearchResult.txt"

Using start time 6/24/2013 8:14:37 AM

Using end time 6/24/2013 8:44:37 AM

Starting search against 5 computers

Success Code - 0, Successful on 4 agents

Search results written to c:\SearchResult.txt

PS C:\> Sync-CsClsLogging

Success Code - 0, Successful on 5 agents

The “AlwaysOn” Scenario

The special scenario “AlwaysOn” can be on all the time. It logs INFO level for many common components

When an issue occurs there may be enough info in the AlwaysOn component logs to debug the issue

This avoids having to turn on logging and repro the issue, which was always required in Lync Server 2010

If the logs from AlwaysOn are not sufficient, turn on the specific scenario relevant to your issue, repro the issue, and get a higher level of logging

At any given time, you can have one extra scenario enabled along with AlwaysOn

Lync Media on Wi-Fi

Lync Media over Wi-Fi

Lync 2013 Wi-Fi support

Goal is to provide general deployment guidance and recommendations for Wi-Fi deployment

Working with Wireless Local Area Network (WLAN) infrastructure vendors using Lync in their deployments

Wi-Fi Problem Areas2.4-GHz Band

• Shared Medium

• Interference in 2.4 gigahertz (GHz)

• Bluetooth

• Appliances and cordless phones

• Other access points (APs) on the same channel nearby

Limited band selection in 2.4-GHz band

• Only three non-overlapping channels

5-GHz band • Less interference

• Wider channel selection

• Reduced range

• Requires APs and devices to support 5-GHz band

Deploy concurrent dual-band AP• Applicable for enterprise, home, and hotspot

• Move devices to 5 GHz (if possible)

Consider quality and features of Wireless NICs• Wi-Fi alliance certification

• Multiple antennae support (11n)

Wi-Fi Deployment Recommendations for Lync

Enterprise Wi-Fi ConsiderationsUnderstand usage requirements

• Nomadic usage

• Mobile usage

• Endpoints that need Lync over Wi-Fi support

Managed WLAN infrastructure

• Central WLAN controller

• Thin Enterprise-grade APs

High-density AP deployments

Fast inter-AP handover support

Select WLAN infrastructure vendors with voice over Wi-Fi experience

Enterprise• High-density AP deployment with support 11a/11g, and recently adding 11n

support

• Enterprise WLAN Controller with thin Enterprise grade APs, dual-band 2.4 and 5 GHz

• WPA2 Enterprise requires fast AP Handover support

• Enterprise class notebooks with at least 2x2:2 11n high-quality wireless NIC recommended

• Wireless QoS (WMM) can be implemented

Lync 2013 Wi-Fi Deployment Guidance

Home• Consumer grade AP, typically integrated router, hub, AP• No handover need; problems areas are interference, and low signal strength• Minimum recommended for Lync 11g router. Not recommended is 11b • Trend: 11n concurrent dual-band router with separate Service Set Identifier

(SSID)

Hotspot• 11b and 11g, with only a few 11n implementations• Deployments may not be suitable for Lync media: Throttling; port blocking;

quota limitation

Lync 2013 Wi-Fi Deployment Guidance

Troubleshooting Tools

QoE ReportsLync Debug toolsLync Network Monitor with Lync ParsersSnooperRemote connectivity analyzerLync 2013 BPATelnet Event logging (Capi2)HLB isolation

Questions?

ReferenceDebugging Tools

http://www.microsoft.com/en-us/download/details.aspx?id=35453

Lync Server Networking Guide v2http://www.microsoft.com/en-us/download/details.aspx?id=39084

QoE/CDR DB Schemahttp://technet.microsoft.com/en-us/library/gg398687.aspx

Lync Ressource Kit Toolshttp://www.microsoft.com/en-us/download/details.aspx?id=36821

NetMon + Lync Parserhttp://www.microsoft.com/en-us/download/details.aspx?id=4865

http://www.microsoft.com/en-us/download/details.aspx?id=22440

Remote Connectivity Analyzerhttps://testconnectivity.microsoft.com/

BPA & System Center Advisorhttp://technet.microsoft.com/en-us/library/gg558584.aspx

https://www.systemcenteradvisor.com/

RAP as a Servicehttps://services.premier.microsoft.com/