cx collector...march 19, 2020 collector overview cisco confidential.all printed copies and duplicate...

Cisco Confidential. All printed copies and duplicate soft copies are considered uncontrolled and the original online version should be referred to for the latest version.

CX Collector

Collector Overview

April 20, 2020

April 20, 2020 Collector Overview Cisco Confidential. All printed copies and duplicate soft copies are considered uncontrolled

and the original online version should be referred to for the latest version.

of 72

Contents

CONTENTS ....................................................................................................................................................... 2

1 OVERVIEW .................................................................................................................................................... 4

1.1 DEPLOY CX COLLECTOR WITH CUSTOMER PORTAL ................................................................................................... 5 1.2 BEFORE YOU BEGIN ............................................................................................................................................ 5 1.3 ABBREVIATIONS ................................................................................................................................................. 6 1.4 CISCO DNA CENTER SUPPORTED VERSIONS ............................................................................................................ 6 1.5 SUPPORTED BROWSERS....................................................................................................................................... 6

2 DEPLOYING CX COLLECTOR ........................................................................................................................... 7

3 CONNECTING CX COLLECTOR TO CUSTOMER PORTAL ................................................................................. 11

4 MANAGE USERS .......................................................................................................................................... 17

5 DEPLOYMENT AND IP CONFIGURATION ...................................................................................................... 20

5.1 OVA DEPLOYMENT .......................................................................................................................................... 20 5.1.1 Thick Client ESXi 5.5/6.0 Installation.................................................................................................... 20 5.1.2 Web Client ESXi 6.0 Installation ........................................................................................................... 24 5.1.3 Web Client vCenter Installation ........................................................................................................... 29 5.1.4 Oracle Virtual Box 5.2.30 Installation .................................................................................................. 33 5.1.5 Microsoft Hyper-V Installation ............................................................................................................. 36

5.2 IP CONFIGURATION .......................................................................................................................................... 41

6 SET UP SYSLOG FORWARDING ON CISCO DNA CENTER ............................................................................... 44

6.1 PREREQUISITE ................................................................................................................................................. 44 6.2 CONFIGURE SYSLOG FORWARDING SETTING .......................................................................................................... 44 6.3 ENABLING INFO LEVEL SYSLOG SETTINGS .............................................................................................................. 45

7 ADMIN SETTINGS ........................................................................................................................................ 46

7.1 UPGRADE CX COLLECTOR .................................................................................................................................. 46 7.2 CHANGE THE PASSWORD ................................................................................................................................... 47 7.3 UPDATE CISCO DNA CENTER CREDENTIALS .......................................................................................................... 49 7.4 ADD CISCO DNA CENTER .................................................................................................................................. 50

8 SECURITY .................................................................................................................................................... 52

8.1 PHYSICAL SECURITY .......................................................................................................................................... 52 8.2 USER ACCESS .................................................................................................................................................. 52 8.3 ACCOUNT SECURITY ......................................................................................................................................... 52 8.4 NETWORK SECURITY ......................................................................................................................................... 52 8.5 AUTHENTICATION ............................................................................................................................................ 52 8.6 HARDENING .................................................................................................................................................... 52 8.7 DATA SECURITY ............................................................................................................................................... 53 8.8 DATA TRANSMISSION ....................................................................................................................................... 53 8.9 LOGS AND MONITORING ................................................................................................................................... 53 8.10 SECURITY SUMMARY ...................................................................................................................................... 53

9 FREQUENTLY ASKED QUESTIONS ................................................................................................................ 55

9.1 CX COLLECTOR ................................................................................................................................................ 55 9.2 CX COLLECTOR CONNECTION WITH CISCO DNA CENTER ......................................................................................... 58 9.3 CX COLLECTOR USED DIAGNOSTIC SCAN .............................................................................................................. 59 9.4 CX COLLECTOR SYSTEM LOGS ............................................................................................................................. 59



of 72

10 TROUBLESHOOTING .................................................................................................................................. 61

10.1 COLLECTION FAILURE RESPONSES ...................................................................................................................... 63 10.2 DIAGNOSTIC SCAN FAILURE RESPONSES ............................................................................................................. 65

11 PORTAL SUPPORT ..................................................................................................................................... 68

12 COLLECTOR PATCH INSTALLATION ............................................................................................................ 69

12.1 RELEASE ARTIFACTS ........................................................................................................................................ 69 12.2 PATCH INSTALLATION...................................................................................................................................... 69 12.3 TROUBLESHOOTING STEPS ............................................................................................................................... 70

13 PERFORMANCE TEST SUMMARY ............................................................................................................... 71

14 ADDENDUM .............................................................................................................................................. 72



of 72

1 Overview

Cisco Customer Experience Collector (CX Collector) is a modernized modular on-prem software platform that unifies all existing on-prem applications into lightweight containerized microservice capabilities. These capabilities could be installed, configured, and managed on customer premise from the cloud. CX Collector is a major advancement in how we create, deploy, and manage our on-prem software capabilities that are tied to business offers. It expedites the monetization of new offers, scales existing capabilities, and helps to develop next-gen services driven by big data, analytics, automation, ML/AI, and streaming.

Figure 1: CX Collector Architecture



of 72

1.1 Deploy CX Collector with Customer Portal

CX Collector manages data collection with CX cloud and display devices with legacy support contracts if mapped to common Smart Account in Customer Portal. CX Collector uitlizies Smart Account /Virtual Account Structure, leverage product-sourced telemetry (Cisco DNA Center) and close data gaps.

Figure 2: Deployment Model

1.2 Before you begin

CX Collector runs as a virtual machine (VM) and is available for download as an OVA. The requirements to deploy are: Any of the following hypervisor:

• VMware ESXi version 5.5 and above

• Oracle Virtual Box 5.2.30

• Microsoft Hyper-V The hypervisor should be able to host a VM which needs the following resources:

• 8 Core Cpu

• 16 GB Memory/RAM,

• 200GB Disk Space

• The deployed CX Collector should able to connect to the concsoweb-prd.cisco.com on https port 443 directly or via a proxy to send data to Cisco

• For local management of the CX Collector, port 443 and port 22 should be accessible

Other notes on CX Collector:

• An IP will be automatically detected if DHCP is enabled in the VM environment else will need to have a free IPv4 address to be assigned to the CX Collector, know about the Subnet mask for the network, the IP of the Default Gateway and optionally the IP of DNS server

• Only IPv4 is supported, not IPv6

• Ensure, the single node Cisco DNA Center version is 1.2.10, 1.2.12, 1.3(.x).



of 72

• Ensure, HA Cluster Cisco DNA Center version is 1.3.3.1.

Attention Migration Users: If you are migrating from an existing EFT deployment you must first need to decommission/delete your old VM prior to installing the new VM/OVA to ensure that installation is not corrupted.

Those customers who have deployed CX Collector version 0.9.0, need to perform a manual upgrade (i.e. by downloading/applying the collector upgrade patch from cisco.com) as one-time activity to enable the remote upgrade feature

System events (AFM): In order to raise automated TAC cases, CCO ID given at the time of registration should have devices contracts associated with it.

1.3 Abbreviations

The list of abbreviations used in this document are:

• API – Application Program Interface

• BDB – Big Data Broker

• CX – Customer Experience

• DHCP – Dynamic Host Configuration Protocol

• Cisco DNA Center – Cisco Digital Network Architecture Center

• OVA – Open Virtual Appliance

• SSH – Secure Socket Shell

• TLS – Transport Layer Security

1.4 Cisco DNA Center Supported Versions

• Supported single node Cisco DNA Center versions are 1.2.10, 1.2.12, 1.3(.x)

• Supported HA Cluster Cisco DNA Center version is 1.3.3.1

Figure 3: Multi-Node HA cluster Cisco DNA Center

1.5 Supported Browsers

Browsers supported are Chrome and Firefox.



of 72

2 Deploying CX Collector

1. Login to CX Customer Portal cx.cisco.com, then navigate to Assets & Coverage tile. Click Continue Setup to start the deployment.

Figure 4: CX Collector landing page

In this release AFM features are automatically enabled on the CX Collector. This does not need your manual intervention.

https://cx.cisco.com/



of 72

2. Read through the prerequisites. Click Continue.

Figure 5: Prerequisites

3. Verify the auto populated information First name, Last name, Email, COO User ID and select the business division’s functions and notify if you are any government entity. Check the box and click Accept to agree to the Encryption agreement.



of 72

Figure 6: Encryption Agreement

This screen recurs until you accept the agreement.



of 72

4. Click Accept to accept the end user license agreement.

Figure 7: End User License Agreement

This screen recurs until you accept the end user license agreement.

5. Select the location to secure your data. A prompt appears, click Yes to confirm the storage location or click NO to reject the storage location.

6. Select the format you require to install. Click Download Image to get the installation file.

Figure 8: Download Image

Figure 9: Prompt Message

7. If you need help to deploy the setup, click View step-by-step tutorial on the next screen, else skip it. The step-by-step tutorial is explained in section Deployment and IP Configuration. Download depends on the network speed.



of 72

3 Connecting CX Collector to Customer Portal

1. Enter the IP Address of the Virtual Machine you just configured and click Continue.

Figure 10: Connect to CX Collector



of 72

2. To accept the certificate in your browser, click Proceed to Certificate. This opens a new window in a browser to perform the steps as shown in the figure below.

Figure 11: Browser Certificate



of 72

3. Confirm that the collector IP you had setup is validated. Switch back to the portal window. This is Self-signed certificate to establish secure connectivity with CX Portal

Figure 12: Validated

4. Enter the password for CX Collector and click Continue. Password Change and Proxy Server details are optional.

Figure 13: CX Collector Access Password Optional



of 72

Figure 14: CX Collector Proxy Optional

Figure 15: Registering CX Collector



of 72

Figure 16: CX Collector Registered Successfully



of 72

5. Once the registration is completed, enter the FQDN or IP Address, and credentials of Cisco DNA Center.

6. Select the location from the list and you can choose either run now or schedule the collection later.

7. You can schedule the ongoing Inventory collection and click Continue.

8. Click Add another Cisco DNA Center for collection and repeat steps 5, 6 and 7. You can add up to 10 Cisco DNA Center clusters.

Two different assets having configured for same IP address behind two separate DNA Center clusters is not supported currently.

Figure 17: Connect Cisco DNA Center



of 72

4 Manage Users

You can add, remove, and edit user using manage users. Search users using the search option. Expand the arrow next to each role to view the roles tagged to user. You can send the welcome mail to newly added users.

Figure 18: Manage User

1. Click Add User next to search to add new users

Figure 19: Add Users

2. Enter the CCO ID and Email. Choose Role from the drop down and check the box to authenticate the user.

3. Click the button highlighted in the figure below and select the account to remove.



of 72

Figure 20: Remove Account

4. A prompt appears click Remove to delete the smart account.

Figure 21: Prompt

5. CX Portal landing page appears as shown figure. If the data is not populated on the screen within 8 to 24 hrs., then you need to contact support for the queries.



of 72

Figure 22: CX Portal Landing Page



of 72

5 Deployment and IP Configuration

You can choose any one of below options to deploy CX Collector:

Figure 23: Deployment Environments

• If you select VMware vSphere/vCenter Thick Client ESXi 5.5/6.0 go to Thick Client

• If you select VMware vSphere/vCenter Web Client ESXi 6.0 go to Web Client vSphere or vCenter

• If you select Oracle Virtual Box 5.2.30 go to Oracle VM

• If you select Microsoft Hyper-V go to Hyper-V

5.1 OVA Deployment

5.1.1 Thick Client ESXi 5.5/6.0 Installation

This client allows you to deploy CX Collector OVA using the vSphere thick client. 1. After downloading the image, launch the VMware vSphere Client and login using the credentials.

Figure 24: Login



of 72

2. Go to File > Deploy OVF Template

Figure 25: vSphere Client

3. Browser to select OVA file. Click Next> to proceed.

Figure 26: OVA Path



of 72

4. Verify the OVF Details and click Next> to proceed

Figure 27: Template Details

5. Enter a Unique Name and click Next> to proceed

Figure 28: Name and Location



of 72

6. Select Disk Format and click Next> to proceed (recommended Thin Provision)

Figure 29: Disk Format

7. Check the Power on after deployment and click Finish

Figure 30: Ready to Complete



of 72

8. Deployment may take several minutes. Wait until you get success message.

Figure 31: Deployment in Progress

Figure 32: Deployment Completed

9. Select the virtual machine you just deployed and open the console. Go to IP Configuration.

5.1.2 Web Client ESXi 6.0 Installation

This client allows you to deploy CX Collector OVA using the vSphere web. 1. Login to VM Ware UI using credentials.

Figure 33: VMware ESXi Login



of 72

2. Open Virtual Machine > Create / Register VM

Figure 34: Create VM

3. Select Deploy a virtual machine from an OVF or OVA file and click Next

Figure 35: OVA Deployment



of 72

4. Enter the Name of the VM, browse to select the file or drag and drop the downloaded OVA file. Click Next

Figure 36: OVA Selection

5. Select the Standard Storage and click Next

Figure 37: Select Storage



of 72

6. Select the deployment options and click Next

Figure 38: Deployment Options

7. Review the settings and click Finish

Figure 39: Ready to Complete



of 72

Figure 40: Successful Completion

8. Select the virtual machine you just deployed and click Console > Open browser console.

Figure 41: Open Console

9. Go to IP Configuration.



of 72

5.1.3 Web Client vCenter Installation

1. Use Login to vCenter Client using the credentials.

Figure 42: Login

2. On Home page click Hosts and Clusters.

Figure 43: Home Screen

3. Select the VM and click Action>Deploy OVF Template.



of 72

Figure 44: Actions

4. You can either add URL directly or browse to select the ova file and click Next.

Figure 45: Select Template

5. Enter a unique name and only if required browse to the location. Click Next.

Figure 46: Name and Location

6. Select the source and click Next.



of 72

Figure 47: Select a Source

7. Review the details and click Next.

Figure 48: Review Details

8. Select the virtual disk format and click Next.

Figure 49: Select Storage

9. Click Next.



of 72

Figure 50: Select Networks

10. Click Finish.

Figure 51: Finish

11. You can see new VM is added and to see the status click Home>Tasks.


12. Once installed power on the VM and open the console.



of 72

Figure 53: VM Installations

13. Go to IP Configuration

5.1.4 Oracle Virtual Box 5.2.30 Installation

This client allows you to deploy CX Collector OVA using the Oracle Virtual Box 1. Open the Oracle VM UI click File > Import Appliance.

Figure 54: Oracle VM

2. Browse to import the OVA file.

Figure 55: Select File

3. Click Import



of 72

Figure 56: Import File

Figure 57: Import InProgress

Figure 58: Open the Console

4. Select the virtual machine you just deployed and click Start.



of 72

Figure 59: VM Console Startup




of 72

5.1.5 Microsoft Hyper-V Installation

1. Click on Import Virtual Machine as highlighted on the screen.

Figure 60: Hyper-V Manager

2. Click Next> to start Import.

Figure 61: Introduction Screen



of 72

3. Browse and select the download folder. Click Next>.

Figure 62: Folder to Import

4. Select the virtual Machine and click Next>.

Figure 63: Select VM



of 72

5. Choose Copy the virtual machine (create a new unique ID), click Next>.

Figure 64: Import Type

6. Browse to select the folder for VM files. It is recommended to use default paths, click Next>.

Figure 65: Choose folders



of 72

7. Browse and select the folder to store VM hard disk. It is recommended to use default paths, Click Next>.

Figure 66: Folder to Store Virtual Hard Disks

8. Virtual Machine summary appears, if all inputs are fine, then click Finish.

Figure 67: Summary



of 72

9. After import is completed successfully a new VM is created on Hyper-V. Open the VM setting. Select the network adaptor on the left pane and choose the available Virtual Switch from drop-down.

10. Click connect as shown on the figure to start the VM.



of 72

Figure 68: Starting VM


5.2 IP Configuration

Figure 69: VM Console

1. Click Setup Password to add new password for cxcadmin



of 72

Figure 70: Set Password

2. Enter the password for cxcadmin

Figure 71: New Password

3. Re-enter the password to confirm

Figure 72: Confirm Password



of 72

4. Enter the IP Address, Subnet Mask, Gateway, and DNS server. Click Begin Configuration

Figure 73: Network Configuration

5. Confirm the entries and click YES, Continue

Figure 74: Confirmation

6. Configuration may take 15 -20 minutes to complete.

Figure 75: Configuration in Progress

7. Return to CX Portal to continue the setup. Refer Connecting to Customer Portal.

Figure 76: CX Collector Setup Completed



of 72

6 Set Up Syslog Forwarding on Cisco DNA Center

6.1 Prerequisite

Ensure that you are using supported Cisco DNA Center 1.3 or 1.2.12 Versions.

6.2 Configure Syslog Forwarding Setting

To configure Syslog Forwarding to CX Collector in Cisco DNA Center using UI, perform the following: 1. Launch Cisco DNA Center. Go to Design > Network Settings > Network. For each site, add the

CX Collector IP as the Syslog Server.

Figure 77: Syslog Server

• Once configured, all the devices associated with that site are configured to send syslog with level critical to CX Collector.

• The devices should be associated to some site for enabling the syslog forwarding from the device to CX collector.

When a syslog server setting is updated all the device associated to that site will be automatically set to default critical level.



of 72

6.3 Enabling Info Level Syslog Settings

To make Syslog Info Level visible, perform the following: 1. Navigate to Tool > Telemetry.

Figure 78: Tool Menu

2. Click Site View Tab. Expand and select a site from site hierarchy.

Figure 79: Site View

3. Select the required site and select all devices using the check box before Device name and under Actions select Optimal Visibility.

Figure 80: Actions



of 72

7 Admin Settings

This helps you to upgrade, change the CX collector credentials, add new Cisco DNA Center and update the existing DNA Center credentials.

7.1 Upgrade CX Collector

To upgrade to latest CX Collector, perform the following: 1. Click the config icon highlighted on the top right corner of the home screen.


2. Click the available patch, you can select Upgrade Now or Schedule Upgrade for later. Click Upgrade Now.

Figure 82: Upgrade Patch

Figure 83: Release Update



of 72

Figure 84: Upgrade Completed

7.2 Change the Password

To change the credentials of CX Collector, perform the following: 1. Click Change password and Settings on Admin Settings page

Figure 85: Admin Page

2. Enter the CX Collector IP and click Continue.

Figure 86: CX Collector IP

3. To accept the certificate in your browser, click Proceed to Certificate. This opens a new window in the browser to perform the steps as shown in the figure below.



of 72

Figure 87: Connecting CX Collector

4. Confirm that the collector IP you had setup is validated. Switch back to the portal window. This is Self-signed certificate to establish secure connectivity with CX Portal

Figure 88: Validated

5. Enter the Current Password, New Password and re-enter the password to confirm it. Click Save Changes.



of 72

Figure 89: Change Password Settings

7.3 Update Cisco DNA Center Credentials

To update Cisco DNA Center password, perform the following: 1. Click on Update Credentials on the Admin Settings page.

Figure 90: Admin Page



3. Enter the CX Collector Password and Cisco DNA Center Credentials.

Figure 92: Change Credentials



of 72

7.4 Add Cisco DNA Center

To Add new Cisco DNA Center, perform the following: 1. Click Add Cisco DNA Center on Admin Settings page

Figure 93: Admin Settings Page



3. Enter CX Collector Password, enter Cisco DNA Center FQDN or IP Address, and credentials.

4. Select the location from the list to connect to Cisco DNA Center and you can choose either run now or schedule the collection later.

5. You can schedule the ongoing Inventory collection, click Continue.

6. Click Add another Cisco DNA Center for collection and repeat steps 3, 4 and 5. You can add up to 10 Cisco DNA Center clusters.



of 72

Figure 95: Connect Cisco DNA Center



of 72

8 Security

8.1 Physical Security

You need to deploy CX Collector OVA image in a secured VMWare server firm. The OVA is shared securely through cisco software download center.

Bootloader (Single user mode) password is set with a randomly unique password. User must refer FAQ to set this bootloader (single user mode) password.

8.2 User Access

CPP portal (User) uses the CX-Collector APIs exposed to access the features/functionalities of the CX Collector.

User can login to the appliance only through ssh.

8.3 Account Security

On deployment, user-account ‘cxcadmin’ is created. User is forced to set a password for the same during the initial configuration.

Same cxcadmin user /credentials are used to access both the CX-Collector APIs and to connect to appliance over ssh.

This password follows the security policy and is one-way hashed. It has an expiry time of 90 days.

8.4 Network Security

CX Collector VM can be accessed using ssh with cxcadmin user credentials.

CX Collector VM can access the exposed APIs through token based authentication. The token is obtained as part of the registration process. On expiry of this token, it can be regenerated by providing the cxcadmin credentials.

Incoming ports are restricted to 22 (ssh), 514(Syslog) and 443 (HTTPS).

8.5 Authentication

There are two ways of authentication:

• Password based authentication: Appliance maintains a single user - ‘cxcadmin’ which enables the user to authenticate and communicate with the CX Collector.

The password of this user is one-way hashed using SHA-512 algorithm.

• Token based authentication:

Application users can obtain the token, on successful registration. The token is generated using RSA-256 algorithm and is uniquely identified in every appliance. On expiry of this token, it can be regenerated by providing the cxcadmin credentials. This token has an expiry time of 30min from the time of creation and can be used as an authentication mechanism, for communicating with the appliance.

8.6 Hardening

CX Collector appliance follows CIS hardening standards and has achieved high scores.



of 72

8.7 Data Security

CX Collector appliance does not store any customer personal information.

Device credential application (running as one of the pods) stores encrypted Cisco DNA Center server credentials inside secured database. Cisco DNA Center collected data is not stored in any form inside the appliance. The data collected is uploaded to the backed, soon after the collection is complete, and the data is purged.

8.8 Data Transmission

As shown in the architecture diagram, a secure TLS 1.2 channel is established between CX-Collector and RP (Reverse proxy) server.

The Reverse proxy does appliance user authentication and it is based on the user_id and the password. The reverse proxy checks against cisco’s LDAP server. The user_id is not the user-id of someone who logged into CX-Collector, but it is the CCO-ID assigned to the appliance itself and is not known externally. So, each appliance has been assigned its own CCO-ID and password.

Following ciphers are supported: AES256-SHA, AES128-SHA

8.9 Logs and Monitoring

Logs do not contain any form of sensitive information. Audit logs capture all security sensitive actions performed on the collector appliance.

8.10 Security Summary

Security Features Description

Bootloader Password Bootloader (Single user mode) password is set with a randomly unique password. User must refer FAQ to set his bootloader (single user mode) password.

User Access ssh (cxcadmin credentials used for authentication)

API communication (requires token for authentication)

User Accounts cxcadmin (Only one user account used)

cxcadmin password policy • Password is oneway hashed using SHA-256 and stored securely

• Min 8 characters +contain three of the following categories: upper cases, lower case, numbers and special characters.

Token Base Authentication • Application users can obtain the token, on successful registration.

• This token is uniquely generated using RSA 256 algorithm.

• On expiry of this token, it can be regenerated by providing the cxcadmin credentials.

• This token has an expiry time of 30min from the time of creation and can be used as an authentication mechanism, for communicating with the appliance.



of 72

Security Features Description

ssh login password policy • Min 8 characters +contain three of the following categories: upper cases, lower case, numbers and special characters.

• 5 failed login attempts will lock the box for 30min. - Expiry – 90 days

Ports Open Incoming Ports – 514(Syslog), 443(HTTPS) and 22 (ssh)

Data Security No Customer information stored.

No Device data stored.

Cisco DNA Center server credentials encrypted and stored in the database.

Data transfer from CX-Collector to Cisco Backend

Refer Data Transmission



of 72

9 Frequently Asked Questions

9.1 CX Collector

Deployment

Q With "Re-install" option, can the user deploy the new collector with new an IP Address? A Yes Q Flavours of installable? A

• OVA

• VHD Q What is the environment on which the installable can be deployed? A

• OVA

o VMWare ESXi version 5.5 or above

o Oracle Virtual Box 5.2.30 or above

• VHD

o Hyper-V

Q Can CX Collector detect IP address in a DHCP environment? A Yes, in case of DHCP environment, the IP address assignment during IP configuration is taken

care. However, the IP address change expected for the CX Collector at any point in future is not supported. Also, the customer is recommended to reserve the IP for the collector in their DHCP environment.

Q Does CX Collector support both IPv4 and IPv6 configuration? A No, only IPV4 is supported. Q During IP configuration, is IP address validated? A Yes, IP address syntax and duplicate IP address assignment will be validated. Q What is the approximate time taken for the OVA deployment and IP configuration? A The OVA deployment depends on the speed of the network to copy the data. The IP configuration

takes approximately 15-20 minutes that includes Kubernetes and container creations. Q Is there any limitation with respect to any hardware type? A Host machine on which OVA is deployed must meet the requirements provided as part of CX

portal setup. The CX collector is tested with VMware/Virtualbox running on hardware with Intel Xeon E5 processors with vCPU to CPU ratio set at 2:1. If less powerful processor CPU or larger ratio is used, the performance might degrade.

Q Which is the command that shows services and their status; and how does the output look like? A The command is kubectl get pods and the output looks as below:

The actual values might differ than the instance shown below.



of 72

Figure 96: Pods

Authentication and Proxy configuration

Q What is the default user of the CX Collector application? A cxcadmin Q How the password is set for the default user? A Password is set during IP configuration and the user is provided an option to reset the password

from the customer portal during CX Collector setup Q Is there option available to reset the password after Day 0? A Yes, the password can be reset at any time from Admin Settings→Change Password & Settings Q What are the password policies to configure CX Collector? A

• Password Maximum Age (length) set to 90 days

• Password Minimum age (length) set to 8

• Password Maximum length 127 characters.

• At least one upper case and one lower case should be provided.

• Should Contain at least one special character (for example, !$%^&*()_+|~-=\`{}[]:";'<>?,/). • The following characters should not be not permitted

➢ Special 8-bit characters (for example, ¬£, √Å √´, √¥, √ë, ¬ø, √ü)

➢ Spaces

• The password should not be the last recently used 10 passwords.

• Should not contain regular expression ie should not contain the following words or derivatives

thereof cisco, sanjose, and sanfran Q How to set Grub password?

A To set the Grub Password, perform the following: 1. Login to the CX collector console



of 72

2. ssh to cxcadmin and provide the password

3. Execute sudo su command and provide the password

4. Execute the command grub-mkpasswd-pbkdf2 and set the GRUB password. Hash of the

provided password will be printed, copy the content.

5. vi to the file /etc/grub.d/00_header. Navigate to the end of file and replace the hash output

followed by the content password_pbkdf2 root ***** with the obtained hash for the

password you got in step 3

6. Save the file with the command :wq!

7. Execute the command update-grub

Q What is the password for user cxcadmin? A The password set during the IP configuration is the password for cxcadmin user. If the user has

reset the password for cxcadmin in the portal during IP configuration, then the new password will be used for logging in to the collector.

Q What is the expiry period for password of cxcadmin? A The password expiry in 90 days. Q Does the system disable the account after consecutive unsuccessful login attempts? A Yes, the account gets disabled after 5 consecutive unsuccessful attempts. The lockout period is

30 minutes. Q How do we configure proxy? A Proxy configuration can be set from the customer portal. Refer Proxy Server Q Can proxy be configured later the box? A No. Q Does proxy host support both hostname and IP? A Yes, but in case of hostname user should provide the DNS IP during IP configuration, otherwise

proxy hostname resolution fails, and thus the registration fails. Q Are both IPv4 and IPv6 supported for proxy? A No, only IPv4 is supported. Q What happens in initial CX Collector setup? A As part of the initial setup, a license would be generated with your Cisco account and applied on

the CX Collector which establishes the communication channel between CX Collector and Cisco backend.

Secure Shell SSH

Q What are the ciphers supported by ssh shell? A [email protected], [email protected], [email protected],

aes256-ctr, aes192-ctr, aes128-ctr Q How to login to console? A Follow the steps to login: 1. Login as cxcadmin user.

2. Provide the cxcadmin password that is set.

3. Run command sudo su.

4. Provide the same password.

Q Are ssh logins logged?

mailto:[email protected]



of 72

A Yes, it is logged as part of the var/logs/audit/audit.log Q What is the idle session out time? A ssh session timeout occurs if the collector is idle for five minutes

Ports and Services

Q What are the ports kept open by default on the CX collector? A

• Outbound port: The deployed CX Collector can connect to concsoweb-prd.cisco.com on HTTPS port 443 or via a proxy to send data to Cisco. The deployed CX Collector can connect to Cisco DNA Center on HTTPS port 443

• Inbound port: For local management of the CX Collector, port 443 and port 22 should be accessible.

9.2 CX Collector Connection with Cisco DNA Center

Q What is the purpose and relationship of Cisco DNA Center with CX collector? A Cisco DNA Center is the collector which manages the customer premise network devices. CX

Collector collects the inventory information of the devices from the configured Cisco DNA Center and uploads the inventory information that is available as “Asset View” in customer portal.

Q Where can user provide Cisco DNA Center details on the CX portal? A During the CX Customer Portal setup, the user will be prompted to provide the Cisco DNA Center

details. Q How many Cisco DNA Centers can be added? A 10 Cisco DNA Centers. Q What role the Cisco DNA Center user should have? A The user can have of any of these role admin or observer Q How are the Cisco DNA Center details stored in CX Collector? A Cisco DNA Center credentials are encrypted using AES-256 and stored in CX Collector database.

CX Collector database is protected with a secured user ID and password. Q What kind of encryption will be used while accessing Cisco DNA Center API from CX Collector? A HTTPS over TLS 1.2 is used for the communication between Cisco DNA Center and CX

Collector. Q What are the operations performed by CX Collector on the integrated Cisco DNA Center

collector? A

• CX Collector collects data that Cisco DNA Center has about the network devices.

• It uses the Cisco DNA Center command runner interface to talk to end devices and execute CLI commands (show command).

• no config change commands are executed. Q What are default data collected from Cisco DNA Center and uploaded to backend? A

• Network Entity

• Modules

• Show version

• Config

• Device image information

• Tags



of 72

Q What are the additional data collected from Cisco DNA Center and uploaded to SDP? A You get all the information here. Q How is the inventory data uploaded to backend? A CX Collector uploads the data via TLS 1.2 protocol to cisco backend server (https://concsoweb-

prd.cisco.com). Q What is the frequency of inventory upload? A Everyday collection gets triggered at 0000 hrs. UTC and gets uploaded to backend. Q Can the user re-schedule inventory? A Yes, an option is available to modify the schedule information from Admin Settings → Policy.

Q What does "Cisco DNA Centre should be reachable from Collector box" means before the

user adds/configures the Cisco DNA Centre in the collector?

A The collector tries to connect Cisco DNA Centre via port 443. Make sure, this connection is successful.

9.3 CX Collector Used Diagnostic Scan

Q What are the commands executed on the device for scan? A The commands that need to be executed on the device for the scan is dynamically determined

during the scanning process. The set of commands can change over time, even for the same device (and not in control of Diagnostic Scan).)

Q Where are the scan results stored and profiled? A The scanned results are stored and profiled in cisco backend. Q Are the duplicates (By hostname or IP) in Cisco DNA Center, added to Diagnostic Scan when

Cisco DNA Center source is plugged in? A No, the duplicates will be filtered and only the unique devices will be extracted. Q What happens when one of the command scan fails? A The device scan will be completely stopped and will be marked as unsuccessful. Q Who stores the scan results? A IRONBANK

9.4 CX Collector System Logs

Q List of health information that is sent to the CX Portal? A Application logs, Pod status, Cisco DNA Center details, audit logs, system details, and hardware

details. Q What system details and hardware details are collected? A Sample output:

system_details":{

"os_details":{

"containerRuntimeVersion":"docker://18.6.1",

"kernelVersion":"4.15.0-55-generic",

"kubeProxyVersion":"v1.11.8",

"kubeletVersion":"v1.11.8",

https://cisco.box.com/s/9hlsfyjcj2ba4sr53edjyk7w70guzczy

https://concsoweb-prd.cisco.com/

https://concsoweb-prd.cisco.com/



of 72

"machineID":"65a9626a463d44bb85efcc31d1a7d2d0",

"operatingSystem":"linux",

"osImage":"Ubuntu 18.04.2 LTS",

"systemUUID":"423E0DEF-081D-8E99-AF98-65641272633C"

},

"hardware_details":{

"total_cpu":"8",

"cpu_utilization":"110.2%",

"total_memory":"16035MB",

"free_memory":"324MB",

"hdd_size":"214G",

"free_hdd_size":"194G"

}

}

}

Q How is the health data sent to backend? A With CX Collector, the health service (serviceability) streams the data (via kafka), after which

websocket connection is established with the SDP and from there the data is streamed (via kafka).

Q What are the types of uploads available? A Types of uploads are Full upload and Partial upload.

• Full upload - Scheduled for every hour. Collects and sends details like Pod status, Cisco DNA Center details, audit logs, system details, and hardware details.

• Partial upload – Scheduled for every five minutes. Difference with the last collected information is uploaded.



of 72

10 Troubleshooting

Issue: Not able to access the configured IP. Solution: Execute ssh using configured IP. If you get connection timeout the possible reason might be the IP misconfiguration. So, reinstall by configuring a valid IP. This can be done via portal with the reinstall option provided in the Admin Setting page. Issue: How to verify if the services are up and running after the registration? Solution: Execute the below command and check if the pods are up and running. 1. ssh to the configured IP as cxcadmin. 2. Provide the password. 3. Execute sudo su and it will prompt for the password

4. Provided the same password set for cxcadmin 5. Execute the command kubectl get pods

The pods can be in any of the state say (running, Initializing, Container creating) approx. 20 minutes after, which the pods should be in running state. If you observe the states other than running and PodInitialaizing, check the pod description with the below command

Kubectl describe pod <podname>

The output will have the information on the pod status. Issue: kubectl commands gets failed and if it shows the error as “The connection to the server X.X.X.X:6443 was refused - did you specify the right

host or port”

Solution:

• Verify for the resource availability. [example: CPU, Memory]

• Wait for the Kubernetes service to start

Issue: If any of the default pods are not available after installation, execute the below commands:

helm search

helm repo update

If the error is as below,

“Unable to get an update from the "remote-node" chart repository

(https://insight.engine/helm):error converting YAML to JSON: yaml: line 1: mapping

values are not allowed in this context

Update Complete. ⎈ Happy Helming!⎈ “

Solution: Restart ngnix with the command

sh /opt/cisco/ie/ie-config/startnginx.sh



of 72

Issue: How to get details of collection failure for a command/device Solution:

• Execute kubectl get pods and get the collection pod name.

• Execute kubectl logs <collectionPodName> to get the command/device specific details.

Issue: kubectl command not working with error [authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate has expired or

is not yet valid, x509: certificate has expired or is not yet valid]

Solution:

• Use the command kubeadm alpha certs check-expiration to check the certificate

expiration details

• Use the command kubeadm alpha certs renew all to renew the certificate for one year

Issue: After restarting VM/Collector, pods are not in running state. Solution: 1. Execute command kubectl get pods. If the output of the command looks like the below

image even after 60-70 min, then continue with step 2.

2. Execute the command kubectl logs nfs-provider-nfs-server-provisioner-0

and check if this error message NFS server Exited Unexpectedly with err:

ganesha.nfsd failed with error: exit status 2, output: is displayed

continuously in the logs. If yes, then continue with next steps. 3. Copy the configuration file using the command cp -rf /opt/cisco/ie/volume/nfs-

data/vfs.conf_1.1.0.0<datetime>* /opt/cisco/ie/volume/nfs-data/vfs.conf

4. Choose the correct filename (ex.: vfs.conf_1.1.0.0_2020_03_18_06_21_AM).

5. Execute the command nohup docker container prune -f && sleep 10 && kubectl get pods | cut -d' ' -f 1 | xargs kubectl delete pod &

6. Execute the command kubectl get pods to verify the pod status and pods should be in

Running state after 10-15 minutes.

http://1.1.0.0/



of 72

10.1 Collection Failure Responses

Collection failure cause can be any constraints or issues seen with the added controller or devices present in the controller. The below table has the error snippet for few use cases seen under Collection microservice during the collection process.

Use Case Log Snippet in collection microservice

If the requested device is not found in Cisco DNA Center

{

"command": "show version",

"status": "Failed",

"commandResponse": "",

"errorMessage": " No device found

with id 02eb08be-b13f-4d25-9d63-

eaf4e882f71a "

}

If the requested device is not reachable from Cisco DNA Center

{


"status": "Failed",


"errorMessage": "Error occured

while executing command : show

version\nError connecting to device

[Host: 172.21.137.221:22]No route

to host : No route to host "

}

If the requested device is not reachable from Cisco DNA Center

{


"status": "Failed",




version\nError connecting to device

[Host: X.X.X.X]Connection timed

out: /X.X.X.X:22 : Connection timed

out: /X.X.X.X:22"

}

If the requested command is not available in device

{

"command": "show run-config",

"status": "Success",

"commandResponse": " Error

occured while executing command :

show run-config\n\nshow run-

config\n ^\n% Invalid input

detected at \u0027^\u0027

marker.\n\nXXCT5760#",

"errorMessage": ""

}

If the requested device is not having SSHv2 and Cisco DNA Center tries to connect the device with SSHv2

{




of 72

Use Case Log Snippet in collection microservice

"status": "Failed",




version\nSSH2 channel closed :

Remote party uses incompatible

protocol, it is not SSH-2

compatible."

}

If command is disabled in Collection microservice

{

"command": "config paging

disable",

"status": "Command_Disabled",

"commandResponse": "Command

collection is disabled",

"errorMessage": ""

}

If the Command Runner Task failed and task URL is not returned by Cisco DNA Center

{


"status": "Failed",


"errorMessage": "The command

runner task failed for device %s.

Task URL is empty."

}

If the Command Runner Task failed to get created in Cisco DNA Center

{


"status": "Failed",



runner task failed for device %s,

RequestURL: %s. No task details."

}

If the Collection microservice not receiving response for a Command Runner request from Cisco DNA Center

{


"status": "Failed",




RequestURL: %s."

}

If Cisco DNA Center is not completing the task within the configured timeout (5 mins per command in Collection microservice)

{


"status": "Failed",


"errorMessage": "Operation

Timedout. The command runner task



of 72

Use Case Log Snippet in collection microservice failed for device %s, RequestURL:

%s. No progress details."

}

If the Command Runner Task failed and file ID is empty for the submitted task by Cisco DNA Center

{


"status": "Failed",




RequestURL: %s. File id is empty."

}

If the Command Runner Task failed and file ID tag is not returned by Cisco DNA Center

{


"status": "Failed",




RequestURL: %s. No file id

details."

}

If the device is not eligible for command runner execution

{


disable",

"status": "Failed",


"errorMessage": "Requested

devices are not in inventory,try

with other devices available in

inventory"

}

If the command runner is disabled for the user {


"status": "Failed",


"errorMessage":

"{\"message\":\"Role does not have

valid permissions to access the

API\"}\n"

}

10.2 Diagnostic Scan Failure Responses

Scan failure and the cause can be from any of the listed components When the user initiates a scan from the portal, occasionally it results as “failed: Internal server error” The cause for the issue might be any of the listed components

• Control Point



of 72

• Network Data Gateway

• Control Point Agent

• Diagnostic Scan

• CX Collector Microservice [devicemanager, collection]

• Cisco DNA center

• Reverse Proxy

• APIX

• Mashery

• Ping Access

• IRONBANK

• IRONBANK GW

• BDB The below table has the error snippet seen under Collection microservice and Control Point Agent microservice logs that occurs due to the issues/constraints with the components To see the logs, perform the following: 1. Login to the CX collector console 2. ssh to cxcadmin and provide the password 3. Execute sudo su command and provide the password

4. Execute kubectl get pods

5. Get the pod name of collection and controlpoint Agent microservice 6. To verify the collection microservice/controlpointAgent logs

Execute kubectl logs <collectionpodname>

Execute kubectl logs <controlpointagent>

Use case Log snippet in collection microservice

The device might be reachable and supported, but the commands to execute on that device is blacklisted on the Collection microservice

{


disable",

"status": "Command_Disabled",

"commandResponse": "Command

collection is disabled",

}

If the device which is attempted for scan is not available.

Occurs in a scenario, when there is a sync issue between the components such as portal, diagnostic Scan, CX component, and Cisco DNA Center

No device found with id 02eb08be-b13f-4d25-9d63-eaf4e882f71a

If the device that is attempted for scan is busy, (in a scenario) where the same device is been part of other job and no parallel requests are handled from Cisco DNA Center for the device.

All requested devices are already being queried by command runner in another session. Please try other devices".

If the device is not supported for scan Requested devices are not in inventory, try with other devices available in inventory

If the device which is attempted for scan is unreachable

"Error occurred while executing command: show udi\nError connecting to

device [Host: x.x.x.x:22] No route

to host : No route to host

If Cisco DNA Center is not reachable from collector or Collection microservice of the collector is not receiving response for a

{


"status": "Failed",



of 72

Use case Log snippet in collection microservice

Command Runner request from Cisco DNA Center




RequestURL: %s."

}

Use Case Log snippet in Control Point Agent microservice

If the scan request has schedule details missing Failed to execute request

{"message":"23502: null value in

column \"schedule\" violates not-

null constraint"}

If the scan request has device details missing Failed to create scan policy. No valid devices in the request

If the connection between the CPA and connectivity is down

Failed to execute request.

If the requested device for scan is not available in Diagnostic Scans

Failed to submit the request to scan. Reason = {\"message\":\"Device with

Hostname=x.x.x.x' was not found\"}



of 72

11 Portal Support

Use the highlighted button on the screens to get answers for you queries

Figure 97: Portal Support

Select the classification and enter the problem faced and submit the query.

Figure 98: Contact Support



of 72

12 Collector Patch Installation

This section is applicable only to existing customers who have already deployed the CX Collector prior to the March 2020 Collector release. Customers who are installing the March 2020 CX Collector for the first time do not need to apply this patch.

This patch is to upgrade the collectors for existing customers with CX Collector version 0.9.0/0.9.1 to the latest release (CX Collector 0.9.2). This is a one-time activity (manual installation).

12.1 Release Artifacts

File Name Download Link Forum Size

iepa-1.1.0.0-lnx64-b79.zip Link cb145f065155bb6e50c5858640b2a589

469580757 bytes

12.2 Patch Installation

To install the patch, perform the following: 1. Download the patch here.

2. Unzip the file using the command unzip iepa-1.1.0.0-lnx64-b79.zip

3. Change directory using the command cd iepa-1.1.0.0-lnx64-b79

4. Execute ./install.sh

Figure 99: Successful execution

5. The patch installation may take around 20-30 minutes to complete and then machine will reboot.

6. After reboot, it may take up to 20 minutes for all the applications and processes to be up and running.

7. Execute kubectl get pods to verify if all the pods are in running state.

8. If any of the pods are in ‘Error’ or ‘CrashLoopBackOff’ state (as shown in Figure 100), follow the Troubleshooting steps.

https://software.cisco.com/download/home/286324649/type/286324666/release/0.9.2

https://software.cisco.com/download/home/286324649/type/286324666/release/0.9.2



of 72

Figure 100: Pods in ‘Error’ or ‘CrashLoopBackOff’ state

12.3 Troubleshooting steps

1. Copy the configuration file using the command cp -rf /opt/cisco/ie/volume/nfs-data/vfs.conf_1.1.0.0<datetime>* /opt/cisco/ie/volume/nfs-data/vfs.conf

Choose the correct filename (example: vfs.conf_1.1.0.0_2020_03_18_06_21_AM).

2. Execute nohup docker container prune -f && sleep 10 && kubectl get pods | cut -d' ' -f 1 | xargs kubectl delete pod &

3. Execute kubectl get pods to verify the pod status. It takes 10 -15 minutes to reach Running

state.

Figure 101: Pods in ‘Running’ state

http://1.1.0.0/



of 72

13 Performance Test Summary

• Burst tests of 300 concurrent users.

• Scenario 1: 60 uploads to single Cisco DNA Center.

• Scenario 2: Multi Cisco DNA Center with 10 accounts, four Cisco DNA Centers per account in total 40 Cisco DNA Center standalones that is equivalent to small and large network sizes.

• Scenario 3: Scenario 1 and Scenario 2 along with syslogs and AFMs. (Also manually initiate the ML job that runs for 12hrs); uploads test will cover all the compliance scenarios (opt-in compliance for all the 10 accounts).

• Scenario 4: NeoLoad 30 concurrent users for all the updated tracks in present release.

• Scenario 5: Scenario 3 and Scenario 4 that comprises overall testing.



of 72

14 Addendum

Terms Description

Oneway Hashing Mechanism involving an algorithm that turns messages or text into a unique fixed string which is nearly impossible to derive the original text from the string.

Token Based Authentication Users enter their password in order to obtain a token. Once this token has been obtained, the user can offer the token to access specific resource for a time period.

Hardening Hardening is the process of securing a system by reducing its surface of vulnerability.

CIS CIS (Center for Internet Security) benchmark is a recognized global standard and best practices for securing IT systems and data against attacks.

Ciphers A cipher (or cypher) is an algorithm for performing encryption or decryption.

Bootloader password Password that protects the Linux boot loader.

cx collector...march 19, 2020 collector overview cisco confidential.all printed copies and duplicate...

Documents