cge ivt master test plan

Date Modified: August-18-2014 CGE IVT Master Test Plan

Copyright 2014 Cisco Systems 1

Document Number EDCS-1291267, Rev 14 Based on Template EDCS-206096 Rev.40 Created By Anuj Dewangan (adewanga) –

TME (IOTG), Malay Thaker (mathaker) – PLM (IOTG)

Project Cisco DevNet – Cisco Enabled Grid Devices

CGE IVT Master Test Plan Reviewers/Approvers

Department Name/Title

IOTG Product Management/Business Development

Patrick Grossetete (pgrosset)/TME, Marty Collins (mcollins)/TME, Gaurav Agarwal (gauraaga)/TME, Felix Davis (felixdav)/PLM, Brian Mansfield (bmansfie)/PM, Nitin Nayar (nnayar)/PM, Joe Huber (joehuber)/BDM

IOTG Engineering (Development) Wei Hong (wehong)/Director.Engineering, Jonathan Hui (johui)/Technical Leader, Phil Buonadonna (pbuonado)/Technical Leader, Chris Wilson (chwilso3)/Hardware Engineer, Yuwen Lan (yulan)

IOTG Engineering (Support/Test) Arun Annavarapu (aannavar)/Software Engineer, Jonathon Paul (jopaul)/Technical Leader, Linda Yang (linyan)/Technical Leader



Modification History

Revision Name User Id Date Comments

0.1 Anuj Dewangan adewanga 09/11/2013 Added test cases without detailed procedure

0.2 Anuj Dewangan adewanga 09/18/2013 Added test case details + edits

1.0 Anuj Dewangan adewanga 09/19/2013 Malay edits + Added for Internal Review (EDCS-1291267 Rev 8)

1.1 Anuj Dewangan

Malay Thaker

adewanga

mathaker

12/04/2013 • Review comments

• Additional details in test cases based on requirements document

1.2 Anuj Dewangan

Malay Thaker

adewanga

mathaker

01/23/2014 • Re-arranged sections

• Added IVT recertification requirements

1.3 Anuj Dewangan

adewanga

01/31/2014 • Added new test case CGE_IVT_APP_TC010

2.0 Anuj Dewangan adewanga 05/16/2014 • Added multi-hop PAN migration topology

• Added requirement for pre-IVT testing as entry criteria for IVT

• Added test cases for two phases of certification:

A] Pre-IVT Testing

B] IVT Testing

3.0 Linda Yang

Anuj Dewangan

linyan

adewanga

08/18/2014 • Added dual-phy topology and test cases

• Added PLC test cases for modulation and LMAC OTA

• Modifications to PON/PRN test cases

• Added test case to verify high QoS markings are not present

• Added test case to verify maximum 800 byte L3 application packet size

Printed September 11, 2002

This copy is uncontrolled. Look on-line for controlled version.



• Modified test case to verify relative Tx-power versus iTron meters and Tx-power configurability (CGE_IVT_SIG_TC001)

• Removed QoS considerations, as they are part of the requirements document

• Updated content for dual-phy design



Table of Contents 1 Introduction ............................................................................................................................................ 7

1.1 Connected Grid Endpoints and Field Area Network ....................................................................... 7 1.2 System Development Kit (SDK) and Cisco DevNet ....................................................................... 9 1.3 CG-REDI and partner development infrastructure ....................................................................... 11

2 Test Coverage ..................................................................................................................................... 13 2.1 What Will Be Tested ..................................................................................................................... 14 2.2 What Will Not Be Tested .............................................................................................................. 15

3 Test Setups ......................................................................................................................................... 16 3.1 MCU Diagnostics Setup ............................................................................................................... 16 3.2 One CGE in a PAN ....................................................................................................................... 17 3.3 Four (at least) CGEs in a PAN ..................................................................................................... 17 3.4 Large scale, multi-hop, multi-vendor CGE PAN ........................................................................... 18 3.5 Multi-hop PAN Migration .............................................................................................................. 18 3.6 Four (at least) multi-vendor CGEs in a PAN ................................................................................ 19 3.7 Multi-phy CGEs in a PAN ............................................................................................................. 19

4 Test and Certification Approach and Considerations .......................................................................... 21 4.1 Test Phase Entry Criteria ............................................................................................................. 21 4.2 IVT Considerations specific to a certification ................................................................................ 22 4.3 Test Execution tracking and reporting .......................................................................................... 23 4.4 Defect Tracking ............................................................................................................................ 23 4.5 Requirements Traceability Approach ........................................................................................... 23 4.6 IVT Certification Elements ............................................................................................................ 23 4.7 IVT Recertification criteria ............................................................................................................ 24 4.8 IVT Certification Numbering scheme ............................................................................................ 25 4.9 IVT Certification Report ................................................................................................................ 25 4.10 IVT Deliverables ......................................................................................................................... 26 4.11 Test Exit/Certification Criteria ..................................................................................................... 27 4.12 Performance test (PR) and Scalability test (SC) ........................................................................ 27 4.13 Pre-IVT Testing Matrix ............................................................................................................... 28 4.14 IVT Testing Matrix ...................................................................................................................... 31

5 Test Resources ................................................................................................................................... 36 5.1 People .......................................................................................................................................... 36 5.2 Hardware ...................................................................................................................................... 36 5.3 Firmware ...................................................................................................................................... 36 5.4 Software ....................................................................................................................................... 37 5.5 Test Equipment ............................................................................................................................ 37 5.6 Test Tools (software) .................................................................................................................... 37

6 Risks, Mitigations, and Contingencies ................................................................................................. 39 7 Detailed Test Cases (Pre-IVT) ............................................................................................................ 40

7.1 CGE HW Schematics and diagnostics (Pre-IVT): ........................................................................ 40 7.2 Workforce enablement (Pre-IVT): ................................................................................................ 41 7.3 CG-Mesh Connectivity (Pre-IVT): ................................................................................................. 45 7.4 CG-Mesh Security (Pre-IVT): ....................................................................................................... 54 7.5 Remote Management and Notifications (Pre-IVT) ....................................................................... 56 7.6 CGE Application (Pre-IVT): .......................................................................................................... 63 7.7 Fault resiliency and availability: .................................................................................................... 70

8 Detailed Test Cases (IVT) ................................................................................................................... 73 8.1 CGE HW Schematics and diagnostics (IVT): ............................................................................... 73 8.2 RF/PLC physical and MAC layer (IVT): ........................................................................................ 74 8.3 Workforce enablement (IVT): ....................................................................................................... 78 8.4 CG-Mesh Connectivity (IVT): ....................................................................................................... 83 8.5 CG-Mesh Security (IVT): .............................................................................................................. 88



8.6 Remote Management and Notifications (IVT): ............................................................................. 92 8.7 CGE Application (IVT): ................................................................................................................. 98 8.8 Fault resiliency and availability (IVT): ......................................................................................... 105 8.9 Soak and Performance/Stress: .................................................................................................. 107

9 References ........................................................................................................................................ 110



Table of Figures Figure 1.1 Protocol Stack for Connect Grid Endpoint SDK – IEEE 802.15.4g/e ......................................................... 7 Figure 1.2 Protocol Stack for Connect Grid Endpoint SDK – IEEE 1901.2 PLC ......................................................... 8 Figure 1.4 CGE SDK – NIC (PPP) model ................................................................................................................... 10 Figure 1.5 CGE SDK – Integrated Execution (IE) model ........................................................................................... 10 Figure 1.6 CG-REDI architecture for partner development ......................................................................................... 11 Figure 1.7 Partner Premises – “Thin Starter Kit” topology ......................................................................................... 12 Figure 3.1 IVT Topology 1: CGE diagnostics ............................................................................................................. 16 Figure 3.2 IVT Topology 2: 1 CGE in a PAN ............................................................................................................. 17 Figure 3.3 IVT Topology 3: 4 CGEs in a PAN ............................................................................................................ 17 Figure 3.4 IVT Topology 4: Large scale, multi-hop, multi-vendor CGE PAN ........................................................... 18 Figure 3.5 IVT Topology 5: Multi-hop PAN Migration .............................................................................................. 18 Figure 3.6 IVT Topology 6: 4 CGEs in a multi-vendor CGE PAN ............................................................................. 19 Figure 3.7.1 IVT Topology 7.1: 4 CGEs in a multi-Phy CGE PAN ........................................................................... 19 Figure 3.7.2 IVT Topology 7.2: 4 CGEs in a multi-Phy CGE PAN ........................................................................... 20

Table of Tables Table 4.6.1: IVT elements: ........................................................................................................................................... 23 Table 4.14.1: CGE diagnostic tests (Pre IVT): ............................................................................................................ 28 Table 4.14.2: Workforce enablement (Pre IVT): ......................................................................................................... 28 Table 4.14.3: CG-Mesh Connectivity (Pre IVT): ........................................................................................................ 28 Table 4.14.4: Mesh Security (Pre IVT): ....................................................................................................................... 29 Table 4.14.5: Remote Management and Notifications (Pre IVT): ............................................................................... 29 Table 4.14.6: CGE Application (Pre IVT): .................................................................................................................. 30 Table 4.14.7: Fault resiliency and availability (Pre IVT): ........................................................................................... 30 Table 4.15.1: CGE HW Schematics and diagnostics (IVT): ....................................................................................... 31 Table 4.15.2: RF/PLC physical and MAC layer (IVT): ............................................................................................... 31 Table 4.15.3: Workforce enablement (IVT): ............................................................................................................... 31 Table 4.15.4: CG-Mesh Connectivity (IVT): ............................................................................................................... 32 Table 4.15.5: Mesh Security (IVT): ............................................................................................................................. 32 Table 4.15.6: Remote Management and Notifications (IVT): ..................................................................................... 33 Table 4.15.7: CGE Application (IVT): ........................................................................................................................ 33 Table 4.15.8: Fault resiliency and availability (IVT): .................................................................................................. 34 Table 4.15.9: Soak and Performance/Stress (IVT): ..................................................................................................... 35 Table 5.2.1: Hardware requirement for certification: .................................................................................................. 36 Table 5.3.1: Firmware requirements for certification: ................................................................................................. 36 Table 5.4.1: Software requirements for certification: .................................................................................................. 37 Table 5.5.1: Test equipment requirements for certification: ........................................................................................ 37 Table 5.6.1: Test tool requirements for certification: .................................................................................................. 37 Table 6.1: Risks ............................................................................................................................................................ 39



1 Introduction

This document serves as the Master Test Plan for inter-operability verification testing (IVT) of Connected Grid Endpoints (CGE) developed by Cisco Connected Grid ecosystem partners, engaged through the Cisco-Enabled Grid Devices Technology Center within Cisco DevNet (henceforth, referred to as ‘DevNet’). It defines the scope, approach, resources, schedule, risks/mitigations, entry/exit criteria and certification requirements for the IVT performed on the CGE and the Field Area Network implementation. Detailed test plans for each IVT certification testing for a CGE partner will be derived from this document and will add or remove test cases based on implementation specifics.

This plan, including test cases, will be reviewed and approved to assure completeness of the testing and to determine the testing schedule. Once testing starts all test cases can be managed within a test case management system. If major changes to testing scope are made, once execution starts, then the test plan, including test cases, will be re-reviewed and approved. The test results review will become the final validation of the test plan along with any other exit criteria.

1.1 Connected Grid Endpoints and Field Area Network Cisco supports two major communication technologies in the connected grid endpoint space for the last mile: Radio Frequency (RF) mesh and Power Line Communications (PLC) mesh. Vendor partners also have an option to create an endpoint supporting both the technologies. Such an endpoint is referred to as ‘Dual-phy’ (as a reference to having two different physical and MAC layers for communication, but a single layer 3 instance).

Cisco’s implementation of these technologies is based on and driven by industry standards. The following diagrams illustrate the protocol stack implemented for each of the two technologies and the associated standards:

Figure 1.1 Protocol Stack for Connect Grid Endpoint SDK – IEEE 802.15.4g/e



Figure 1.2 Protocol Stack for Connect Grid Endpoint SDK – IEEE 1901.2 PLC

A Cisco Connected Grid Endpoint (CGE) needs to implement one or both of the above protocol stacks and needs to be appropriately configured to be able to join and communicate with a Neighborhood Area Network (NAN) rooted at a Cisco’s Connected Grid Router (CGR) 1000 series.

A CGE connected to a NAN/CG mesh (RF, PLC or dual-phy) is capable of end-to-end layer 3 communication using IPv6. When a CGE attempts to join a CG-Mesh network, it is required to authenticate itself to the network, get an IPv6 address along with options and prefix-delegation if required, register itself to network management services using CoAP Simple Management Protocol (CSMP), and communicate with required application servers (collection engines, SCADA master, MDMS etc) to deliver grid functionalities. Typically the network services are hosted in a utility Network Operations Center (NOC). The following topology diagram shows the end-to-end solution architecture of a Field Area Network (FAN) with the Wide Area Network (WAN) and Neighborhood Area Network (NAN) tiers:

Figure 1.3 Multi-Service Field Area Network Architecture with tiers:



In the diagram above, the Cisco Connected Grid Router (CGR) 1000 Series acts as a Field Area Router (FAR). Each FAR advertises a unique Personal Area Network (PAN) – recognized by a combination of a SSID and PAN ID. CGEs are programmed to join a PAN with a given SSID. CGEs can migrate between PANs based on a set of metrics for the PAN (very rarely) and for fault tolerance.

The testing scenarios and use-cases described in this document are focused on the NAN tier and also deal with interactions between head-end elements and the CGEs.

1.2 System Development Kit (SDK) and Cisco DevNet

Cisco supports integration of Cisco’s endpoint technologies in partner devices through a System Development Kit (SDK). Cisco provides the following to a CGE partner to enable the development of CGE devices that interact with the NAN:

• HW Reference design

• Bill of Materials (BOM)

• SDK firmware binary

• CG-REDI cloud services

• Reference guides and training material

• Requirements, guidelines and best practices document

• CGE IVT Master test plan (this document)

The delivery of the elements listed above is handled through the Cisco Connected Grid DevNet program. More specifically, the development of CGE is supported through the ‘Cisco-Enabled Grid Devices’ Technology Center within the DevNet program.

Cisco provides several versions of Hardware reference designs to its partners for development along with a BOM. For RF, the hardware reference designs are CGEREF1 (100-pin reference design, formerly known as ‘Banshee’. This reference design is now deprecated.), CGEREF2 (64-pin reference design) and CGEREF3 (144-pin reference design). For PLC implementations, the hardware reference designs are CGEPLCREF1 (100-pin reference design, formerly known as ‘Alamo’. This reference design is now deprecated), CGEPLCREF2 (64-pin reference design) and CGEPLCREF3 144-pin reference design). It is recommended to implement either the 64-pin or the 144-pin variants of the MCU for either of the communication technologies. Newer features and firmware will only be released for these two variants, and firmware for the 100-pin MCU will not receive these updates in the future.

Along the HW reference design, the partner may choose to develop their endpoints using one of the two supported endpoint software models:



Figure 1.4 CGE SDK – NIC (PPP) model

The NIC (PPP) model, also known as dual MCU model, has Cisco’s Reference design being implemented as a Network Interface Card (NIC) or Communications Module (CM) providing CG communication services to the CGE. The partner also develops an Application Module (AM) that communicates with the CM over a PPP link. The AM, typically along with a micro-kernel Operating System (OS), hosts the partner AMI/DA application like DNP3 client, ANSI C12.22 with C12.19, DLMS/COSEM server etc. Cisco provides the firmware binary for the CM, which the partner installs and runs on the CM MCU.

Figure 1.5 CGE SDK – Integrated Execution (IE) model



The Integrated Execution (IE) model implements a single Micro-controller Unit (MCU), which runs both the communications stack as well as the partner application. Cisco provides a firmware binary OS kernel with integrated protocols stack. The firmware also exposes APIs for communication with MCU peripherals (GPIO and I2C). The partner develops applications using the APIs to communicate with the sensor devices connected as peripherals. The application along with the OS kernel is packaged into a single image for deployment.

Note that the firmware binary provided by Cisco is specific to the communication technology and the software model. So there are different binaries for each hardware reference platform (64, 100 or 144 pin) as well as for each communication technology (RF, PLC or dual-phy).

1.3 CG-REDI and partner development infrastructure

To enable rapid development of CGE, Cisco provides cloud-based network infrastructure services – Connected Grid – Rapid Endpoint Development Infrastructure-as-a-service or CG-REDI. This enables the CGE partners to focus on the key development steps while using the required network services from cloud-hosted facility at Cisco’s DMZ facility. The following diagram illustrates the CG-REDI architecture to support partner development:

Figure 1.6 CG-REDI architecture for partner development



The following diagram illustrates the partner premises topology (‘Thin Starter Kit’) required for development and IVT:

Figure 1.7 Partner Premises – “Thin Starter Kit” topology

The starter kit connects to the CG-REDI via IPv6 GRE over IPv4 IPSec VPN. Also as seen, the Application Server, which hosts the endpoint management and collection software like SCADA master, MDMS, Collection Engine etc., resides at the partner premises and connects to the CG-REDI through a SSL VPN connection. More details on CG-REDI and the starter kit topology can be found in the web-link here.



2 Test Coverage

The testing defined within this test plan is focused on features and functionality related to the Connected Grid Endpoints. This test plan encompasses implementations of the NIC model and the IE model across the RF and PLC HW reference designs. To this affect, some test cases are only applicable to a particular implementation model or a SDK technology. Each test case will point out its applicability to a model and/or the technology.



2.1 What Will Be Tested

The testing would encompass the following areas:

• CGE HW Schematics and diagnostics: HW Schematic review and MCU diagnostics

• RF/PLC characterization: signal strengths and quality, comparative tests

• Local debug ability and manageability/Workforce enablement: Local CSMP, field CG-Mesh firmware upgrades, reboot of CM/IE MCU, field tool security (partially)

• CG-Mesh Connectivity: CG-Mesh L2 connectivity, IPv6 Configuration (managed), IPv6 Routing (RPL) and forwarding over mesh topologies, end-to-end bidirectional communication, multicast forwarding and reachability, Quality of Service (QoS), Time synchronization

• Mesh Security: Access control, data confidentiality and privacy, data authenticity, firmware authenticity

• Remote Manageability and Notifications: CSMP queries, CG-NMS manageability, Firmware management and Over-The-Air firmware update (OTA) through CG-NMS, Power Outage Notifications (PON)/Power Restoration Notifications (PRN)

• Fault resiliency and availability: PAN migration, long duration testing, recovery and reachability after power failure, node failure

• CGE Application: CGE management and functionality (eg. Meter reads), Application QoS markings, OTA AM firmware upgrades, Secure field AM firmware upgrades, Partial application certification for “mesh-friendliness”, Time synchronization of AM application, Remote upgrade of CGE, server virtualization

• Multi-vendor device interoperability: A subset of the above test cases would be executed in a multi-vendor PAN environment

• Performance and Stress: Latencies, performance and reachability with scaled traffic forwarding, ping and meter reads in multi-device, multi-hop network



2.2 What Will Not Be Tested

The following elements will not be tested:

• Application functionality and security:

This includes any functionality that the CGE device provides like metering, SCADA control, transformer/feeder metering, voltage/amp regulation etc. Also no features on the Application Server(s) like UI/CLI interface, performance/scale or stress will be tested.

Test cases defined here also do not test the digital security of the CGE device and the application server. These include attacks like flooding, denial-of-service, man-in-the-middle, intrusion detection and prevention. IVT certification also does not test authenticity and privacy implementations of application traffic. End-to-end application and data security/integrity may be partially tested and the appropriate test cases have been documented here (see Test Suite – CGE Applications).

• No/Partial coverage of application compatibility over CG-Mesh:

Low power and Lossy Networks (LLNs) require applications designed to operate in these environments. These include several design considerations like algorithmic efficiencies of the application protocol (for less intensive CPU and IO operations), protocol design considerations like notification instead of polling, “randomization” of notification to avoid medium access, connectionless transport services with lightweight acknowledgements etc. Application compatibility and mesh-friendliness will not be tested in all cases. The related test cases are documented here (see Test Suite – CGE Applications).

• Physical security:

The testing does not take into consideration the physical security of the CGE.

• Security of data stored or exchanged within the CGE:

The IVT process does not test for security of instructions, data, certificates or other data stored in the MCU of CM/IE, AM or any other storage devices. It also does not test for security of data being exchanged between components on the HW.

• Manufacture time programmability and configurations:

Many of the test cases require the capability to upgrade the boot-loader, CG-Mesh application firmware and configurations on the CGE device. Many of these functions are required at manufacturing or staging time, with configurations/firmware specific to a customer.



However, Cisco does not recommend that the boot-loader and the configuration binary files be updated by any means in a device once it is deployed in the field, as they raise security concerns. The test cases document these capabilities if present in the field tool, and more specifically features that may raise security concerns. However IVT certification does not mandate compliance to these recommendations

• Any other use-cases identified outside this document

3 Test Setups

The test setups in all cases (except the test setup described in Section 3.4) use the CG-REDI infrastructure as described earlier: The partner premises hosts the PAN and the endpoints, whereas CG-REDI hosts all head-end applications except the Application Server which is typically hosted at the partner premises and connects to the CGEs through a VPN connection via the CG-REDI infrastructure. It will be required for the partner to also provide Cisco with 5 CGE DUTs for IVT certification. These CGEs would be integrated in multi-vendor and large-scale topologies.

For the purpose of certification, the PAN will need to be configured for the following topologies for different test scenarios. The required topologies are described in the test cases:

3.1 MCU Diagnostics Setup

The MCU diagnostic setup requires connecting the CM’s debug UART port with Serial Console application on a PC (like Hyperterminal, Putty) via a UART-to-USB converter:

Figure 3.1 IVT Topology 1: CGE diagnostics



The topology and connections are detailed here (link requires access to ‘Cisco-Enabled Grid Devices’ Technology Center).

3.2 One CGE in a PAN

This simple one-device PAN topology is recommended for validation of basic features before testing of the features in more complex topologies: Figure 3.2 IVT Topology 2: 1 CGE in a PAN

3.3 Four (at least) CGEs in a PAN

This is the recommended topology for most feature, functionality, security, management and fault resiliency tests. It should be noted that four is the minimum suggested number for CGEs in this topology. It is a requirement to have at least 3-hops in the mesh network. Also note that the CGEs in this topology are the partner devices in test i.e. not a multi-device topology. Figure 3.3 IVT Topology 3: 4 CGEs in a PAN



3.4 Large scale, multi-hop, multi-vendor CGE PAN

In this topology, the CGE DUT (Device Under Test) is one of many multi-vendor devices connected to the same PAN. The test topology requires the presence of multiple CGE DUTs in this topology in the roles of mesh nodes and leaf nodes. The details are noted in the test cases. Figure 3.4 IVT Topology 4: Large scale, multi-hop, multi-vendor CGE PAN

3.5 Multi-hop PAN Migration In this topology, multiple PANs with same SSIDs are advertised. Some CGE DUTs (Devices 1, 2 and 3) are capable of reaching and joining either PAN, while some others cannot join both PANs directly (Device 4). The CGEs should choose a PAN as primary (shown in blue below) and after failure would migrate to a secondary PAN (show in dotted red below). Figure 3.5 IVT Topology 5: Multi-hop PAN Migration



3.6 Four (at least) multi-vendor CGEs in a PAN

This topology is similar to Section 3.3, except that multi-vendor devices are present in the topology. The exact placements of the CGE DUTs are noted in the test cases. This topology is a subset of the topology described in Section 3.4, and that topology may be used for the test cases referencing this section. Figure 3.6 IVT Topology 6: 4 CGEs in a multi-vendor CGE PAN

3.7 Multi-phy CGEs in a PAN

This topology is focusing on multiple Phys in the same network. Figure 3.7.1 IVT Topology 7.1: 4 CGEs in a multi-Phy CGE PAN



Figure 3.7.2 IVT Topology 7.2: 4 CGEs in a multi-Phy CGE PAN



4 Test and Certification Approach and Considerations

A CGE partner will have access to the test cases that the final product needs to pass for Cisco certification (this document) while reviewing the development process, along with the requirements, guidelines and best practices document (see References). The partner would thus be aware of all requirements for development and certification, in addition to the AM application and functionality.

The partner is also required to perform and submit pre-IVT testing, submit test and endpoint product documentation and provide CGE DUTs to Cisco for the certification (identified in the following sub-section)

4.1 Test Phase Entry Criteria

Each of the criteria identified below need to be met before IVT certification testing can begin:

• Partner has reviewed this test plan and understands the requirements for certification

• Partner has completed development (all intended features) of the CGE including CM/IE (and AM) hardware and firmware, product integration is complete and server application has been developed and integrated with the CGE

• HW Schematics for the CM have been shared with, and reviewed/approved by Cisco

• The vendor partner performs all pre-IVT testing. The test results are available to Cisco for review in an agreed format. All high priority test cases identified for pre-IVT testing have passed.

• All software including field tools and head-end applications are made available, including installation of the applications if applicable, for IVT testing

• Partner has submitted all the requirements needed for certification (identified below in ‘Documentation Requirements’)

• Partner has submitted the product for IVT testing through their DevNet ‘Developer Dashboard’ portal

• Partner premises ‘Starter Kit’ is deployed and VPN connections from CGR and Application Server to CG-REDI are working. All services and infrastructure has been provisioned in CG-REDI. If applicable, remote access to Application Server as well as other software required for IVT is made available to Cisco IVT team for IVT certification.

Documentation requirements from vendor partner before IVT entry (versioning is vendor partner specific):

• Product name and version



• Product Hardware versions (Communication Module, Application Module and versions other relevant electronic components in the CGE)

• Product firmware versions (including CM boot-loader and application; and if applicable all firmware on AM)

• Product data sheets, clearly specifying the application protocol (C12.22, DLMS/COSEM, DNP3 etc.) and intended use-cases

• End-to-end communication architecture and network topology for product deployment

• Product documentation (version information, data sheets, use cases and topology of deployment, and product installation instructions, known issues)

• Application Server product name, version, platform, version and deployment topology

• Field tool name, version and deployment details

• 7 CGEs to Cisco for local testing and debugging

• Accompanying instructions for the CGE, for setup, configuration and local management

• Known issues with the product

4.2 IVT Considerations specific to a certification

Detailed Test Plan document specific to an IVT certification may be required. The testing strategy for each certification needs to be identified in the Detailed Test Plan. New test cases may be identified and existing test cases may be removed, based on design and implementation of the CGE.



4.3 Test Execution tracking and reporting

• Cisco DevNet team will use the test plan identified for IVT certification (this document or a derived detailed test plan document specific to an IVT) and create a Cisco TIMS Results (Database Name - SGBU)

• Vendor partner will perform pre-IVT testing at vendor premises using CG-REDI based services. The vendor partner will share the test results with Cisco DevNet team. Once all IVT entry criteria have been met, Cisco DevNet team will update the pre-IVT test case results in TIMS.

• All test results executed as part of IVT will also be tracked in TIMS database

• Detailed Test result summary will be captured in the final IVT certification report. Intermediate IVT test reports can also be generated to track progress, defects and open issues, and shared with the vendor partners.

4.4 Defect Tracking

Defect tracking to be performed through shared documentations between Cisco and partner. Details TBD.

4.5 Requirements Traceability Approach

The requirements for the CGE certification in terms of features and functionality are traced from the use-cases and subsystems recognized in the Zanzibar project EC and DIG documents. New test cases are added to the test plan based on availability and implementation of new requirements and features.

4.6 IVT Certification Elements

The following elements are tied to a certification process and recorded in the certification report Table 4.6.1: IVT elements:

Category Description Device/Location

Hardware Connected Grid Endpoint Hardware version (CM and AM if applicable)

Connected Grid Endpoint

Firmware Connected Grid Endpoint CM/IE secure boot loader firmware version


Firmware Connected Grid Endpoint CM/IE application firmware




4.7 IVT Recertification criteria

The vendor CGE product will require recertification if any of the following conditions are met:

• Certification is required with a different version of Cisco SDK firmware – bootloader or application (including bug fixes or feature enhancements) which are determined to have dependencies on hardware implementation or include software/host interface (PPP) changes

• Any changes by the vendor partner to the CGE hardware components, schematics, I/O interfaces or layouts, specifically to the CG-Mesh communications hardware, or its interfacing with other components

• Any changes by the vendor partner to the AM hardware or firmware, which is determined to impact mesh communications

• Recertification may be required to establish interoperability between a specific version of CGR PAN firmware and CGE firmware

version

Firmware Connected Grid Endpoint AM firmware version(s)


Firmware Connected Grid Router PAN firmware version

Connected Grid Router

Firmware Connected Grid Router CG-OS/IOx software version


Software CG-NMS version CG-REDI

Network infrastructure Certificate authority platform, software and version

CG-REDI

Network infrastructure AAA server platform, software and version

CG-REDI

Network infrastructure DHCPv6 Server platform, software and version

CG-REDI

Network infrastructure HSM device and firmware version

CG-REDI

Network infrastructure NTP Server CG-REDI

Application Server(s) Application server elements, software, platform, version and deployment topology

Application Server(s)



• Recertification may also be required to establish interoperability between FAN Head-end software and hardware components, and their interactions with CGE products

• Any other changes to elements recognized in Section 4.7, as determined by Cisco’s DevNet and engineering teams

4.8 IVT Certification Numbering scheme The following numbering scheme will be used for certification:

• The Certification number will start with ‘CGE-IVT’, recognizing the certification of CGE through the DevNet IVT process.

• ‘CGE-IVT’ will be appended by the EDCS document number of the Certification Report (Defined in Section 4.11 of this document)

A sample certification number would thus be ‘CGE-IVT-1291267’ where 1291267 is the EDCS document number of the Certification Report (described in Section 4.10). This scheme allows us to track the details of the certification from the number.

4.9 IVT Certification Report The following items are required to be part of the certification report:

• Certification Program: CGE IVT Certification

• Partner Name, product name and product version

• Product description and deployment topology

• Certification number: Defined in Section 4.8 of this document

• Certification date

• Certification personnel

• Cisco EDCS document number of the certification report

• Certification Elements as defined in Section 4.6 of this document

• Any deviations to the recommended topology including supplemental devices used during certification testing

• Reference to the Approved Test Plan (this document or supplemental documents derived from this document) and version of the Test Plan used for certification

• Test summary including failed and dropped test cases, and the reason for dropping

• Any deviations from Cisco’s recommendations as noted in the test cases

• Reference to TIMS database



4.10 IVT Deliverables The following items are deliverables to be shared with the vendor partner after successful IVT certification:

• Test Plan (this document or supplemental documents derived from this document)

• Test Execution results: Will be archived in TIMS database, and will also be part of the IVT certification

• Configuration files used for CGR WPAN interface and the CGEs

• Certification Report: Described in Section 4.9, checked into EDCS: This is made available to the partner.



4.11 Test Exit/Certification Criteria

All of the following conditions should be met for successful IVT certification:

• 100% execution of high priority test cases.

• All high priority test cases should pass (100%)

• If any medium or low priority tests were dropped, the reason for dropping will need to be reviewed and approved by stakeholders. These need to be documented in the certification report.

• If medium or low priority tests failed, appropriate documentation should be provided in certification report.

• Test results uploaded into TIMS and test summary provided in the certification report.

4.12 Performance test (PR) and Scalability test (SC)

A stress test for a CGE would include performing the following operations:

• CGE device would be positioned in a large scale, multi-hop, multi-device PAN (Section 3.4) as root of a sub tree

• CGE performs forwarding of large volumes of IPv6 datagrams downstream and upstream (i.e. using both source and default routing) across multiple destinations

• A traffic intensive remote operation over the mesh to the CGE is performed, like file/firmware transfer

• A combination of the above, where several operations are performed together.

The application performance (meter reads), forwarding efficiency (end-to-end packet drops and latency values for upstream and downstream traffic flows), FAN features (eg. PON/PRN), Manageability (CSMP queries), long duration/soak testing should be tested in the proposed stress test environment. Criteria for passing a stress test will be identified in the associated test cases.



4.13 Pre-IVT Testing Matrix

The vendor partner runs the test cases described below at their premises

Table 4.14.1: CGE diagnostic tests (Pre IVT):

Logical ID Test Setup Priority Title

CGE_PRE_IVT_DIAG_TC001 Figure 3.1 High Run MCU diagnostics and review results

CGE_PRE_IVT_DIAG_TC002 Figure 3.1 High RF MCU Diagnostic test

Table 4.14.2: Workforce enablement (Pre IVT):

Logical ID Test Setup

Priority Title

CGE_PRE_IVT_WFE_TC001 N/A High User and device authentication in the field tool

CGE_PRE_IVT_WFE_TC002 N/A High Capability to upgrade CM/IE application firmware and LMAC firmware (for PLC and dual-phy) in the field

CGE_PRE_IVT_WFE_TC003 N/A High Data privacy for field tool CGE_PRE_IVT_WFE_TC004 N/A High Debugability through CSMP CGE_PRE_IVT_WFE_TC005 N/A High Network debugability through

ICMP pings CGE_PRE_IVT_WFE_TC006 N/A High Capability to query

networking configuration Table 4.14.3: CG-Mesh Connectivity (Pre IVT):

Logical ID Test Setup Priority Title CGE_PRE_IVT_MESH_TC001 Figure 3.2 High Layer 2 Mesh connectivity CGE_PRE_IVT_MESH_TC002 Figure 3.2 High Managed IPv6 configuration CGE_PRE_IVT_MESH_TC003 Figure 3.3 High RPL advertisements CGE_PRE_IVT_MESH_TC004 Figure 3.3 High IPv6 mesh connectivity and

forwarding CGE_PRE_IVT_MESH_TC005 Figure 3.3 High Bidirectional connectivity to

head-end servers CGE_PRE_IVT_MESH_TC006 Figure 3.3 High Time Synchronization CGE_PRE_IVT_MESH_TC007 N/A High Organizationally Unique



EUI64 values for CG-Mesh communications

CGE_PRE_IVT_MESH_TC008 Figure 3.7.1 High Preference of physical medium (dual-phy only)

CGE_PRE_IVT_MESH_TC009 Figure 3.7 High Multi-phy multi-hop test (dual-phy only)

Table 4.14.4: Mesh Security (Pre IVT):

Logical ID Test Setup Priority Title CGE_ PRE_IVT_SEC_TC001 Figure 3.3 High Access Control with 802.1X

EAP-TLS CGE_ PRE_IVT_SEC_TC002 Figure 3.3 High 802.11i handshake, key

exchange and data privacy for all data

CGE_ PRE_IVT_SEC_TC003 Figure 3.3 High Firmware upgrade with secure boot-loader

Table 4.14.5: Remote Management and Notifications (Pre IVT):

Logical ID Test Setup Priority Title CGE_PRE_IVT_MGMT_TC001 Figure 3.3 High CG-NMS registration,

reachability, configuration template and periodic updates

CGE_PRE_IVT_MGMT_TC002 Figure 3.3 High CG-NMS CSMP data integrity and authenticity

CGE_PRE_IVT_MGMT_TC003 Figure 3.2 High CG-NMS firmware upgrade over unicast CSMP messages

CGE_PRE_IVT_MGMT_TC004 Figure 3.3 High CG-NMS firmware upgrade over multicast CSMP messages

CGE_PRE_IVT_MGMT_TC005 Figure 3.3 Medium PON/PRN notifications (RF and dual-phy only)

CGE_PRE_IVT_MGMT_TC006 Figure 3.3 High Capability to derive Application Server(s) address from DHCP Option 17

CGE_PRE_IVT_MGMT_TC007 Figure 3.3 High LMAC FWDL (PLC and dual-phy only)

CGE_PRE_IVT_MGMT_TC008 Figure 3.7 High Dual-phy CSMP TLVs (dual-phy only)



Table 4.14.6: CGE Application (Pre IVT):

Logical ID Test Setup Priority Title CGE_PRE_IVT_APP_TC001 Figure 3.3 High Application Server

functionality (Meter reads, SCADA Comms etc)

CGE_PRE_IVT_APP_TC002 Figure 3.3 High Application QoS markings (bidirectional)

CGE_PRE_IVT_APP_TC003 Figure 3.3 High Time synchronization of Application Server and AM firmware for event correlation and DA use-cases

CGE_PRE_IVT_APP_TC004 Figure 3.3 High Application protocol “mesh-friendliness”

CGE_PRE_IVT_APP_TC005 Figure 3.1 High IPv6 MTU and fragmentation CGE_PRE_IVT_APP_TC006 Figure 3.3 Medium Conformance to 800 byte L3

packet size Table 4.14.7: Fault resiliency and availability (Pre IVT):

Logical ID Test Setup Priority Title CGE_PRE_IVT_FRA_TC001 Figure 3.3 High Recovery after power failure

of CGE CGE_PRE_IVT_FRA_TC002 Figure 3.3 High Recovery after power failure

of FAR CGE_PRE_IVT_FRA_TC003 Figure 3.3 High RPL convergence after node

failures



4.14 IVT Testing Matrix

All test cases described below are run in a multi-vendor environment in IVT test labs

Table 4.15.1: CGE HW Schematics and diagnostics (IVT):


CGE_IVT_HW_TC001 N/A High Review HW Schematics

Table 4.15.2: RF/PLC physical and MAC layer (IVT):


CGE_IVT_SIG_TC001 Figure 3.2 High Transmit Power CGE_IVT_SIG_TC002 Figure 3.2 High RSSI/LQI Comparative test CGE_IVT_SIG_TC003 Figure 3.2 High PLC Modulation and

throughput (PLC and dual-phy only)

CGE_IVT_SIG_TC004 Figure 3.2 High Throughput comparison test (dual-phy only)

Table 4.15.3: Workforce enablement (IVT):

Logical ID Test Setup Priority Title CGE_IVT_WFE_TC001 N/A High User and device

authentication in the field tool CGE_IVT_WFE_TC002 N/A High Capability to upgrade CM/IE

application firmware and LMAC firmware (for PLC and dual-phy) in the field

CGE_IVT_WFE_TC003 N/A Medium Document capability to upgrade CM/IE boot loader in the field

CGE_IVT_WFE_TC004 N/A Medium Document capability to upgrade CM/IE configuration in the field

CGE_IVT_WFE_TC005 N/A Medium Data Privacy for the field tool CGE_IVT_WFE_TC006 N/A High Debugability through CSMP CGE_IVT_WFE_TC007 N/A High Network debugability through

ICMP pings



CGE_IVT_WFE_TC008 N/A Low Capability to reboot/reset communication HW

CGE_IVT_WFE_TC009 N/A High Capability to query networking configuration

CGE_IVT_WFE_TC010 N/A Medium Capability to upgrade AM boot-loader, application firmware and configuration in the field

CGE_IVT_WFE_TC011 N/A Low Capability to reboot/reset communication module HW

Table 4.15.4: CG-Mesh Connectivity (IVT):

Logical ID Test Setup

Priority Title

CGE_IVT_MESH_TC001 Figure 3.6

Medium CSMP QoS markings

CGE_IVT_MESH_TC002 N/A High Organizationally Unique EUI64 values for CG-Mesh communications

CGE_IVT_MESH_TC003 Figure 3.7.1

High Preference of physical medium (dual-phy only)


Medium Use of advanced metrics for physical medium choice (dual-phy only)


Medium RPL advertisements (dual-phy only)


Medium Use of advanced RPL metrics (dual-phy only)

CGE_IVT_MESH_TC007 Figure 3.7

High Multi-phy multi-hop test (dual-phy only)

Table 4.15.5: Mesh Security (IVT):

Logical ID Test Setup Priority Title CGE_IVT_SEC_TC001 Figure 3.6 High Access Control with 802.1X

EAP-TLS CGE_IVT_SEC_TC002 Figure 3.6 Low Access control for

unauthenticated/untrusted certificate on device (device rejection)

CGE_IVT_SEC_TC003 Figure 3.6 High Firmware upgrade with secure boot-loader



CGE_IVT_SEC_TC004 Figure 3.6 High Failed untrusted firmware upgrade with secure boot-loader

CGE_IVT_SEC_TC005 Figure 3.7 High Secure mesh with dual-phy (dual-phy only)

Table 4.15.6: Remote Management and Notifications (IVT):

Logical ID Test Setup Priority Title CGE_IVT_MGMT_TC001 Figure 3.6 High CSMP over CG-Mesh CGE_IVT_MGMT_TC002 Figure 3.6 High CG-NMS registration,

reachability, configuration template and periodic updates

CGE_IVT_MGMT_TC003 Figure 3.6 High CG-NMS CSMP data integrity and authenticity

CGE_IVT_MGMT_TC004 Figure 3.6 High CG-NMS CSMP missing/invalid signatures

CGE_IVT_MGMT_TC005 Figure 3.6 High CG-NMS firmware upgrade over unicast CSMP messages

CGE_IVT_MGMT_TC006 Figure 3.6 High CG-NMS firmware upgrade over multicast CSMP messages

CGE_IVT_MGMT_TC007 Figure 3.6 Medium PON/PRN notifications (RF and dual-phy only)

CGE_IVT_MGMT_TC008 Figure 3.6 High Capability to derive Application Server(s) address from DHCP Option 17

CGE_IVT_MGMT_TC009 Figure 3.6 High LMAC FWDL (PLC and dual-phy only)

CGE_IVT_MGMT_TC010 Figure 3.7 High Dual-phy CSMP TLVs (dual-phy only)

Table 4.15.7: CGE Application (IVT):

Logical ID Test Setup Priority Title CGE_IVT_APP_TC001 Figure 3.6 High Application Server

functionality (Meter reads, SCADA Comms etc)

CGE_IVT_APP_TC002 Figure 3.6 High Application QoS markings (bidirectional)



CGE_IVT_APP_TC003 Figure 3.6 High Time synchronization of Application Server and AM firmware for event correlation and DA use-cases

CGE_IVT_APP_TC004 Figure 3.6 Low Document capability to perform AM application firmware, boot-loader and configuration updates over-the-air

CGE_IVT_APP_TC005 Figure 3.6 High Application protocol “mesh-friendliness”

CGE_IVT_APP_TC006 Figure 3.6 Low Document application data integrity and authenticity checks for Application messages

CGE_IVT_APP_TC007 Figure 3.6 Low Document capability and method to securely update (CM/IE) boot-loader and configuration over-the-air

CGE_IVT_APP_TC008 N/A Low Document device inventory, firmware and configuration management capabilities of Application Server(s)

CGE_IVT_APP_TC009 N/A Medium Document server virtualization capability

CGE_IVT_APP_TC010 Figure 3.6 High IPv6 MTU and fragmentation CGE_IVT_APP_TC011 Figure 3.6 Medium Conformance to 800 byte L3

packet size Table 4.15.8: Fault resiliency and availability (IVT):

Logical ID Test Setup Priority Title CGE_IVT_FRA_TC001 Figure 3.6 High Recovery after power failure

of CGE CGE_IVT_FRA_TC002 Figure 3.6 High Recovery after power failure

of FAR CGE_IVT_FRA_TC003 Figure 3.5 High PAN migration after DAG

failure CGE_IVT_FRA_TC004 Figure 3.6 High RPL convergence after node

failures



Table 4.15.9: Soak and Performance/Stress (IVT):

Logical ID Test Setup Priority Title CGE_IVT_SOAK_TC001 Figure 3.6 High Soak Test CGE_IVT_STRESS_TC001 Figure 3.4 Optional App performance with long

duration large volume packet forwarding



5 Test Resources

5.1 People

All testing will be performed as a joint activity between the partner development and test team (at the partner premises setup – Figure 1.6) and Cisco’s DevNet team (at IVT test labs).

5.2 Hardware Table 5.2.1: Hardware requirement for certification:

5.3 Firmware

Table 5.3.1: Firmware requirements for certification:

Hardware Location

Connected Grid Endpoint device Partner premises and/or at Cisco test labs

Connected Grid Router Partner premises and/or at Cisco test labs

Application Server(s) Partner premises or CG-REDI PaaS cloud

Head-end Router (HER) ASR1K CG-REDI

SSL VPN Gateway CG-REDI

Server hosting CG-NMS CG-REDI

Servers hosting Network-infrastructure Services

CG-REDI

Hardware Security Module (HSM) CG-REDI

VPN router (optional) Partner premises

Firmware Device

Connected Grid Endpoint CM/IE firmware


Connected Grid Endpoint AM firmware


Connected Grid Router WPAN firmware




5.4 Software Table 5.4.1: Software requirements for certification:

5.5 Test Equipment

Table 5.5.1: Test equipment requirements for certification:

5.6 Test Tools (software) Table 5.6.1: Test tool requirements for certification:

Connected Grid Router CG-OS release


Software Location

CG-NMS CG-REDI

Public key infrastructure (PKI)/Certificate authority (CA)

CG-REDI

AAA server CG-REDI

DHCPv6 Server CG-REDI

NTP Server CG-REDI

Application Server(s) Partner Premises or CG-REDI PaaS cloud

Equipment Location

Test Traffic Generator (TBD) TBD

Tool Comments

MCU Diagnostics Firmware provided by Cisco through DEVNET portal



CSMP CLI/GUI Java based tool provided by Cisco for CSMP functionality

Partner Field tool(s) Installed in hardened field device for workforce enablement

Cisco cfgwriter For configuration generation

Segger JLink Pro Programming CGE through JTAG



6 Risks, Mitigations, and Contingencies

Table 6.1: Risks

Id Risk Probability Impact Mitigation/Contingency Plans

1 Incorrect implementation of HW Reference Design

Medium High Review HW Schematic before IVT Certification (Test Entry Criteria); Mandate MCU diagnostics

2 CM/IE firmware code quality

Low High Cisco Engineering provides new firmware for CM/IE and certification

3 Defects identified in HW Reference Design

Medium High Cisco Engineering provides new reference design and suggested fixes to existing hardware implementations

4 Missing/Incorrect implementation of high priority features in CGE

Medium High High priority features would require fix from CGE partner

5 Missing/Incorrect implementation of medium priority features in CGE

Medium Medium Conditional certification and clearly marked deficiencies in the certification report

6 Lack of available CGEs for certification

Medium Medium Test cases involving larger number of CGEs can be substituted with mixed device topologies

7 Delay in fixing critical defects in firmware

Low High -



7 Detailed Test Cases (Pre-IVT)

7.1 CGE HW Schematics and diagnostics (Pre-IVT):

Test Case Details

Title Run MCU diagnostics and review results

ID

Logical ID CGE_PRE_IVT_DIAG_TC001

Owner(s) mathaker

Description Run MCU diagnostics firmware tests on the MCU of the CGE

Test Setup Figure 3.1

Procedure • Download MCU diagnostics firmware tests based on HW Reference design (PLC or RF) from DevNet Web portal

• Load and run the MCU diagnostic on CM/IE MCU as per the instructions posted on DevNet Web portal

• Make sure all tests pass with the exception of the RF test.

• Record results

Test Phase Pre-IVT System Test

Test Suite CGE HW Schematics and diagnostics

Pass/Fail Criteria All tests pass with the exception of the RF test should pass

Priority High

Test Case Details

Title RF MCU Diagnostic test

ID

Logical ID CGE_PRE_IVT_DIAG_TC002

Owner(s) mathaker

Description Run RF MCU diagnostics firmware tests on the MCU of the CGE

with TIMS

with TIMS




Procedure • Download MCU diagnostics firmware tests based on HW Reference design (PLC or RF or Dual Phy) from DevNet Web portal

• Load and run the MCU diagnostic on CM/IE MCU as per the instructions posted on DevNet Web portal

• Two nodes will be required.

• Make sure RF test passes and record the results.



Pass/Fail Criteria As noted in the procedure

Priority High

7.2 Workforce enablement (Pre-IVT):

Test Case Details

Title User and device authentication in the field tool

ID

Logical ID CGE_PRE_IVT_WFE_TC001

Owner(s) adewanga

Description The CGE device management field tool access should be secured through user and/or device authentication

Test Setup N/A

Procedure • Access control can be in the form of user based authentication and/or device authentication

• Access control maybe tied to only critical functionality like IE/CM firmware upgrades

• The access control mechanism should be documented in the test report

• For user based authentication, document mechanism of authentication (local vs remote server) and method of storage of passwords if local (hashed, encrypted, plain-text)

with TIMS



• Document the access control mechanism including software components protected. Provide screenshots demonstrating the mechanisms.


Test Suite Workforce Enablement

Pass/Fail Criteria All criteria described in procedure should pass for the test case to pass. Exceptions should be noted.

Priority High

Test Case Details

Title Capability to upgrade CM/IE application firmware and LMAC firmware (for PLC and dual-phy) in the field

ID


Owner(s) adewanga

Description Capability to upgrade CM/IE firmware and LMAC firmware (for PLC and dual-phy) in the field

Test Setup N/A

Procedure • The CGE should provide a mechanism to upgrade CM/IE application firmware in the field

This enables field technicians to upgrade the communications firmware in the field if the CGE is not online.

• For PLC and dual-phy CGEs, the field tool should also provide a mechanism to update the LMAC firmware.

• The mechanism should be documented. Provide supporting screenshots.



Pass/Fail Criteria Described in procedure.

Priority High

Test Case Details

with TIMS

with TIMS



Title Data privacy for field tool

ID


Owner(s) adewanga

Description Communication between field tool and the CGE should be secure

Test Setup N/A

Procedure • The medium of communication between the field tool and the CGE should be secure i.e. implement data privacy:

-If using wireless medium, then the wireless connection should be secure through the use of wireless encryption

This is required if secure data is being transferred between the field tool and the CGE like 802.1x EAP-TLS X.509 certificate with private key (see CGE_IVT_WFE_TC004)



Pass/Fail Criteria Described in procedure. Exceptions should be noted.

Priority High

Test Case Details

Title Debugability through CSMP

ID


Owner(s) adewanga

Description The CGE should provide a mechanism to query CSMP TLV from the CM/IE kernel

Test Setup N/A

Procedure • For PPP model, the capability to query CSMP TLVs via the host port over the PPP connection should be available. The AM may relay/pass-through/route the CSMP messages from the field tool to the CM, without any processing on CSMP message contents

• For IE, tool to query CSMP TLVs via API calls to the kernel should be available

• All CSMP TLVs values should be available to query.

with TIMS



Document the procedure and screenshots.



Pass/Fail Criteria All criteria described in procedure should pass for the test case to pass.

Priority High

Test Case Details

Title Network debugability through ICMP pings

ID


Owner(s) adewanga

Description The CGE should provide a mechanism to ping IPv6 nodes in the network

Test Setup N/A

Procedure • For PPP model, the capability to ping from the AM to any IPv6 destination should be available

• For IE, the capability to ping any IPv6 destination should be available

• Make sure IPv6 pings from AM App/IE App to the CGR WPAN interface is successful.

• Make sure IPv6 ping to the AM of another CGE in the same WPAN is successful.

• Provide test logs – topology (RPL atree, DUT location), procedure to initiate ping and successful ping response



Pass/Fail Criteria As described in the procedure

Priority High

Test Case Details

Title Capability to query networking configuration

with TIMS

with TIMS



ID


Owner(s) adewanga

Description Capability to query networking configuration should be available

Test Setup N/A

Procedure • The CGE should provide tools to query or display the following networking configuration on the AM/IE kernel:

-For PPP model, PPP interface IPv6 addresses (link local and global) and prefix-length on the AM

-Routing table on the AM/IE kernel

-Application Server address: This is derived from contents of DHCP Option 17. Each vendor will be assigned a unique Option index for Option 17, and the IPv6 address corresponding to the Option index will contain the Application server address. This assignment will be specific to the utility customer. The CGE must not “hard-code” this index as it may vary from utility to utility.

• CG-NMS IPv6 address: This is derived from DHCP Option 17 – Option index 1 value.

• Make sure that the tool reports expected values for the above parameters

• Document procedure to access the networking tools and execute the tests. Provide screenshots and/or test logs




Priority High

7.3 CG-Mesh Connectivity (Pre-IVT):

Test Case Details

Title Layer 2 Mesh connectivity

ID

Logical ID CGE_PRE_IVT_MESH_TC001

Owner(s) adewanga

with TIMS



Description The CGE should join the FAR PAN at Layer 2


Procedure • Program the CGE CM/IE MCU flash with the boot loader, application image and the configuration

• Deploy the CGE at the intended distance and environment from the FAR as per intended deployment specifications

• Reboot the CGE

• Monitor and document the CGE time to join the PAN mesh at layer 2. This should be comparable to the Cisco reference implementation of the same HW Reference design within the same environment with the same configuration (attenuation, distance, same PAN, and configuration). Make note of any deviations.

• Commands for on CGR:

CG-OS

show wpan in-hardware linknbrs

show wpan linknbrs table

IOX

show wpan x/1 hardware link-neighbors

show wpan x/1 link-neighbors table

• Query CSMP TLV 52 on CM/IE kernel and make sure CGR is seen as a CG-Mesh neighbor

• Provide the times documented and the following CLI outputs from CGR–

CG-OS

show wpan in-hardware linknbrs

show wpan linknbrs table

IOX

show wpan x/1 hardware link-neighbors

show wpan x/1 link-neighbors table


Test Suite CG-Mesh Connectivity


Priority High



Test Case Details

Title Managed IPv6 configuration

ID


Owner(s) adewanga

Description The CGE nodes should be able to Solicit and configure IPv6 addresses


Procedure • Make sure provisioning with CG-REDI is completed and the connection is working. Make sure CGR is able to reach the DHCP Server through pings

• Program the CGE CM/IE MCU flash with the boot loader, application image and the configuration

• Deploy the CGE at the intended distance and environment from the FAR as per intended deployment specifications

• Reboot the CGE

• Monitor and document the CGE time to the first DHCP solicit request from the CGE. Use in-built ‘ethanalyzer’ tool in CGR to capture and decode packets in the PAN network. Make note of any deviations. CLI for ethanalyzer:

ethanalyzer local interface wpan limit-captured-frames 0

• This should be comparable to the Cisco reference implementation of the same HW Reference design, within the same environment with the same configuration (attenuation, distance, same PAN, and configuration) and with the same DHCP Solicit interval configuration. (Skip this step for Pre-IVT testing)

• Verify that the CGE received a DHCPv6 Reply from the CG-REDI DHCP server. Note the address offered in the DHCPv6 Reply message

• For PPP model, verify that AM makes DHCPv6 requests every 60 seconds on the PPP link.

• Verify that the AM configures the IPv6 address to its PPP interface with a /128 mask.

• Verify that the default gateway on AM module is pointing to the PPP interface.

• Provide test logs – Successfully configured IPv6 address as verified from the field tool, times documented above.


with TIMS





Priority High

Test Case Details

Title RPL advertisements

ID


Owner(s) adewanga

Description RPL should advertise all IPv6 destinations as per the deployed topology to the CGR


Procedure • Make sure provisioning with CG-REDI is completed and the connection is working. Make sure CGR is able to reach the DHCP Server through pings

• Program the CGEs CM/IE MCU flash with the boot loader, application image and the configuration

• Deploy the CGEs at the intended distance and environment from the FAR as per intended deployment specifications

• Make sure to place the CGE (or use appropriate attenuations) so that they join the CG-Mesh in the topology as shown in Figure 3.3

• Make sure that all CGEs have joined the CG Mesh and DHCP server has leased an IPv6 address to all the CGEs

• Make sure that all CGE RPL instances advertise the IPv6 addresses to the CGR as per the leases granted by DHCP Server:

CG-OS

show rpl dag 1 table

show rpl dag 1 tree

IOX

show wpan x/1 rpl table

show wpan x/1 rpl tree

• The RPL tree should reflect the topology in Figure 3.3

• For PPP model, the AM IPv6 address should also be advertised through RPL and displayed in these CLI:

with TIMS



CG-OS

show rpl dag 1 table

show rpl dag 1 tree

IOX

show wpan x/1 rpl atable

show wpan x/1 rpl atree

• Document outputs of the above CLIs




Priority High

Test Case Details

Title IPv6 mesh connectivity and forwarding

ID


Owner(s) adewanga

Description All IPv6 destinations in the CG-Mesh should be inter-reachable


Procedure • Follow the procedure noted in CGE_PRE_IVT_MESH_TC003

• Make sure all the CGE and AM (if applicable) are reachable from CGR through pings

• For IE model, make sure that all CGEs within the CG-Mesh can reach each other through pings

• For PPP model, make sure all AMs are able to reach each other through pings

• Document the RPL topology and ping outputs




Priority High

with TIMS



Test Case Details

Title Bidirectional connectivity to head-end servers

ID


Owner(s) adewanga

Description The endpoints should have bidirectional connectivity to head-end management servers


Procedure • Follow the procedure noted in CGE_PRE_IVT_MESH_TC004

• Bidirectional reachability to CG-NMS and all Application Server(s) should be verified using pings from AM/IE and/or the head-end servers

• Document the RPL topology and ping outputs




Priority High

Test Case Details

Title Time Synchronization

ID


Owner(s) adewanga

Description Make sure all CGEs are time synchronized with the CGR


Procedure • Make sure provisioning of DHCP Server with CG-REDI is completed and the GRE/IPSec connection is working between CGR and the CG-REDI.

• Make sure CGR is configured with NTP client pointing to the NTP server in CG-REDI, and is synchronized. Use the following CLIs:

CG-OS

with TIMS

with TIMS



show ntp peer-status

IOX

show ntp associations

show clock

• Make sure all CGEs have joined the PAN.

• Time is distributed in CG-Mesh using 802.15.4e MAC Time Distribution Protocol. The accuracy of the time is expected to be in 10ms to 100ms range from CGR time.

• Check the time on the CGE CM/IE using CSMP query for TLV 18

• Verify that the time reported is synchronized with the system time on CGR to the required accuracy (100s of msec) (Method TBD)

• Document the CLI outputs from CGR and the corresponding TLV outputs from the CGE.




Priority High

Test Case Details

Title Organizationally Unique EUI64 values for CG-Mesh Communication

ID


Owner(s) adewanga

Description EUI64 values for the CGE should be globally unique

Test Setup N/A

Procedure • Partner should use an IEEE Registration Authority assigned Organizationally Unique Identifier (OUI) for the first 3-bytes of the EUI-64 for all their devices.

The remaining 40 bits of the EUI-64 should be unique for every device that is manufactured.

• Document the EUI64 derivation procedure for the CGEs: Through xml configuration or algorithmic derivation

• Document the procedure for algorithmic derivation of the

with TIMS



EUI64 value and make sure that IEEE assigns the OUI being used.




Priority High

Test Case Details

Title Preference of physical medium (dual-phy only)

ID


Owner(s) linyan

Description RF is the preferred medium for dual-phy endpoints

Test Setup Figure 3.7.1

Procedure • Make sure provisioning of DHCP Server with CG-REDI is completed and the GRE/IPSec connection is working between CGR and the CG-REDI. Make sure CGR is able to reach the DHCP Server using the WPAN intf address(es).

• Setup the bridges with RF module as a master and PLC as a slave:

http://solutionpartner.cisco.com/web/cegd/dual-phy-configuration

• When the CGE comes up, check to ensure that CGE tries to use RF media before the PLC media to form the mesh network.

• Make sure all the CGEs have joined the mesh in Layer 2 and Layer 3 and are reachable via the preferred RF medium. Verify using the counters below.

• Query TLVs for the network stats: LowpanPhyCounters LowpanRfStats LowpanPlcStats




with TIMS



Priority High

Test Case Details

Title Multi-phy multi-hop test (dual-phy only)

ID


Owner(s) linyan

Description Mesh network with different mediums at various hop level



• Setup the bridges with RF module as a master and PLC as a slave (Figure 3.7.1):


• When the CGE comes up, check to ensure that CGE tries to use RF media before the PLC media to form mesh as the primary medium.


• Ensure that the network can ping the CGEs using 1- 1280 packet size

• Next, setup the bridges with PLC module as a master and RF as a slave

• Ensure that the network can ping the CGEs using 1- 1280 packet size




Priority High

with TIMS



7.4 CG-Mesh Security (Pre-IVT):

Test Case Details

Title Access Control with 802.1X EAP-TLS

ID

Logical ID CGE_PRE_IVT_SEC_TC001

Owner(s) adewanga

Description Make sure client functionality for access control using 802.1x EAP-TLS is working



• Follow procedure documented here to setup security:

http://solutionpartner.cisco.com/web/cegd/configure-security-in-starter-kit

• Make sure devices at one hop from CGR were able to authenticate (EAP + Radius)

• Make sure multi-hop devices were able to authenticate with the Radius server (EAPoUDP + Radius)

• CGR CLI:

show dot1x all details

• Make sure all CGEs get configured with IPv6 addresses and are reachable via pings from CGR

• Document CLI output for dot1x and RPL topology


Test Suite CG-Mesh Security

Pass/Fail Criteria As noted in procedure

Priority High

Test Case Details

Title 802.11i handshake, key exchange and data privacy for all data

ID

with TIMS

with TIMS




Owner(s) adewanga

Description Trusted CGEs should be able to derive PMK, and exchange PTK and GTK keys through handshake for CG-Mesh security


Procedure • Follow procedure documented in CGE_PRE_IVT_SEC_TC001

• Make sure that CGEs were able to derive PMK, and download PTK and GTK keys using a 4-way handshake from CGR, by querying CSMP TLV 34 – which should show the keys and their lifetimes.

• Make sure pings to the CGEs are working

• Document the TLV values and the ping logs




Priority High

Test Case Details

Title Firmware upgrade with secure boot loader

ID


Owner(s) adewanga

Description Trusted CM/IE application firmware should be successfully verified by secure boot loader and the upgrade should work


Procedure • Perform a firmware upgrade of a trusted CM/IE application firmware using the field tool

Trusted CM application firmware is signed by Cisco’s CA and verified by the CM secure boot-loader, which recognizes Cisco’s CA as a trustpoint

A trusted IE application firmware is signed by the partner CA and validated by the secure boot-loader, which recognizes the partner CA as a trustpoint

with TIMS



• Make sure the firmware upgrade is successful by verifying CSMP TLV 11 on the CM/IE kernel

• Modify 802.11i GTK (mesh-security keys) on FAR to ensure cold start and reboot the CGE

• The device should join, authenticate and communicate over the CG-Mesh




Priority High

7.5 Remote Management and Notifications (Pre-IVT)

Test Case Details

Title CG-NMS registration, reachability, configuration template and periodic updates

ID

Logical ID CGE_PRE_IVT_MGMT_TC001

Owner(s) adewanga

Description CGE CM/IE kernel should register with CG-NMS and should be manageable


Procedure • Make sure all provisioning for the partner setup is complete and working between CG-REDI and partner premises, including CGR and Application server connectivity.

• Make sure that the PAN environment is configured for security including 802.1x and 802.11i.

• Make sure that Option 17 in DHCP Reply message from DHCP Server to the PAN contains the NMS address as its first option

• Reboot the CGEs to make sure that the correct Option 17 is received, if required.

• Make sure all CGEs are deployed and have IPv6 addresses assigned and are reachable through the CGR. Query CSMP

with TIMS



TLV 43 from the host port/Field application to make sure the correct CG-NMS IPv6 address is received

• Document the time taken to register node with CG-NMS

• Make sure ping is working from CG-NMS to the CGEs

• Update the ‘Report Interval’ from the configuration templates for mesh devices and Push the configuration to the mesh devices

• Make sure in CG-NMS that properties are reported periodically like IP address, serial number, model number, firmware version, SSID PAN ID, Mesh statistics like Tx/Rx speeds, network interfaces, network routes etc.

• Document CG-NMS registration of CGEs through screenshots


Test Suite Remote Management and Notifications


Priority High

Test Case Details

Title CG-NMS CSMP data integrity and authenticity

ID


Owner(s) adewanga

Description The CGE should successfully verify CSMP message signature from CG-NMS




• Make sure CG-NMS is signing all CSMP messages sent to the CGEs. Browse to Admin > Certificates > Certificate for CSMP in CG-NMS.

• Configure the CGEs for security mode and signature validation for Post and Responses through xml configuration:

http://solutionpartner.cisco.com/web/cegd/configure-security-

with TIMS



in-starter-kit

• Reboot the CGEs and make sure they register with CG-NMS

• Query TLV 78 and 79 on the CGE CM App to make sure that the certificate is being verified and the certificate contents

• Perform a CSMP Post (or a firmware download) and make sure that the operation is successful

• Document the successful post operation/firmware download




Priority High

Test Case Details

Title CG-NMS firmware upgrade over unicast CSMP messages

ID


Owner(s) adewanga

Description Firmware upgrade of CM/IE firmware from CG-NMS should work over unicast CSMP messages


Procedure • Repeat the procedure documented in CGE_PRE_IVT_MGMT_TC002

• Perform a firmware upgrade through CG-NMS. Follow procedure documented here:

http://solutionpartner.cisco.com/web/cegd/tutorial-cgnms#OTA

• Make sure that the firmware download to the CGE is using unicast messages

• Make sure the firmware download is successful

• After the firmware download is completed, schedule a reload time

• Make sure that the CGE CM/IE board reloads at the scheduled time

• After CGE reloads, make sure that it joins the CG-Mesh and requests and receives IPv6 address successfully

with TIMS



• Make sure the CGR registers with CG-NMS

• Verify the statistics reported in CG-NMS and make sure that the new firmware version is reported

• Also query TLV 11 from the Field tool to make sure that the firmware download was successful

• Query TLV 75 to verify the contents of the downloading, backup and running firmware slots



• Document the successful firmware download through screenshot in CG-NMS and logs.




Priority High

Test Case Details

Title CG-NMS firmware upgrade over multicast CSMP messages

ID


Owner(s) adewanga

Description Firmware upgrade of CM/IE firmware from CG-NMS should work over multicast CSMP messages


Procedure • Repeat the procedure documented in CGE_PRE_IVT_MGMT_TC003

• Make sure that the firmware download to the CGE is using multicast messages



with TIMS



• Document the successful firmware download through screenshot in CG-NMS and logs.




Priority High

Test Case Details

Title PON/PRN notifications (RF and dual-phy only)

ID


Owner(s) adewanga

Description Verify PON/PRN notifications



• Make sure that the PAN environment is configured for security including 802.1x and 802.11i, and the CGEs are configured with the credentials as well as all required certificates (for 802.1x, CG-NMS and Application Server(s) if applicable)

• Make sure the CGEs join the mesh and can communicate using pings. Make sure CGEs have registered to CG-NMS and CG-NMS is configured for CSMP signing.

• Make sure CGR is configured to relay the PON/PRN messages to the CG-NMS.

• Introduce Power Outage event in the multi-hop leaf CGE for a period of at least 30 seconds

• The CGE should broadcast 3 PONs over a 15 second period to its neighbors and the CG-NMS would forward the PON along with the EUI64 and timestamp to CG-NMS

• Make sure PON is seen in CGR: show wpan outage table

• Make sure PON is notified in CG-NMS. The CGE should be marked with ‘Outage’

• Restore power on the CGE.

with TIMS



• Restoration messages would be broadcast in Layer 2 and CG-NMS will be notified by CGR

• Make sure PRN is seen in CGR: show wpan restoration table

• CG-NMS should indicate that power is restored successfully

• Repeat the test with one hop CGE

• Repeat the test will all the CGEs in the topology

• Document the CGR CLI outputs as well as CG-NMS screenshots




Priority Medium

Test Case Details

Title Capability to derive Application Server(s) address from DHCP Option 17

ID


Owner(s) adewanga

Description While configuration of IPv6 address through DHCPv6, the DHCP server returns Option 17 values to the CGE.

The Option 17 contains a list of applications and the related IPv6 addresses of the Application Server in the Head-End. The Option 17 list including indexes and service names, is utility specific (and can be PAN specific) and should be programmable in the CGE





with TIMS



• The CGE should have the capability to either program the parsing algorithm to derive the Application Server Address from Option 17 OR should mandate the use of a partner specific Option 17 service name which the customer will use.

• Depending on the solution above, program the CGEs to derive the Application server address from Option 17

• Using field tool, validate that the correct Application Server Address was derived

• Document the results




Priority High

Test Case Details

Title LMAC FWDL (PLC and dual-phy only)

ID


Owner(s) linyan

Description Verify FWDL capability of LMAC using unicast and multicast


Procedure • Follow same procedure as CGE_PRE_IVT_MGMT_TC003 for setup, but upgrade LMAC image instead of the CG-Mesh App

• Repeat the procedure as described in CGE_PRE_IVT_MGMT_TC004 for LMAC firmware upgrade using multicast.

Test Phase System Test



Priority High

Test Case Details

with TIMS

with TIMS



Title Dual-phy CSMP TLVs (dual-phy only)

ID


Owner(s) linyan

Description Test the Dual Phy CSMP TLV added, namely: • LowpanPhyCounters • LowpanRfStats • LowpanPlcStat • LowpanPhyStats




• Make sure all CGEs are deployed and have IPv6 addresses assigned and are reachable through the CGR. Query CSMP TLV 43 from the host port/Field application to make sure the correct CG-NMS IPv6 address is received

• Query the following CSMP TLVs specific to dual-phy:

LowpanPhyCounters

LowpanRfStats

LowpanPlcStats

LowpanPhyStats

• Verify the counter values are as expected




Priority High

7.6 CGE Application (Pre-IVT):

Test Case Details

Title Application Server functionality (Meter reads, SCADA Comms etc) with TIMS



ID

Logical ID CGE_PRE_IVT_APP_TC001

Owner(s) adewanga

Description Application Server should be able to communicate with and manage the endpoints


Procedure • Make sure the Application Server is connected to CG-REDI using VPN and has a valid and reachable IPv4 and IPv6 address

• Make sure routing is configured between the CGR and the Application Server for reachability

• Make sure that the DHCP server Option 17 serves out the option for the partner specific Application server IPv6 address.



• Using the field tool, make sure that the CGEs have the correct Application Server address configured

• If applicable, make sure that the CGEs register themselves to the Application Server

• Make sure that the Application Server learns about the CGEs (through registration or configuration) and is able to manage the endpoint

• Make sure the connection/connectivity between the Application Server and the CGE is successful (TCP/UDP)

• Make sure that Application server functionality is successful eg., SCADA Master commands, on-demand DA or Meter reads, periodic notifications from CGEs etc


Test Suite CGE Application


Priority High



Test Case Details

Title Application QoS markings (bidirectional)

ID


Owner(s) adewanga

Description • The Application should mark the correct QoS values

• No packet should be marked for High queue




• Make sure the CGEs join the mesh and can communicate using pings. Make sure CGEs have registered to CG-NMS and Application Server(s) if applicable, and CG-NMS is configured for CSMP signing.

• Refer Table 4.2 for reference of QoS priority for traffic

• Create on-demand application traffic event eg. Meter reads, if applicable

• Using tools on HER and CGR like ACLs, Policy-maps note that the QoS values in the IPv6 packets

• Make sure that packets from Application Server to the CGEs have the suggested QoS markings of Medium (AF21)

• Make sure that packets from the CGEs to the Application Server have the suggested markings of Normal (AF11)

• Measure and validate the QoS values in periodic meter reads/reports as Normal (AF11) for traffic in both directions, if applicable

• Measure and validate the QoS values for (meter) notification events as Medium (AF21) for traffic in both directions, if applicable

• Validate the QoS markings for HAN traffic as Low (0)

• QoS for any configuration or firmware upgrade should also be Low (0)

• No application traffic should be marked for High queue (AF3x)

• Test DSCP values for any other major traffic types for the

with TIMS



CGE and validate

• Document results and QoS markings for different traffic types




Priority High

Test Case Details

Title Time synchronization of Application Server and AM firmware for event correlation and DA use-cases

ID


Owner(s) adewanga

Description Application Server should time synchronize with head-end servers for event correlation and DA use-cases



• Make sure that the PAN environment is configured for security including 802.1x and 802.11i, and the CGEs are configured with the credentials as well as all required certificates (for 802.1x, CG-NMS and Application Server(s) if applicable),


• Time synchronization is required across the CG-Mesh and the CGEs for AMI time stamping in Meter reads, notification, log entries etc. (10 ms accuracy)

• Time synchronization is also required for fault detection, classification, location, isolation and restoration services in DA (10 us accuracy)

• Time synchronization for other use-cases are identified in Zanzibar and Zanzibar should be used as reference for time accuracy needs

• Verify that the time on the AM/IE Application is synchronized

with TIMS



with head-end servers as per the requirements (exact procedure TBD)

• Document procedure and protocols used for AM time synchronization (NTP, IEEE 1588, GPS, SCADA)




Priority High

Test Case Details

Title Application protocol “mesh-friendliness”

ID


Owner(s) adewanga

Description Application protocol being used should be “mesh-friendly”


Guidelines • Make sure all provisioning for the partner setup is complete and working between CG-REDI and partner premises, including CGR and Application server connectivity.



• The application protocol being used (eg. C12.22, DLMS/COSEM, SCADA) should be designed to run over Low Power Lossy Networks (LLNs)

• The application protocols should support periodic statistics reporting instead of polling mechanisms, wherever applicable

• The application protocols should support “randomization” in accessing the CG-Mesh, so that all CGEs do not attempt to send messages together, in use-cases like periodic notifications

• Connection based services (eg. TCP) and keepalives are not suggested, as these are expensive operations in larger scales

• The application/transport protocol lightweight reliable delivery of

with TIMS



messages (ACKs), where applicable

• The application protocol should be tested to scale over large NAN deployments (1000s of devices) over extended period of time

• Make sure that the features of the application are performing in scale like periodic statistics notification, periodic meter reads, on-demand events etc.

• The features and functionality of other devices in the topology should be monitored eg. Periodic notifications, on-demand events like DA control and meter reads etc.

• Other procedures TBD




Priority High

Test Case Details

Title IPv6 MTU and fragmentation considerations

ID


Owner(s) adewanga

Description For IE model, verify fragmentation and reassembly behavior (optional).

For PPP model, ensure that a minimum MTU of 1280 bytes is supported by AM firmware without fragmentation.

For PPP model, document fragmentation and reassembly behavior for the AM firmware

For both models, ensure no IPv6 packet length exceeds 1280 bytes




• Make sure the CGEs join the mesh and can communicate using pings. Make sure CGEs have registered to CG-NMS and Application

with TIMS



Server(s) if applicable, and CG-NMS is configured for CSMP signing.

• In all the tests here, make sure to initiate the pings from a device with interface MTU of 1280 bytes. This will ensure IPv6 fragmentation for any packet with IPv6 length greater than 1280 bytes. One option is to use SSL VPN to CG-REDI for initiating the ping and configure the SSL VPN MTU to 1280 bytes (default setting for CG-REDI)

• (Optional) For IE model, make sure that ping to CGE with 1280 bytes length IPv6 packet is successful. From Windows: ping –l 1232 <ipv6_address>

• (Optional) For IE model, make sure that ping to CGE with 1281 bytes length IPv6 packet is unsuccessful. From Windows: ping –l 1233 <ipv6_address>

• (Optional) Capture ping packets in both the above cases, and ensure that no fragmentation is taking case in 1280 byte ping and fragmentation is seen in 1281 byte ping

• For PPP/NIC model, ping the AM module with 1280 byte length IPv6 packet. This should be successful. Make sure through packet captures that no fragmentation is taking place in the network in either direction i.e. Echo request or reply

• For PPP/NIC model, ping the AM module with 1281 byte length IPv6 packet. If AM performs reassembly of Echo Request and fragmentation of Echo Reply, the ping will be successful. Else ping will fail. Make sure none of the IPv6 fragments exceed 1280 bytes in length. Document the results and packet capture.

• For PPP/NIC model, initiate ping with large packet size of 1500 bytes from the AM module. If fragmentation is supported, make sure that the IPv6 fragments are not larger than 1280 bytes for IPv6 packets in either direction. If fragmentation is not supported, then the IPv6 Echo request will not be sent and the ping will fail.

• For both models, verify that all application traffic packets have MTU equal to or lesser than 1280 bytes in either direction. Use policy map on CGR and match the packet size to confirm.

• Document results of all steps above



Pass/Fail Criteria • Ping to AM module with 1280 bytes should be successful and no fragmentation should be seen in either direction.

• If fragmentation and reassembly of IPv6 packets is supported, then ping with packets larger than 1280 bytes should result in maximum fragment size of 1280 bytes to avoid PMTUD and ICMP packet exchanges.

• None of the packets exchanged between head-end applications and CGE should exceed length of 1280 bytes.

Priority High



Test Case Details

Title Conformance to L3 packet size of 800 bytes

ID


Owner(s) adewanga

Description Packet size for application traffic should not exceed L3 packet size of 800 bytes





• Verify that no application traffic Layer 3 packet size exceeds 800 bytes in either direction. L3 packet size should not exceed 800 bytes to avoid L2 fragmentation in the CG-mesh, which will lead to inefficient use of mesh bandwidth and may result in reordering of packets.




Priority Medium

7.7 Fault resiliency and availability:

Test Case Details

Title Recovery after power failure of CGE

ID

with TIMS

with TIMS



Logical ID CGE_PRE_IVT_FRA_TC001

Owner(s) adewanga

Description Recovery after power failure of CGE





• Take a note of the RPL topology and verify that it conforms to topology in Figure 3.3

• Reboot all the CGEs

• Make sure that all CGEs join the CG-Mesh after reboot, in the same RPL topology as Figure 3.3 (RPL may take several hours to converge). Verify IPv6 connectivity and forwarding for all the CGEs. Make sure for PPP model, AM is reachable.

• For PPP, repeat the test and reload just the CM. Make sure that after RPL converges, both CM and AM are reachable.

• Make sure that the CM/IE and the AM is reachable from CG-NMS and Application server.

• Make sure that Application server functionality is successful


Test Suite Fault resiliency and availability


Priority High

Test Case Details

Title Recovery after power failure of FAR

ID


with TIMS



Owner(s) adewanga

Description CGEs should be able to rejoin the CG-Mesh after power failure of FAR





• Take a note of the RPL topology and verify that it conforms to topology in Figure 3.3 (RPL may take several hours to converge)

• Reboot the CGR (RPL DAG root)

• Make sure that all CGEs join the CG-Mesh after CGR reboot, in the same RPL topology as Figure 3.3. Verify IPv6 connectivity and forwarding for all the CGEs. Make sure for PPP model, AM is reachable.






Priority High

Test Case Details

Title RPL convergence after node failures

ID


Owner(s) adewanga

with TIMS



Description RPL should converge after node failures in the CG-Mesh network





• Make sure the RPL topology reflects the CGE topology in Figure 3.3

• Move Node 4 in next-hop range for Node 2 and fail Node 3 (shutdown)

• Make sure Node 4 joins the CG-Mesh with Node 2 as the next-hop by querying TLV 52 in Node 4, verifying the RPL topology in CGR and looking at routes as reported by Node 4 to CG-NMS






Priority High

8 Detailed Test Cases (IVT)

8.1 CGE HW Schematics and diagnostics (IVT):



Test Case Details

Title Review HW Schematics

ID Twg306736c

Logical ID CGE_IVT_HW_TC001

Owner(s) chwilso3

Description Review Hardware Schematics of the partner implementation of Cisco’s Hardware Reference Design (CGEREF1/CGEPLCREF2/ CGEREF3/CGEPLCREF3)

Test Setup N/A

Procedure • Review submitted design for any deviations from Cisco's Connected Grid Endpoint reference design.

• Review for any missing items that are considered required.

• Review for HW design best practices.

• Record all violations, observations, and recommendations.




Priority High

8.2 RF/PLC physical and MAC layer (IVT):

Test Case Details

Title Transmit Power

ID Twg306740c

Logical ID CGE_IVT_SIG_TC001

Owner(s) mathaker, adewanga

Description Measure transmit power of CGE during normal operation


Procedure • Leave the transmit power configuration of the CGE at default

• During normal operation of the endpoint (not running

with TIMS

with TIMS



MCU diags or other test apps), verify that the transmit power of the node as recorded by the CGR's WPAN neighbor table display is within acceptable range.

• RF Setup: CGR at a 3 ft distance from the node. Node with 50dB of attenuation on the antenna. The acceptable RSSIR value should be in the -45 dBm to -55 dBm range.

• (NB: need to specify the details for PLC as well; Actual PLC parameters will depend on the PA power supply which may vary from vendor to vendor)

• Program an iTron meter to join the same mesh and with default transmit power.

• Compare the transmit power of the iTron meter and the CGE relatively as measured by the CGR, when they are placed adjacent to each other.

• Any discrepancies should be noted against the test results

• Modify the transmit power on the CGE through configuration and verify the transmit power of the node as recorded by the CGR’s WPAN neighbor table. Verify that the change is transmit power through configuration is reflected in the difference between the current and earlier measurements.


Test Suite RF/PLC physical and MAC layer


Priority High

Test Case Details

Title RSSI/LQI Comparative test

ID Twg306741c


Owner(s) mathaker

Description Perform a comparative test for RSSI/LQI values against CGEREF1/CGEPLCREF2/CGEREF3/CGEPLCREF3 reference platforms


Procedure • Compare the transmit and receive power levels of the

with TIMS



endpoint under test with Cisco's reference platform. (CGEREF1/CGEPLCREF2/CGEREF3/CGEPLCREF3)

• Place the endpoint with 50 dB of attenuation at a 3 ft distance from the CGR. Make sure CGR 'txpower' has been set to 'high'.

• Place a reference RF platform at the same distance with the same attenuation.

• Make sure both join the mesh.

• Make sure that the RSSIR and RSSIF for both nodes are within +/- 5 dB of each other.

• Record the result and obtain an explanation from the vendor for any discrepancy larger than the recommend value.

• (NB: need to specify the details for PLC as well).




Priority High

Test Case Details

Title PLC Modulation and throughput (PLC only)

ID


Owner(s) linyan

Description Verify modulation scheme corresponding to the modulation scheme.


Procedure • During normal operation of the endpoint, enforce node to operate in these modulations:

ROBO

BPSK

DBSK

8DBSK

• Make sure that the endpoint supports all four-modulation techniques by ensuring communication using each modulation.

with TIMS



• Enforce Node 2 with different modulation schemes on uplink to CGR and downlink to another node. Make sure that communication is working with Node 3 and Node 4.




Priority High

Test Case Details

Title Throughput comparison test (Dual-phy only)

ID


Owner(s) linyan

Description Perform a comparative test for throughput values between dual-phy and single-phy platforms


Procedure • Test to ensure that when dual-phy is operating in RF, it should have comparable performance numbers as a single-phy RF node. Typically, 75kbps for single RF and 100 ~150 ms latency for first hop single RF node.

• Test to ensure that when dual-phy is operating in PLC, it should have comparable performance number as single PLC phy which is around ¼ of RF

o Need to force the dual-phy to operate using PLC medium, as by default, RF is always used.

o Use test utility to setup back-to-back traffic, capture the result for the rate. Compare the result from dual phy board vs the single-phy board. The numbers should be comparable.




Priority High

with TIMS



8.3 Workforce enablement (IVT):

Test Case Details

Title User and device authentication in the field tool

ID Twg306743c

Logical ID CGE_IVT_WFE_TC001

Owner(s) adewanga

Description The CGE device management field tool access should be secured through user and/or device authentication

Test Setup N/A

Procedure • Same procedure as CGE_PRE_IVT_WFE_TC001



Pass/Fail Criteria All criteria described in procedure should pass for the test case to pass. Exceptions should be noted.

Priority High

Test Case Details

Title Capability to upgrade CM/IE application firmware and LMAC firmware (for PLC and dual-phy) in the field

ID Twg306744c


Owner(s) adewanga

Description Capability to upgrade CM/IE firmware and LMAC firmware (for PLC and dual-phy) in the field

Test Setup N/A




with TIMS

with TIMS



Pass/Fail Criteria Described in procedure.

Priority High

Test Case Details

Title Document capability to upgrade CM/IE boot loader in the field

ID Twg306745c


Owner(s) adewanga

Description Document capability to upgrade the CM/IE boot loader in the field

Test Setup N/A

Procedure • Cisco does not recommend capability of upgrading CM/IE boot loader, as this would allow a hacked boot loader to be used in conjunction with custom CM/IE firmware, applications and configuration.

• Document the level of upgradeability that the CGE has:

-CGE Field tool provides mechanism to upgrade the boot loader

-The boot loader can be updated via available JTAG interface

-The boot loader can be updated through available UART serial interface (serial boot loader) of the STM MCU on the CGE

-Any other mechanism through which the boot loader can be upgraded



Pass/Fail Criteria This test case does not have pass/fail criteria.

Priority Medium

Test Case Details

Title Document capability to upgrade CM/IE configuration in the field

ID Twg306746c


with TIMS

with TIMS



Owner(s) adewanga

Description Document capability to upgrade CM/IE configuration in the field

Test Setup N/A

Procedure • Cisco does not recommend capability to upgrade the CM or IE communication configuration binary in the field, as CGE configuration including ‘birth’ certificates should be tied to a CGE during manufacturing.

Field upgrade of configuration also adds security risk of transportation and installation of CGE certificates.

• Document capability to upgrade the configuration similar to CGE_IVT_WFE_TC003




Priority Medium

Test Case Details

Title Data privacy for field tool

ID Twg306747c


Owner(s) adewanga

Description Communication between field tool and the CGE should be secure

Test Setup N/A




Pass/Fail Criteria Described in procedure. Exceptions should be noted.

Priority High

Test Case Details

with TIMS

with TIMS



Title Debugability through CSMP

ID Twg306748c


Owner(s) adewanga

Description The CGE should provide a mechanism to query CSMP TLV from the CM/IE kernel

Test Setup N/A




Pass/Fail Criteria All criteria described in procedure should pass for the test case to pass.

Priority High

Test Case Details

Title Network debugability through ICMP pings

ID Twg306749c


Owner(s) adewanga

Description The CGE should provide a mechanism to ping IPv6 nodes in the network

Test Setup N/A





Priority High

with TIMS



Test Case Details

Title Capability to reboot/reset communication HW

ID Twg306750c


Owner(s) adewanga

Description Capability to reset the CM/IE hardware MCU should be available

Test Setup N/A

Procedure • Capability to reset the CM/IE MCU should be available. Without this ability, the CGE may have to be power cycled following a field firmware upgrade, which is an expensive and time consuming (backup battery) activity.

• Document procedure for reset




Priority Low

Test Case Details

Title Capability to query networking configuration

ID Twg306751c


Owner(s) adewanga

Description Capability to query networking configuration should be available

Test Setup N/A





Priority High

with TIMS

with TIMS



Test Case Details

Title Capability to upgrade AM boot-loader, application firmware and configuration in the field

ID Twg306752c


Owner(s) adewanga

Description Make sure that the AM application firmware is securely upgradable in the field. Document capability to upgrade boot-loader and configuration for the AM

Test Setup N/A

Procedure • The CGE should provide a mechanism to upgrade AM firmware in the field

• Document capability to upgrade AM boot-loader (not suggested) and configuration file




Priority Medium

8.4 CG-Mesh Connectivity (IVT):

Test Case Details

Title CSMP QoS markings

ID Twg306760c

Logical ID CGE_IVT_MESH_TC001

Owner(s) adewanga

Description QoS marking for CSMP message from CM/IE kernel



with TIMS

with TIMS



• Make sure all CGEs are deployed and have IPv6 addresses assigned and are reachable through the CGR.

• Deploy CSMP GUI/CLI on an server in CG-REDI or the Application Server and query TLVs on the CGE

• Make sure to capture CSMP packets traversing the CG-REDI infrastructure to and from the CGE

• Verify the QoS markings on the IPv6 packets of the CSMP messages from the CGE to the CSMP GUI/CLI as Normal (AF11)

• Verification can be done on the CGR using ACLs, Policy maps etc.




Priority Medium

Test Case Details

Title Organizationally Unique EUI64 values for CG-Mesh Communication

ID Twg306761c


Owner(s) adewanga

Description EUI64 values for the CGE should be globally unique

Test Setup N/A

Procedure • Partner should use an IEEE Registration Authority assigned Organizationally Unique Identifier (OUI) for the first 3-bytes of the EUI-64 for all their devices.

The remaining 40 bits of the EUI-64 should be unique for every device that is manufactured.

• Document the EUI64 derivation procedure for the CGEs: Through xml configuration or algorithmic derivation

• Note the procedure for algorithmic derivation of the EUI64 value and make sure that IEEE assigns the OUI being used.

• If partner is using Cisco’s OUI, note this in the test report as undesirable.


with TIMS





Priority Medium

Test Case Details

Title Preference of physical medium (dual-phy only)

ID


Owner(s) linyan

Description RF is the preferred medium for dual-phy endpoints


Procedure • Same procedure as CGE_PRE_IVT_MESH_TC008




Priority High

Test Case Details

Title Use of advanced metrics for physical medium choice (dual-phy only)

ID


Owner(s) linyan

Description Logic to switch physical medium




with TIMS

with TIMS






• When RF media is not available, the CGE should attempt to use PLC. As of now, the logic is to try using PLC when RF is not available (RSSIR > 95 dbm). Later, when more logic is added such as latency and throughput metric, these factors will be used in the physical medium decision-making. Ensure that the physical medium is selected as per the ascertained metrics (more details TBD).

• Query the following TLVs to verify test results: LowpanPhyCounters LowpanRfStats LowpanPlcStats




Priority Medium

Test Case Details

Title RPL advertisement (dual-phy only)

ID


Owner(s) linyan

Description Dual-phy RPL advertisement




http://solutionpartner.cisco.com/web/cegd/dual-phy-

with TIMS



configuration



• Configure RF with more aggressive beaconing rate than PLC.

• Check the bridge log (debug wpan packet) that RPL advertisement messages are sent properly when there are different configuration for RF and PLC and CGE could sync up with bridge root’s schedule.




Priority Medium

Test Case Details

Title Use of advanced RPL metrics (dual-phy only)

ID


Owner(s) linyan

Description Dual-phy RPL formation metrics







with TIMS



• As of now, we are always using RF as the primary media for link layer and then shift to PLC when RF failed. In future firmware releases, advanced metrics will be added for the RPL formation, such as latency matrix or throughput. Ensure that the endpoints are creating the RPL tree as per the metric values (more details TBD).

Test to ensure that RPL formation will be using below matrix: • ETX • Latency Matrix • Throughput Matrix




Priority Medium

Test Case Details

Title Multi-phy multi-hop test (dual-phy only)

ID


Owner(s) linyan

Description Mesh network with different mediums at various hop level

Test Setup Figure 3.7.1, 3.7.2

Procedure • Same procedure as CGE_PRE_IVT_MESH_TC009




Priority High

8.5 CG-Mesh Security (IVT):

Test Case Details

with TIMS

with TIMS



Title Access Control with 802.1X EAP-TLS

ID Twg306763c

Logical ID CGE_IVT_SEC_TC001

Owner(s) adewanga

Description Make sure client functionality for access control using 802.1x EAP-TLS is working


Procedure • Same procedure as CGE_PRE_IVT_SEC_TC001, but topology as in Figure 3.6




Priority High

Test Case Details

Title Access control for unauthenticated/untrusted certificate on device (device rejection)

ID Twg306764c


Owner(s) adewanga

Description Untrusted devices and devices with incorrect certificates should not be permitted to join CG-Mesh



• Make sure PKI and AAA/Radius servers are provisioned and configured in CG-REDI.

• Configure CGR for mesh security, mesh security keys (GTK) and AAA/Radius authentication for the PAN

• One or more CGEs should not be configured with security

• Make sure the CGEs without the credentials do not join the CG-Mesh. All other trusted CGEs should still be able to join and communicate over the mesh

with TIMS



• Now, load certificates from untrusted CA in one or more CGEs

• Modify 802.11i GTK (mesh-security keys) on FAR to ensure cold start and reboot untrusted CGEs

• Make sure the untrusted CGEs do not join the CG-Mesh. All other trusted CGEs should still be able to join and communicate over the mesh




Priority Medium

Test Case Details

Title Firmware upgrade with secure boot loader

ID Twg306766c


Owner(s) adewanga

Description Trusted CM/IE application firmware should be successfully verified by secure boot loader and the upgrade should work


Procedure • Same procedure as CGE_PRE_IVT_SEC_TC003, but topology as in Figure 3.6




Priority High

Test Case Details

Title Failed untrusted firmware upgrade with secure boot-loader

ID Twg306767c

with TIMS

with TIMS




Owner(s) Adewanga

Description Untrusted CM/IE kernel application firmware fails to boot





• Perform a firmware upgrade using an unsigned CM/IE application firmware using the field tool

• The firmware upgrade should fail

• Perform a firmware upgrade using an untrusted CM/IE application firmware using the field tool

• The firmware upgrade should fail




Priority High

Test Case Details

Title Secure mesh with dual-phy (dual-phy only)

ID


Owner(s) linyan

Description Ensure secure mesh forms when switching between: 1. Dual-phy configuration 2. Single-phy RF configuration 3. Single-phy PLC configuration 4. Back to dual-phy configuration

Key is to make sure the master /slave are adjusted properly and not create any glitch for the network formation

with TIMS







• All dual-phy CGEs should be configured with security

• Make sure RF module is configured as the master. Verify all nodes are reachable.

• Shift from above to make it a single PLC network (power down RF, make PLC the master). Verify all nodes are reachable.

• Shift from above to make it a single RF secure network (power off PLC and power up RF). Verify all nodes are reachable.

• Shift from above back to a secure network with both RF and PLC (make RF a master and PLC a slave again). Verify all nodes are reachable.




Priority High

8.6 Remote Management and Notifications (IVT):

Test Case Details

Title CSMP over CG-Mesh

ID Twg306769c

Logical ID CGE_IVT_MGMT_TC001

Owner(s) adewanga

Description CSMP communication should work between the head-end and CM/IE kernel


with TIMS





• Deploy CSMP GUI/CLI on an server in CG-REDI or the Application Server at partner premises and query TLVs on the CGE

• CSMP communications with the CM/IE kernel should be successful




Priority High

Test Case Details

Title CG-NMS registration, reachability, configuration template and periodic updates

ID Twg306770c


Owner(s) adewanga

Description CGE CM/IE kernel should register with CG-NMS and should be manageable


Procedure • Same procedure as CGE_PRE_IVT_MGMT_TC001, but topology as in Figure 3.6




Priority High

Test Case Details

with TIMS

with TIMS



Title CG-NMS CSMP data integrity and authenticity

ID Twg306771c


Owner(s) adewanga

Description The CGE should successfully verify CSMP message signature from CG-NMS






Priority High

Test Case Details

Title CG-NMS CSMP missing/invalid signature

ID Twg306772c


Owner(s) adewanga

Description CSMP messages without a signature or invalid signature should be rejected by the CGEs





• Remove the NTLS link between CG-NMS and HSM to disable CSMP signing.

• Configure the CGEs for security mode and signature validation for Post and Responses through xml configuration.

with TIMS



• Use public certificate of the HSM in der format and use cfgwriter to create a configuration binary.

• Load the configuration binary to the CGE using JLink/Field tool

• Reboot the CGEs and make sure they register with CG-NMS

• Perform a CSMP Post (or a firmware download) and make sure that the operation fails because of lack of signature in the CSMP messages

• Next, bring up the NTLS link between the HSM and CG-NMS and make sure CG-NMS is signing all CSMP messages sent to the CGEs

• Load an X.509 certificate from a CA (other than the HSM) in the CGE for CSMP verification and reboot

• Make sure the CGE registers with CG-NMS

• Perform a CSMP Post (or a firmware download) and make sure that the operation fails because of invalid signature in the CSMP messages




Priority High

Test Case Details

Title CG-NMS firmware upgrade over unicast CSMP messages

ID Twg306773c


Owner(s) adewanga

Description Firmware upgrade of CM/IE firmware from CG-NMS should work over unicast CSMP messages for CGEREF1, CGEREF2 and CGEREF3





with TIMS




Priority High

Test Case Details

Title CG-NMS firmware upgrade over multicast CSMP messages

ID Twg306774c


Owner(s) adewanga

Description Firmware upgrade of CM/IE firmware from CG-NMS should work over multicast CSMP messages for CGEREF1, CGEREF2, CGEPLCREF2, CGEREF3, CGEPLCREF3






Priority High

Test Case Details

Title PON/PRN notifications (RF and dual-phy only)

ID Twg306775c


Owner(s) adewanga

Description Verify PON/PRN notifications for CGEREF1, CGEREF2, CGEREF3



• For dual-phy setup, test to ensure that the PON will be sent

with TIMS

with TIMS



when there is outage on the PLC side.




Priority Medium

Test Case Details

Title Capability to derive Application Server(s) address from DHCP Option 17

ID Twg306776c


Owner(s) adewanga

Description While configuration of IPv6 address through DHCPv6, the DHCP server returns Option 17 values to the CGE.

The Option 17 contains a list of applications and the related IPv6 addresses of the Application Server in the Head-End. The Option 17 list including indexes and service names, is utility specific (and can be PAN specific) and should be programmable in the CGE






Priority High

Test Case Details

Title LMAC FWDL (PLC and dual-phy only)

ID


Owner(s) linyan

with TIMS

with TIMS



Description Verify FWDL capability of LMAC using unicast and multicast


Procedure • Follow same procedure as CGE_PRE_IVT_MGMT_TC007 but use topology described in Figure 3.6 instead




Priority High

Test Case Details

Title Dual-phy CSMP TLVs (dual-phy only)

ID


Owner(s) linyan

Description Test the Dual Phy CSMP TLV added, namely: • LowpanPhyCounters • LowpanRfStats • LowpanPlcStat • LowpanPhyStats


Procedure • Follow same procedure as CGE_PRE_IVT_MGMT_TC008




Priority High

8.7 CGE Application (IVT):

Test Case Details

Title Application Server functionality (Meter reads, SCADA Comms etc) with TIMS

with TIMS



ID Twg306778c

Logical ID CGE_IVT_APP_TC001

Owner(s) adewanga

Description Application Server should be able to communicate with and manage the endpoints


Procedure • Same procedure as CGE_PRE_IVT_APP_TC001, but topology as in Figure 3.6




Priority High

Test Case Details

Title Application QoS markings (bidirectional)

ID Twg306779c


Owner(s) adewanga

Description The Application should mark the correct QoS values



• In addition, also document the QoS marking trends over a period of time as input and output from the WPAN interface of the CGR using policy-maps. Refer EDCS-1385630 for procedure. These trends should conform to the QoS guidelines posted in Section 4.13




Priority High

with TIMS



Test Case Details

Title Time synchronization of Application Server and AM firmware for event correlation and DA use-cases

ID Twg306780c


Owner(s) adewanga

Description Application Server should time synchronize with head-end servers for event correlation and DA use-cases






Priority High

Test Case Details

Title Document capability to perform AM application firmware, boot-loader and configuration updates over-the-air

ID Twg306781c


Owner(s) adewanga

Description Document capability to perform AM firmware, boot-loader and configuration updates over-the-air


Procedure • Document capability to perform AM firmware, boot-loader and configuration updates over-the-air



Pass/Fail Criteria There is no pass/fail criteria for this test

with TIMS

with TIMS



Priority Low

Test Case Details

Title Application protocol “mesh-friendliness”

ID Twg306782c


Owner(s) adewanga

Description Application protocol being used should be “mesh-friendly”


Guidelines • Same procedure as CGE_PRE_IVT_APP_TC004, but topology as in Figure 3.6




Priority High

Test Case Details

Title Document application data integrity and authenticity checks for Application messages

ID Twg306783c


Owner(s) adewanga

Description Document capability of the application firmware to verify signature of application messages from the application server


Procedure • Document capability of the Application Server to sign messages (CSMP Posts/Gets/Responses, application messages, configuration updates, firmware updates) sent to CGE

• Document capability of the CGE firmware (AM/IE App) to verify the signature in the messages

with TIMS

with TIMS



• Document any other application security mechanism like end-to-end encryption, authentication, authenticity and anti-replay mechanisms



Pass/Fail Criteria This test does not have pass/fail criteria

Priority Low

Test Case Details

Title Document capability and method to securely update (CM/IE) boot-loader and configuration over-the-air

ID Twg306784c


Owner(s) adewanga

Description Document capability and method to securely update (CM/IE) boot-loader and configuration over-the-air


Procedure • Document capability and method to securely update (CM/IE) boot-loader and configuration over-the-air

• Document the procedure and interface followed to upgrade the CM/IE boot-loader i.e. via JTAG port, serial boot-loader etc.




Priority Low

Test Case Details

Title Document device inventory, firmware and configuration management capabilities of Application Server(s)

ID Twg306785c


with TIMS

with TIMS



Owner(s) adewanga

Description Document device inventory, firmware and configuration management capabilities of Application Server

Test Setup N/A

Procedure • Document device inventory, firmware and configuration management capabilities of Application Server




Priority Low

Test Case Details

Title Document server virtualization capability

ID Twg332075c


Owner(s) adewanga

Description Document Application Server(s) virtualization capability

Test Setup N/A

Procedure • Document if the Application Server(s) are designed, built and capable of deployment in a virtualized environment

• Document if the Application Server(s) is specifically tested to work in UCS and VMware environment. Document UCS product, version and configuration. Document VMware version.




Priority Medium

Test Case Details

with TIMS

with TIMS



Title IPv6 MTU and fragmentation considerations

ID


Owner(s) adewanga

Description For IE model, verify fragmentation and reassembly behavior (optional).

For PPP model, ensure that a minimum MTU of 1280 bytes is supported by AM firmware without fragmentation.

For PPP model, document fragmentation and reassembly behavior for the AM firmware

For both models, ensure no IPv6 packet length exceeds 1280 bytes





Pass/Fail Criteria • Ping to AM module with 1280 bytes should be successful and no fragmentation should be seen in either direction.

• If fragmentation and reassembly of IPv6 packets is supported, then ping with packets larger than 1280 bytes should result in maximum fragment size of 1280 bytes to avoid PMTUD and ICMP packet exchanges.

• None of the packets exchanged between head-end applications and CGE should exceed length of 1280 bytes.

Priority High

Test Case Details

Title Conformance to L3 packet size of 800 bytes

ID


Owner(s) adewanga

Description Packet size for application traffic should not exceed L3 packet size of 800 bytes


with TIMS







Priority Medium

8.8 Fault resiliency and availability (IVT):

Test Case Details

Title Recovery after power failure of CGE

ID Twg306787c

Logical ID CGE_IVT_FRA_TC001

Owner(s) adewanga

Description Recovery after power failure of CGE






Priority High

Test Case Details

Title Recovery after power failure of FAR

ID Twg306788c


Owner(s) adewanga

Description CGEs should be able to rejoin the CG-Mesh after power failure of

with TIMS

with TIMS



FAR


Procedure • Same procedure as CGE_PRE_IVT_FRA_TC002, but topology as in Figure 3.6




Priority High

Test Case Details

Title PAN migration after DAG failure

ID Twg306789c


Owner(s) adewanga

Description CGEs should migrate to a backup PAN after DAG failure



• Make sure that the PAN environment is configured for security including 802.1x and 802.11i, and the CGE is configured with the credentials as well as all required certificates (for 802.1x, CG-NMS and Application Server(s) if applicable)

• Make sure the CGE join the mesh and can communicate using pings. Make sure that CGE has registered to CG-NMS and Application Server(s) if applicable, and CG-NMS is configured for CSMP signing.

• Provision a second CGR with CG-REDI and advertise the same SSID as the first but different PAN ID and advertising overlapping PAN as shown in Figure 3.5. Also provision mesh-security on the second CGR.

• Shutdown the PAN being advertised from FAR 1.

• Note the time for L2 migration of the CGE to the PAN advertised from FAR 2 as reported by CGR. The CGE would need to re-authenticate with the new CGR. Make sure that 802.1x EAP-TLS authentication is successful. Also make sure

with TIMS



that the new 802.11i mesh security keys are derived and exchanged successfully.

• Make sure new IPv6 configuration is successful on CM

• Make sure AM has requested and is leased out new IPv6 address within 60 seconds (this is a requirement for rapid convergence)

• Make sure RPL is advertising the new routes for the CGE’s CM and AM.


• Make sure that Application server functionality is successful (including registration, meter reads, SCADA communications etc.)




Priority High

Test Case Details

Title RPL convergence after node failures

ID Twg306790c


Owner(s) adewanga

Description RPL should converge after node failures in the CG-Mesh network


Procedure • Same procedure as CGE_PRE_IVT_FRA_TC003, but topology as in Figure 3.6




Priority High

8.9 Soak and Performance/Stress:

with TIMS



Test Case Details

Title Soak Test

ID Twg306798c

Logical ID CGE_IVT_SOAK_TC001

Owner(s) adewanga

Description CGE devices should function in a multi-vendor environment for long durations of time without service degradation


Procedure • Make sure the Application Server for the CGE DUT is connected to CG-REDI using VPN and has a valid and reachable IPv4 and IPv6 address

• Make sure routing is configured between the CGR and the Application Server for reachability

• Make sure that the DHCP server Option 17 serves out the option for the partner specific Application server IPv6 address.

• Make sure that the PAN environment is configured for security including 802.1x and 802.11i, and all the CGEs are configured with the credentials as well as all required certificates (for 802.1x, CG-NMS and Application Server(s) if applicable)

• Make sure the CGE join the mesh and can communicate using pings. Make sure that CGE has registered to CG-NMS and Application Server(s) if applicable, and CG-NMS is configured for CSMP signing.

• Make sure that Application server functionality is successful for all the CGEs including non-DUT CGEs eg. SCADA Master commands, on-demand DA or Meter reads, periodic notifications from CGEs etc

• Repeat the Application server functionality test and and verify the results periodically using CLI, UI, Scripts etc. for longer duration (in the order of days)


Test Suite Multi-vendor device interoperability


Priority High

with TIMS



Test Case Details

Title App performance with long duration large volume packet forwarding

ID Twg306800c

Logical ID CGE_IVT_STRESS_TC001

Owner(s) adewanga

Description Stress the CG-Mesh packet forwarding functionality while ensuring that Application functionality is still working over longer durations of time


Procedure • Make sure all provisioning of head-end network infrastructure services (PKI, DHCP, AAA/Radius, CG-NMS, Application Servers) for the PAN is complete and working


• Place the CGE DUTs at root of a sub-tree in the PAN eg. Node 2


• Make sure that Application server functionality is successful for all the CGEs including non-DUT CGEs eg. SCADA Master commands, on-demand DA or Meter reads, periodic notifications from CGEs etc

• Initiate traffic streams (file/firmware transfers, echo requests, CSMP TLV Requests etc) to the CGE and downstream nodes eg. Node 3, Node X etc. Make sure all traffic is successfully being forwarded by Node 1

• Initiate a unicast firmware download while continuing the forwarding of traffic flows downstream. Make sure firmware download is successful.

• Make sure that Application server functionality is successful while forwarding of Mesh traffic and firmware download is in progress.

• Continue test for longer durations and make sure that traffic is delivered reliably.


with TIMS



Test Suite Performance and Stress


Priority Optional

9 References

• Cisco Enabled Grid Device DevNet Website: http://solutionpartner.cisco.com/web/cegd/

• Cisco Connected Grid Endpoints – requirements, guidelines, best practices and considerations: http://solutionpartner.cisco.com/documents/3952742/19578379/CGE_Requirements_And_Guidelines.pdf

• CG-REDI architecture description: Link

cge ivt master test plan

Documents