onie: open network install environment @ osdc 2014 netways, berlin

Open Network Install Environment

NETWAYS OSDC 2014, Berlin

Nat Morris

9th April 2014

cumulusnetworks.com 2

•Overview

•What Is ONIE?

•Lessons Learned

•ONIE Development

•Demo

Agenda


IP Fabric Networking Landscape

Netw

ork

H

ard

ware

Netw

ork

O

S

Open Closed


The Expanding Landscape

hardware

operating system

appapp

hardware

operating system

app app

Single Vendor Blob

Multi-Vendor Ecosystem

app app

Understanding Characteristics of a Leaf Switch

5cumulusnetworks.com

10/40 Gigabit spine uplink ports

Serial console port

Ethernet Out-of-Band

Management Port

1/10 Gigabit Ethernetnetwork access ports

* SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options* QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options

Understanding Characteristics of a Spine Switch

6cumulusnetworks.com

Serial console port

Ethernet Out-of-Band Management Port

* QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options

40G / 10G Fabric ports


Bare Metal Switch Provisioning

Similar approach to installing OS on server

BIOS + PXE = U-Boot + ONIE (Open Network Install Environment)

Supported hardware (HCL) preloaded with ONIE

ONIE available on GitHub• http://onie.github.io/onie/

bare metal server

operating system

app

app

app

BIOS and PXEbare metal

switch

operating system

app

app

app

U-Boot and ONIE

http://onie.github.io/onie/


8

Choice

cumulusnetworks.com

9

Choice

cumulusnetworks.com

10

What Is ONIE?

Network OS Install Environment Provides an environment for network OS installer

discovery and execution Like a pre-installed kickstarter Defined by its behaviors Implemented using a modern Linux kernel and BusyBox

An Open Source Project within OCP http://www.onie.org/

Evolving …

cumulusnetworks.com

https://github.com/onie/onie/



11

Hardware Vendors

cumulusnetworks.com


Operating System Vendors

ONIE: Anatomy of a Network Switch


( Management Interfaces ) ( Data Plane )

CPUSoC

DRAMBootFlash

MassStorage

SwitchingASIC

SerialConsole

EthernetMgmt Port

10GbPort

40GbPort…10Gb

Port40GbPort

…

PCIe

ONIE: Uses Management Interfaces


( Management Interfaces ) ( Data Plane )

CPUSoC

DRAMBootFlash

MassStorage

SwitchingASIC

SerialConsole

EthernetMgmt Port

10GbPort

40GbPort…10Gb

Port40GbPort

…

PCIe

ONIE: Bare Metal Install – First Time Boot Up


Boot Loader(HW Vendor Supplied)

ONIE(HW Vendor Supplied)

Installer(OS Vendor)

Boot Loader• Low Level boot loader, configures CPU complex• Loads and boots ONIE

ONIE• Linux Kernel with Busybox• Configures management Ethernet interface• Locates and executes an OS installer• Provides tools and environment for installer

OS Installer• Available from network or USB• Linux executable• Installs vendor OS into mass storage

Network OS(OS Vendor Supplied)

Fetches

Installs

ONIE: Subsequent Reboots – Vendor’s OS is Already Installed


Boot Loader(HW Vendor Supplied)

ONIE(HW Vendor Supplied)

Boot Loader• Low Level boot loader, configures CPU complex• Loads and boots OS vendor’s installed OS

Network OS• Configures Switching ASIC• Runs Network Protocols• Provides CLI

Network OS(OS Vendor Supplied)

ONIE• Still exists, but is not used• Available for uninstall / re-install operations

ONIE: Network OS Installer Discovery and Install Behavior


Configure Network Interface

Locate Installer

Run Installer

• Uses DHCPv4, DHCPv6• Configures Ethernet interface for IPv4 / IPv6• Configures DNS and hostname

• Determines the location of an installer executable• Examines local file systems, e.g. USB flash drives• Uses DHCP options, DNS Service Discovery, Multicast DNS

and IPv6 Neighbors

• Downloads installer via URL• Passes various environment variables to installer• Launches installer

Why Not Use PXE?


• IPv6 and HTTP out of the box• Use existing Linux device drivers. No need to write new ones• Integrated automation


ONIE: Other Behaviors

Provides a mechanism for a network OS to invoke the above behaviors.

Reinstall Remove currently installed OS and return to the “out of box” provisioning state

Uninstall Completely wipe out everything, except ONIE

Rescue Reboot box into ONIE for repair, debug and forensics

Update Install a new ONIE version

Diag Run HW Vendor’s diag (optional)


Lessons Learned

Hardware Platforms Diverse mix of CPU complex designs, even

within a single CPU family. HW Vendors need freedom to customize ONIE Need mechanism to run HW vendor diag

Recurring Themes TLV based EEPROM format widely adopted Common HW designs reduce development time


ONIE Ongoing Development

Support Multiple CPU Architectures PowerPC – Today x86 – VM Prototype available today Thinking about ARM, MIPS Maintain ONIE behaviors across architectures

Testing and Compliance Enhance and extend regression test suites Develop compliance test suites


ONIE Ongoing Development

New Features DHCPv6 DNS Service Discovery / Multicast DNS

discovery HW Vendor diagnostic

Releases Quarterly release cadence Ongoing maintenance Enhancements


ONIE x86 Strategy

Use existing BIOS from hardware vendors During manufacturing install GRUB2 and

ONIE-x86 on the mass storage block device NOS Installer adds partitions, installs software

and updates GRUB2 configuration.


Block Device Partitioning

Partition using GUID Partition Table (GPT) format GPT supported by Linux, GRUB, UEFI Supports dual-booting multiple operating systems

Partition # Name R/W Notes

1 GRUB Boot r/o Used by GRUB and GPT

2 ONIE BOOT r/o ONIE Kernel

3 ONIE CONFIG r/w Configuration

4+ For NOS use

http://en.wikipedia.org/wiki/GUID_Partition_Table


GRUB Menu


NOS Installer Duties

Create new GPT partition(s) Create file systems on partition(s) Install NOS files into partition(s) Update ONIE-CONFIG using ONIE provided

CLIs onie-boot-entry-add onie-boot-entry-remove onie-boot-entry-show onie-boot-default onie-boot-update


Resources

ONIE Websites Main Page: http://www.onie.org/ Source Code: https://github.com/onie/onie/ Documentation: http://onie.github.io/onie/

http://www.onie.org/








© 2013 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.

Thank You!


Back Up Slides – Cumulus Background


Pushing Changes Down

CPU, RAM, Flash, etc. Switch Silicon

Front Panel Ports

lldpd

Routing Tables

ARP Table

Devices

Bridge FDB Filter Tables

Bonds VLANs

Lin

ux

Kern

el

Virtual Kernel Ports

Bridging

mstpd

ACLRouting Suite

Quaggasnmpd

vconfig

iptable

ebtable

ip6tableiproute

2

VXLAN Bridges

Switch HAL

brctl

SwitchDriver

Use

r Space

Quagga daemon, Quagga.conf, and vtysh

CLI and /etc/network/interfaces

switchd

Cumulus Networks and Partner Confidential 31

Cumulus Linux Hardware Compatibility List

40G Portfolio Part number DescriptionMerchant Silicon

Cumulus Linux Release

Dell S6000 32 x 40G-QSFP+ Trident II 2.1 or later

Edge-Core AS6700-32X 32 x 40G-QSFP+ Trident II 2.0 or later

Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later

Quanta QCT T5032-LY6* 32 x 40G-QSFP+ Trident II 2.0 or later



Agema AG-7448CU48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident 1.5.0 or later

Dell S481048 x 10G-SFP+ and 4 x 40G-QSFP+

Trident 2.0.x or later**

Edge-Core AS5600-52X48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident+ 1.5.0 or later


Trident II 2.1 or later**



Penguin Computing Arctica 4804X48 x 10G-SFP+ and 4 x 40G-QSFP+


Quanta QCT T-3048-LY2*48 x 10G-SFP+ and 4 x 40G-QSFP+


Quanta QCT T5048-LY8*48 x 10G-SFP+ and 6 x 40G-QSFP+




Edge-Core AS4600-54T 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later

Penguin Computing Artica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2 1.5.1 or later

Quanta QCT T1048-LB9* 48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or later

http://edge-core.com/prodCat.asp?c=1

http://www.penguincomputing.com/products/network-switches

http://www.quantaqct.com/en/01_product/01_list.php?mid=30&sid=114










onie: open network install environment @ osdc 2014 netways, berlin

Technology

onie onie available

os network os

onie hw vendor

onie behaviors

cli network os os vendor

network os installer

boots onie onie linux

installed os