© 2017 Nokia1
Configuration Management- Large Systems challenges
Kari Rossi
Bell Labs Fellow, Head of Technology and Architecture
Network ManagementApplications and Analytics
Nokia
2
• Definitions
• Good software design principles for configuration management
• Infrastructure as Code – software configuration management
• Modeling configurations
• Examples: network management
Contents
33 © Nokia 2016
Definitions
4
• CM is the practice of handling changes systematically so that a system maintains its integrity over time
• CM implements the policies, procedures, techniques, and tools that manage, evaluate proposed changes, track the status of changes, and maintain an inventory of system and support documents as the system changes
• CM programs and plans provide technical and administrative direction to the development and implementation of the procedures, functions, services, tools, processes, and resources required to successfully develop and support a complex system
Configuration Management (CM) definition (Wikipedia)
5
• In software engineering, software configuration management (SCM) is the task of tracking and controlling changes in the software, part of the larger cross-disciplinary field of configuration management
• SCM practices include revision control and the establishment of baselines. If something goes wrong, SCM can determine what was changed and who changed it
Software engineering - Configuration Management
6
• Configuration Management is an Information Technology Infrastructure Library (ITIL) and an IT Service Management (ITSM) process that tracks all of the individual Configuration Items (CI) in an IT system which may be as simple as a single server, or as complex as the entire IT department
• Configuration Management means software for managing the configuration and orchestration of of a large number of computers using a central configuration model. Examples: Ansible, Chef, Puppet
Configuration Management in production systems
• FCAPS is the ISO Telecommunications Management Network model and framework for network management. The goals of Configuration Management include:
• to gather and store configurations from network devices (this can be done locally or remotely)
• to simplify the configuration of the device
• to track changes that are made to the configuration
• to configure ('provision') circuits or paths through non-switched networks
• to plan for future expansion and scaling.
7
• What is a large software development project for you?
• What is large system for you?
• What is large software system for you?
Large projects and systems
8
• Millions of lines of code, lots of components and dependencies• Is part of a bigger system• 100 - 1000+ persons• Persons in several countries, several time zones• Strict non functional requirements (security, availability, …)• New non functional features being the most difficult ones to implement• Continuous integration within and between different systems
• Parallel and continuous release models multiplies the issues!
Some characteristics of large projects and systems
9
9
Good software design principles for configuration management
10
Good design principles
• What are good design principles for software?
• What is special for configuration management?
11
• Application as set of loosely coupled and collaborating services, each service implements a set of narrowly, related micro service
• Stateless design
• Externalize configuration
• Dynamic service registry and lookup with collaborating parties
• Treat backing service as attachable resource
• Use non-blocking asynchronous communication in a loosely coupled architecture
• Scale automatically using proactive and reactive actions cost-optimizes to run efficiently, not wasting resources
Cloud native principles - example
Refer to Twelve Factor App principles https://12factor.net
• Scale automatically using proactive and reactive actions cost-optimizes to run efficiently, not wasting resources
• Upgrades without downtime
• In development, build and run stages are strictly separated
12
• What is a small and big software development project for you?
• What is big system for you?
• What is big software system for you?
Code and configuration
• Why you should separate code from configuration?
13
• Configuration of an application are the installation and run-time parameters which are different in various deployments (SW development, customer environments)
• Examples• Installation parameters, such as number of servers, IP addresses, directories
• Language of an application, appearance (fonts, color), templates
• Resource handles to the database and other backing services
• Credentials to external services such as Amazon S3
• Applications should not store configuration as constants in the code• Configurations are different between different customer deployment, code is not
Separate of code and configuration
14
• Strict separation between the software development and live environments• For example, it is impossible to make changes to the code at runtime, since
there is no way to propagate those changes back to the build stage
Strictly separated software development and live stages
Source: Twelve Factor App principles https://12factor.net
15
15
Infrastructure as CodeSoftware Configuration Management
16
• Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools
• This Code replaces manual tasks and can be version controlled, automated tested, integrated to Continuous Integration/delivery pipelines
• Code that can be modular, re-usable and provides direct documentation of what really happens to the system
• Examples: Ansible, Chef, Puppet
Definition – infrastructure as Code
17
• An automation engine that automates cloud provisioning, configuration management, and application deployment
• Automation jobs are defined by Playbooks (YAML)• Ansible connects to the nodes, and pushes small programs (modules) to the nodes. The
modules are resource models of the desired state of the system• As Playbooks support multi-machine deployment system they are well suited to
deploying complex applications
• Playbooks can launch tasks synchronously or asynchronously• Playbooks can sequence multi-tier rollouts involving rolling updates, and can delegate
actions to other hosts, interacting with monitoring servers and load balancers along the way
Example - Ansible
18
• What is a small and big software development project for you?
• What is big system for you?
• What is big software system for you?
Complex software updates
• What kind of design principles could you use for designing a system so that you can update the software without downtime?
19
19
Modeling configurations
20
Basic concepts
Managed objects
ManagerClientManagement application
AgentServerDeviceNetwork ElementVirtual Network Function
Resources modelledas managed objects
Actions to managed objectstrigger the action to the corresponding real resources
21
Example
Managed objects
new_config.language=“fi”
manager.connect(host=“Mac Kari”, username=“root”);manager.edit_config(config=new_config);
What happens:- remote connection to “Mac Kari”- system language is set to “fi”
Same as the terminal command:
Mac Kari% sudo languagesetup –langspec 14
module user-settings {…leaf language {
type string;}…
Mac KariManager application
22
Modeling
• Think about examples of managed object classes, instances and operations for laptop management?
23
• The Network Configuration Protocol (NETCONF) is a network management• protocol developed and standardized by the IETF
• NETCONF provides mechanisms to install, manipulate, and delete the configuration of network devices
• NETCONF operations are realized on top of a simple Remote Procedure Call (RPC) layer. The NETCONF protocol uses XML based data encoding for the configuration data as well as the protocol messages
• YANG is a data modeling language for the definition of data sent over the NETCONF• YANG can be used to model both configuration data as well as state data of network
elements
NETCONF and YANG
24
• Installing, manipulate, delete the configurations of network devices
• Support both configuration state (e. g. IP address) and operational state (e. g. error counters)
• Supporting several configurations of a device, e. g. running configuration and configuration to be used next
• Distinguishing the distribution of the configuration from the activation of the configuration
• Reporting configuration change events
• Full configuration dumb and full configuration restore
What can be done with NETCONF?
25
25
Network ManagementExamples
26
Simple network management
x10NE NE NE NE x20
With small number of
network elements only
some automation is needed e.g. by custom scripts
With small number of network
elements operator can manage them by making terminal
connections
27
Network management systems
NE NE NE NE X100+NE NE NE NE
Network Management
System
I cannot manage my job manually and with simple
scripts
Network is changing
constantly, automation is
needed
28
Network management: CM configurations life-cycle
CM Actualconfiguration
CM newconfiguration
NENE Active
configuration
NE Pre-activated
configuration
CM
Upload
Create newconfiguration
Pre-activate(Download)
Activate
29
Examples of technical requirements for CM
• NE integration: auto-discovery vs. manual
• Number of elements: from 100s to 100 000s
• Configuration change response: from seconds to minutes lasting operations• Configuration size: from kilobytes to tens of gigabytes
• Configuration change activation: immediate vs. delayed - separation of transfer, validation and activation
• Operations mode: synchronous vs. asynchronous
• Frequency of data emitted: rare vs. frequent vs. continuous • Origins of changes: external (mgmt. system) vs. ”self-inflicted” (from air interface, SW
itself, …)