software update for iot embedded world 2017

Software update for IoTthe current state of play

Chris Simmonds

Embedded World 2017

Software update for IoT 1 Copyright © 2011-2017, 2net Ltd

License

These slides are available under a Creative Commons Attribution-ShareAlike 3.0 license. You can read the fulltext of the license herehttp://creativecommons.org/licenses/by-sa/3.0/legalcodeYou are free to

• copy, distribute, display, and perform the work

• make derivative works

• make commercial use of the work

Under the following conditions

• Attribution: you must give the original author credit

• Share Alike: if you alter, transform, or build upon this work, you may distribute the resulting work onlyunder a license identical to this one (i.e. include this page exactly as it is)

• For any reuse or distribution, you must make clear to others the license terms of this work


http://creativecommons.org/licenses/by-sa/3.0/legalcode

About Chris Simmonds• Consultant and trainer• Author of Mastering Embedded Linux Programming• Working with embedded Linux since 1999• Android since 2009• Speaker at many conferences and workshops

"Looking after the Inner Penguin" blog at http://2net.co.uk/

https://uk.linkedin.com/in/chrisdsimmonds/

https://google.com/+chrissimmonds


http://2net.co.uk/

https://uk.linkedin.com/in/chrisdsimmonds/

https://google.com/+chrissimmonds

Overview

• Software update 101

• Software update for embedded Linux

• Over The Air (OTA) update

• Stateless rootfs


What could possibly go wrong?

• Mirai: a recent > 600 Gbps DDoS attack• Very simple: looks for open Telnet ports

and logs on using default, well-known,name and password

• Prime target: Dahua IP CCTV cameras

Details on PenTestPartners:

https://www.pentestpartners.com/blog/optimising-mirai-a-better-iot-ddos-botnet


https://www.pentestpartners.com/blog/optimising-mirai-a-better-iot-ddos-botnet

Requirements for SW update

• Secure: must not become an attack vector to hijack the device

• Robust: must not render the device unusable

• Atomic: an update must be installed completely or not at all

• Fail-safe: should fall-back to last known working system if there areerrors


What to update?

Freq

uency

Ease of update

Bootloader

Kernel

Root file system

System applications


Update granularity• File:

• not an option: hard to achieve atomicity over a group of file updates

• Package (e.g. RPM, deb):

• apt-get update works fine for servers but not for devices

• Entire filesystem image:

• the most common option: fairy easy to implement and verify

• Atomic differential update

• Uses clever tricks to perform atomic update of groups of files

• Container

• neat idea, so long as you have containerised applications


Device update != server update

• Server

• Secure environment, no power outage, no network outage

• If update fails, human intervention is possible

• Device:

• Intermittent power and network mean update quite likely to beinterrupted

• Failed update may be difficult (and expensive) to resolve

Incremental package updates via RPM/deb are not atomic


Image update

Symmetric A/B (Android afterNougat)

Bootloader Userdata

Bootflag

OS Copy 1

OS Copy 2

Asymmetric normal/recovery(Android before Nougat)

Bootloader

Main OS

Recovery OS

Userdata

Bootflag


Examples of image updaters 1/2

• swupdate: http://sbabic.github.io/swupdate/index.html

• Symmetric and asymmetric image update client

• License: GPLv2

• Mender: https://mender.io

• Symmetric image update client

• Integrates with open source OTA update server

• License: Apache 2


http://sbabic.github.io/swupdate/index.html

https://mender.io

Examples of image updaters 2/2

• RAUC (Robust Auto-Update Controller):https://rauc.readthedocs.org/


• License: LGPLv2.1

• Android Things:https://developer.android.com/things/hardware/index.html


• Android Things is cut-down Android for IoT

• License: Apache 2


https://rauc.readthedocs.org/

https://developer.android.com/things/hardware/index.html

Atomic differential update (OSTree)

• OSTree stores data in a "git-like" object repository

• Actual filesystem created from hard links to objects in the repository

• Checkout is atomic

• Pluses

• Updates are incremental: uses less disk space than A/B image update

• Reduced transfer time: update contains only changed files

• Minuses

• Physically there is only one filesystem: cannot recover from filesystemcorruption


Projects using OSTree

• Automotive Grade Linux (agl)

• Yocto Project meta-updater layer


Atomic differential update (swupd)

• Similar to OSTree

• swupd processes updates in bundles

• Bundle contains file updates

• Updates to bundles are applied atomically


Projects using swupd

• Clear Linux

• Ostro OS

• Yocto Project meta-swupd layer


Containerised updaters

• Consists of

• Immutable base OS

• Applications in containers

• Update client running in base OS is able to update the applicationsatomically

• But, note that:

• Updates applications only

• Need another mechanism to update kernel and rootfs (e.g image A/B)


Examples of containerised updaters

• resin.io: https://resin.io

• Applications distributed in Docker containers

• Managed on device by Resin Container Engine

• Integrates with OTA update server (not open source)

• Licenses: Client: Apache2; Server: proprietary

• Snappy: https://snapcraft.io

• Containers are called snaps

• A snap contains a squashfs file system, mounted on /snap/[app]

• License: GPLv3


https://resin.io

https://snapcraft.io

OTA update

Device softwarebuild system

Firmwareimages

Sign withauthentication

key

Updateserver

DeviceUpdateagent


Complexities of OTA update

• Authentication (is this update legit?)

• Security (am I receiving what you are sending?)

• Roll-back (if update fails to boot, switch to previous version)

• Scale (roll out to large populations)

• Monitoring (keeping track of status of the population of devices)


Roll-back• Boot limit count

• Feature of bootloader (e.g U-Boot)

• Increment count in bootloader

• Reset after successful boot

• If reboot with count > 0, bootloader knows boot failed and loads alternaterootfs

• Hardware watchdog

• If hang in early boot, watchdog times out and resets CPU

• Bootloader checks reset reason

• If watchdog, loads alternate rootfs


Stateless filesystems

• Most of the updaters mentioned require a stateless rootfs

• You need to store run-time changes to files outside filesystems beingupdated

• Methods:

• symlink or bind mount files to writeable storage

• modify existing code to keep state separate

• write new code to be stateless

• See https://www.slideshare.net/chrissimmonds/readonly-rootfs-theory-and-practice


https://www.slideshare.net/chrissimmonds/readonly-rootfs-theory-and-practice

https://www.slideshare.net/chrissimmonds/readonly-rootfs-theory-and-practice

Conclusion• Image update is well-understood and road tested

• swupdate, RAUC, Mender, Android Things

• Atomic update promises finer granularity and smaller updates

• OSTree, swupd

• Containerised update is trendy, but only solves part of the problem

• resin.io, snappy

• Some projects integrate with OTA update servers

• Mender, resin.io, (swupdate, RAUC via Hawkbithttps://eclipse.org/hawkbit)

• Update is more robust with a stateless rootfs, but this is still a newconcept to many developers


https://eclipse.org/hawkbit

• Questions?

Slides on Slide Sharehttps://www.slideshare.net/chrissimmonds/software-update-for-iot-embedded-world-2017

This and other topics associated with building robust embedded systemsare covered in my training courseshttp://www.2net.co.uk/training.html


https://www.slideshare.net/chrissimmonds/software-update-for-iot-embedded-world-2017

https://www.slideshare.net/chrissimmonds/software-update-for-iot-embedded-world-2017

http://www.2net.co.uk/training.html