improving rollback in linux via dsl approach & distributing

Energia Open Source

Improving Rollback in Linuxvia DSL approach & distributing

John Thomson: [email protected]

Paulo Trezentos: [email protected]

http://twitter.com/PauloTrezentosR&D Director

Sunday, 6th February 2011

Presented by:

mailto:[email protected]

mailto:[email protected]

Energia Open Source

06.02.2011 Improving Rollback in Linux via DSL approach & distributing 2

Overview An overview of what Caixa Mágica does.

– MANCOOSI project: European 7th Framework Programme Roll-back

– Definition– Types of roll-back– As part of a bigger system

DSL– Definition– Example

Approach Implementation Distributing data (kad/p2p) Conclusion

Energia Open Source


A brief summary of Caixa Mágica Software and an overview of what we do: Caixa Mágica is a FOSS software and solutions

provider, based in Lisbon, Portugal Linux- Caixa Mágica 15 is the main Linux

Distribution available in Portugal Caixa Mágica work with national companies,

Government and with European partners to promote Open Source in all aspects of work

Over 800,000 Linux CM systems installed (single & dual boot)

Interesting development → CM16 moving to Debian base

Energia Open Source


European based research projects Caixa Mágica works in many multi-national European based

research projects. After EDOS (EU FP6 STREP) based project it is now actively working on MANCOOSI-FP7 Project

Also working on ULOOP and Timbus projects Work with a multitude of top-tier Universities and research

institutions

Energia Open Source


About MANCOOSI MANCOOSI - Managing the complexity of open-

source software. www.mancoosi.orgSolving package management issues that havebeen identified through EDOS project

Other talks Talks yesterday by Ralf and Jaap about solvers + tools André Guerreiro presents Gumby in room H.1302 at 15:30

– Visualise package meta-data Paulo Trezentos presents APT-PBO in room H.1302 at 16:00

– A better class of solver Aim is to provide better tools for package management

http://www.mancoosi.org/

Energia Open Source


Roll-back as simple as traversing time?

Energia Open Source


What does roll-back mean, really? In terms of package configuration,

“roll-back is the process of inverting the changes to the system made by package upgrades to get back to a particular system state.”

Many other mechanisms out there that work on using file system snapshots/saving the state, (next slide)

The mechanism is one part of Transactionally Protected Package Management as spoken about by Jeff Johnson last year at FOSDEM 2010.

Installation TimelinePkgFoo v 1.00Time: 10.00pm

PkgFoo v 2.00Time: 10.20pm

Roll-back, possible?

S1 S2

Energia Open Source


Different types of roll-back Conary, a 2nd Generation Package Manager, aims to

improve current meta-installers. Used in Linux distributions such as Foresight Linux.

Augeas, is a configuration management tool thatmakes manipulating config files much easier

ZFS, used by Nexenta is an example of a file-system, snapshot mechanism to snapshot several system states.

NixOS, a revolutionary system that re-thinks how files and resources are used to make them purely functional

Other mechanisms e.g. etckeeper being developed by Fedora also try and capture configuration files into a VCS. BTRFS – another snapshot based mechanism.

Energia Open Source


Drawbacks of the various approaches File-system Snapshot based mechanisms

– Capture state of the whole system → down to individual files– Agnostic of packages– Granularity is not well matched for packages

Package Based– Same as installing an older version– Maintain configurations and settings

System Call based– Syscall trapping - records changes to the file-system– Capturing files alone is not always enough

Aim is to combine the complimentary parts of these systems to cover a wide range of situations and events

Energia Open Source


Difficulties of Roll-back Package maintainers and developers tend to think in the

forward direction of upgrading packages Working in the reverse direction is a relatively unheard of

concept. Think of a toy slinky

Downgrade is counter-intuitive Already have been done if it was necessary? Rolling-back changes is 'only' needed when a package fails

to work on the system There may be cases where roll-back is impossible using

the techniques that we have investigated, or possibly at all

Energia Open Source


Roll-back is one part of a bigger picture Ability to undo package upgrade/install is just one benefit of

having a system that can capture the package configuration state and maintain a deterministic, state transition model of the system.

By examining the current maintainer scripts and templates provided by deb/rpm-helper, we defined a language that assists with problems not addressed now by meta-installers.

Transactionaly Protected Package Management (TPPM) is what we are aiming to move towards.

roll-back

TPPM

roll-back

Energia Open Source


Advantages and benefits of Roll-back/TPPM By building a framework in which R/B + TPPM is possible we

can capture the functional elements of maintainer scriptsError Detection

Simulate/test the results ofan upgrade on a model of system

Detect errors a-priori orpotential 'slow' errors after upgrades

Detect/(correct) sequencing orders in package installationsRoll-back

Framework → drive roll-back. Now provide more information to a roll-back utility than available to current meta-installers

Identify which packages can/cannot perform roll-back Allow individual packages to change versions and show the

dependencies for those changes

Energia Open Source


Domain Specific Language (DSL) -package upgrades

Our DSL is a language used to abstract from the system and model to solve particular, identified problems

In our case, the DSL is focused on analysing package maintainer scripts

Designed not to be a Turing Complete Language like BASH but rather something which can resolve problems

It is an evolutionary language designed to capture the details of the vast majority of common maintainer scripts

Increase the coverage of DSL by examining new scripts

Many thanks to University of L'Aquila for their creation of the language and for their support in using it

Energia Open Source


System Integration For roll-back we will have a log of

the DSL commands executed in-sequence

Perform roll-back → run inverse statements associated with those commands in 'reverse' order

Have the simulator to pre-check if the package configuration can be rolled-back; whether it will leave it in an erroneus state

Performing LIFO style roll-back we run post commands before we run the pre statements

model_simulator ( )model_simulator ( )

DSL_rollback_pre ( )DSL_rollback_pre ( )

run_transaction ()run_transaction ()

Apt

DSL_rollback_post ( )DSL_rollback_post ( )

Energia Open Source


Domain Specific Language (DSL) Example The DSL has been created to assist with some of the

problems discovered and analysed by prior researchUsing cups.spec %post example from CUPS-1.4.2%post

dslstart postinst_init(cups)/sbin/chkconfig add cups

/sbin/chkconfig cups on

dslstop postinst_init(cups)# Restart cupsd if we are upgrading...

dslstart post_init_restart(cups)if test $1 gt 1; then

/sbin/service cups stop

/sbin/service cups start

fi

dslstop post_init_restart(cups)

Matched DSL Pair

Matched DSL Pair

Energia Open Source


Example continued, Log

dslID TID parentID DSL_CMD bhINVERSE

1 1 1 start postinst_init(cups) TRUE

2 1 1 stop postinst_init(cups) TRUE

3 1 1 start post_init_restart(cups) TRUE

4 1 1 stop post_init_restart(cups) TRUE

rbHist

id parent op pkgName pkgVer1 pkgVer2 dateTime

1 1 inst cups 0 1.4.2 2010-01-30pkgHist

Energia Open Source


Transactions- what happens if a maintainer script fails? If a maintainer script fails in the middle of one of the

operations will have a log like this

Transaction has quite obviously failed. No matching end for a DSL command reached. Odd number of elements etc.

Perform a roll-back for all matching sub-transaction ID elements, but in the reverse order with certain constraints.

If a set of script elements cannot perform roll-back in the middle of operating, then don't create a dsl tag.

dslID TID parentID DSL_CMD bhINVERSE

1 1 1 start postinst_init(cups) TRUE

2 1 1 stop postinst_init(cups) TRUE

3 1 1 start post_init_restart(cups) TRUE

Energia Open Source


Performing the roll-back Identify the sub-transaction/package upgrade to roll-back. Check that all commands in the database have an inverse for

the particular transaction. Run them in reverse-operation order. dslstart postinst_init(cups) [1] dslstop postinst_init(cups) [2] dslstart post_init_restart(cups) [3] //Restart not needed postrm

becomes postrm_init_restart(cups) [3] //Remove nonneeded element prerm_init_stop(cups) [4] //From lookup table. prerm_init(cups) [1+2]

In other cases the order might be [3,2,1]. Identified by tags.

Energia Open Source


WP2

Architecture Modified Package as input → DSL extracted → Log storage

and simulator → Mechanism for executing roll-backs → Maintenance of scripts etc.

.spec file%post/sbin/chkconfig --add cups/sbin/chkconfig cups on...

Inject DSLdslstart postinst_init(cups)dslend postinst_init(cups) Logs

SQLite DB

Simulator

Roll-back

Modified System State

User I/P

Apt-rpm

Energia Open Source


Overall Approach

Analyse maintainer scripts in standard systems

Identify common themes in maintainer scripts

Create first version of the DSL

Modify maintainer scripts to contain DSL

Log DSL into SQLite DB

Logged Data + Stored Info → Perform Roll-back

Create roll-back mechanism

Energia Open Source


[Maintainer Scripts]In terms of DSL

A state/time perspective of roll-back

T0

T0+n

apt-getinstall cups

apt-getrollbackTID-028

apt-getrollbackcups 1.2

apt-getrollback

2010-02-06

apt-rpm DSLLog

cups pkg DSL stmts

[Maintainer Scripts]-1

roll-back

Energia Open Source


Distributing Rollback-Information Rollback data can be distributed

– Transceive data onto Kademlia network.– Distribute rpms/rollback data/packages state to network– Private data submission to network? Opt-in mechanism

LogsSQLite DB

Modified System State

Configdata

cups pkg

Initial System StateNormal upgrades

Kad/P2P Network

Energia Open Source


Todo list: Completed:

– Single package rollback with DSL– Supporting framework e.g. SQLite3 and script injectors.

Now in the process of:– Integrating Simulator/Failure Detector into package upgrade– Generating tool suite for automatic detection/generation of DSL

commands for a given set of packages and their scripts– Creating web interface for changing the auto-created DSL pkgs– Completing rollback mechanism for multiple packages txns.– Testing– Documentation– Release + revisions– Integration into build systems

Energia Open Source


Conclusion The need for Roll-back of packages is growing Not a trivial problem Need to consider the problems of

complex upgrade paths Using the DSL we aim to remove

a lot of the failure cases inpackage upgrades.

Energia Open Source


Questions? Thank you for listening. I hope that it was interesting and

that you have some questions to ask.

improving rollback in linux via dsl approach & distributing

Technology