![Page 1: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/1.jpg)
Backporting the kernel with SmPL
Luis R. Rodriguezhttp://www.do-not-panic.com
Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt
![Page 2: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/2.jpg)
The Linux kernel backports project
Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt
● https://backports.wiki.kernel.org● git://git.kernel.org/pub/scm/linux/kernel/git/backports/backports.git● Started in 2007 - folded under the Linux Foundation backports working group● We strive to backport the Linux kernel automatically● Stable releases, and linux-next daily releases (well sort of)● Well over 800 drivers now, we stopped counting● Had to make compromises recently to scale:
○ dropped DRM drivers, might drop some others soon, only carry things folks need
○ Only support kernels >= 3.0 (match kernel.org)○ Carrot: get upstream, use backports for releases.○ Proprietary drivers cannot use this, this is for upstream drivers
● irc.freenode.net #kernel-backports● 3 core developers, 2 co-maintainers, Hauke Mehrtens now doing most of the work
○ Ethernet, Wireless, Bluetooth, NFC, ieee802154, Media, Regulator
![Page 3: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/3.jpg)
A slightly different use case
Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt
● Coccinelle engine developed to help evolve the Linux kernel, INRIA / IRILL help evolve and maintain it, you may have seen patches from Julia Lawall, Peter Senna Tschudin, and now others
● Collateral evolution concept● Linux kernel developers might use Coccinelle once / twice a
month to help create collateral evolution on a series of device drivers
● On backports we need to use it daily and against all drivers we carry, well over 800 drivers now, and use it for every single SmPL patch we have
![Page 4: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/4.jpg)
The old way
Image: CC BY-NC-SA ecololo
![Page 5: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/5.jpg)
A slightly better way
Image: CC BY-SA Teza Harinaivo Ramiandrisoa
![Page 6: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/6.jpg)
The Coccinelle SmPL way
Image: CC BY-NC Sergiu Bacioiu
@@struct net_device *dev;struct net_device_ops ops;@@-dev->netdev_ops = &ops;+netdev_attach_ops(dev, &ops);
![Page 7: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/7.jpg)
How about complex stuff? Threaded IRQ?
Image: CC-BY m4tik
--- a/drivers/net/wireless/b43/main.c+++ b/drivers/net/wireless/b43/main.c@@ -4290,9 +4299,17 @@ static int b43_wireless_core_start(struc
goto out; }
} else {+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,31)
err = request_threaded_irq(dev->dev->irq, b43_interrupt_handler,
b43_interrupt_thread_handler, IRQF_SHARED, KBUILD_MODNAME, dev);
+#else+ err = compat_request_threaded_irq(&dev->irq_compat,+ dev->dev->irq,+ b43_interrupt_handler,+ b43_interrupt_thread_handler,+ IRQF_SHARED, KBUILD_MODNAME, dev);+#endif
if (err) { b43err(dev->wl, "Cannot request IRQ-%d\n", dev->dev->irq);
![Page 8: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/8.jpg)
Backporting threaded IRQ with SmPL@ threaded_irq @identifier ret;expression irq, irq_handler, irq_thread_handler, flags, name;type T;T *private;@@
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,31)ret = request_threaded_irq(irq,
irq_handler, irq_thread_handler, flags, name, private);
+#else+ret = compat_request_threaded_irq(&private->irq_compat,+ irq,+ irq_handler,+ irq_thread_handler,+ flags,+ name,+ private);+#endif
The data structure type was inferred by the Coccinelle engine !
And we get to modify it!
@ modify_private_header depends on threaded_irq @type threaded_irq.T;@@
T {+#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,31)+ struct compat_threaded_irq irq_compat;+#endif...};
![Page 9: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/9.jpg)
It gets better● Coccinelle revealed
inconsistencies on the backport, the new struct compat_thread_irq was pegged on different data structures on different drivers
● Coccinelle revealed two collateral evolutions were used on that patch
● Amount of time to generate backport reduced, even though it automatically backported this collateral evolution to 10 new drivers ! How and why?
Image: CC BY-SA Teza Harinaivo Ramiandrisoa
![Page 10: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/10.jpg)
Does it scale? Took work, but yes!
● Run on big iron server donated by HP, SUSE, Linux Foundation: 32 cores, 236 GiB RAM
● /pub/mem tmpfs● Generation of code: all in RAM● Compilation tests with ckmake: all in RAM● At first all SmPL patches were concatenated together
and Coccinelle run only once, not parallelized. On uniprocessor runs / unparalleled environments it worked best
● Coccinelle script to parallelize○ requires running Coccinelle a few times, specify
the bucket, written in shell
![Page 11: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/11.jpg)
Parallelizing coccinelle shell script
#!/bin/bash# By Kees Cook# http://comments.gmane.org/gmane.comp.version-control.coccinelle/680set -eMAX=$(getconf _NPROCESSORS_ONLN )dir=$(mktemp -d)for i in $(seq 0 $(( MAX - 1 )) ); do
spatch -max $MAX -index $i -very_quiet "$@" > $dir/$i.out &donewaitcat $dir/*.outrm -f $dir/*.outrmdir $dir
![Page 12: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/12.jpg)
Parallelizing improvements
![Page 13: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/13.jpg)
Helping Coccinelle: needle in the haystack
Image: CC BY-NC Sergiu Bacioiu
@ module_pci @declarer name MODULE_DEVICE_TABLE;identifier pci_ids;@@
MODULE_DEVICE_TABLE(pci, pci_ids);
@ simple_dev_pm depends on module_pci @identifier ops, pci_suspend, pci_resume;declarer name SIMPLE_DEV_PM_OPS;declarer name compat_pci_suspend;declarer name compat_pci_resume;@@+compat_pci_suspend(pci_suspend);+compat_pci_resume(pci_resume);SIMPLE_DEV_PM_OPS(ops, pci_suspend, pci_resume);
@@identifier backport_driver;expression pm_ops;fresh identifier backports_pci_suspend = simple_dev_pm.pci_suspend ## "_compat";fresh identifier backports_pci_resume = simple_dev_pm.pci_resume ## "_compat";@@
struct pci_driver backport_driver = {+#if (LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,29))
.driver.pm = pm_ops,+#elif defined(CONFIG_PM_SLEEP)+ .suspend = backports_pci_suspend,+ .resume = backports_pci_resume,+#endif};
![Page 14: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/14.jpg)
Security folks might like this
Image: CC-BY m4tik
#include
struct net_device_ops {};
struct net_device {struct net_device_ops *netdev_ops;
};
struct bubble_ops {};
struct bubbles {struct bubble_ops *netdev_ops;
};
static struct net_device_ops my_netdev_ops = {};
static struct bubble_ops my_bubble_ops = {};
static struct parent {struct net_device *dev;int b;
};
static struct parent_usb {struct net_device *net;int b;
};
int main(void){
struct parent *p = malloc(sizeof(struct parent));struct parent_usb *p_usb = malloc(sizeof(struct parent));struct net_device *dev = malloc(sizeof(struct net_device));struct bubbles *bubble = malloc(sizeof(struct bubbles));
dev->netdev_ops = &my_netdev_ops;bubble->netdev_ops = &my_bubble_ops;
free(dev);free(bubble);free(p);free(p_usb);
p->dev = dev;p->dev->netdev_ops = &my_netdev_ops;p_usb->net->netdev_ops = &my_netdev_ops;
return 0;}
@@expression dev;expression ops;@@-dev->netdev_ops = ops;+netdev_attach_ops(dev, ops);
@@struct net_device *dev;struct net_device_ops ops;@@-dev->netdev_ops = &ops;+netdev_attach_ops(dev, &ops);
https://github.com/mcgrof/netdev-ops.git
1. make test1
2. git checkout -f
3. make test2
![Page 15: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/15.jpg)
Conclusions
Image: CC-BY m4tik
● We removed support for kernels older than 3.0, so only 5 SmPL patches on backports now
● Overkill for small patches● Use SmPL for the hard ports, to backport real
collateral evolutions● Must break down every patch into separate atomic
pieces● First set of changes submitted upstream to help
with making backporting easier. Example: wrapper for static inlines for data structures
● Thanks INRIA, IRILL !
![Page 16: Backporting the kernel with SmPL · A slightly different use case Slides: CC BY-SA Luis R. Rodriguez | Image: CC BY-SA Torkild Retvedt Coccinelle engine developed to help evolve the](https://reader033.vdocuments.us/reader033/viewer/2022051912/600270a7d923892fb02a0488/html5/thumbnails/16.jpg)
Further reading
Image: CC-BY m4tik
● Coccinelle post with a bit more elaborate details● Backports wiki - https://backports.wiki.kernel.org/