Sound Open Firmware

Liam Girdwood - Intel

Sound Open Firmware is an open source audio digital

signal processing frmware and driver infrastructure.

2

3

SOF Goals 1● DSP architecture and platform agnostic.

● Separate architecture and platform code like Linux.

● Abstract interfaces for architecture APIs (again like Linux).

● Host AP architecture and host AP OS agnostic.

● Drivers are generic and not coupled to any host architecture.

● Nothing in SOF that couples frmmare or lom level driver to any OS.

● Toolchain freedom

● Fully support open source tools – GCC, Octave, M4, Qemu

● Also support proprietary tools - Cadence xcc and emulator, Matlab.

● Permissive licensing of all frmmare and most SDK code.

4

SOF Goals 2● Freedom for users to defne nem audio processing pipelines using opensource

components and tools.

● Freedom for users to develop and integrate proprietary audio processing algorithms.

● All development done in public.

● Support for several CI systems to best match device and infrastructure under test.

Cadence Compiler

5

SDK Birds Eye View 1

Firmmare source

ELF

Image

Image

Signed Image

TopologyMetadata

Android HAL XML(*.xml)

/lib/firmware/intelSOF Linux Driver

rmbox / trace Tool

Crosstool-NG Tool Chain

rimage Tool

SOF Source Code

NuttX OS

SDK diagram – complete SDK

Topologyfle

(*.conf)

Audio DSP

SOF Firmware

m4ITT

alsaplg

Signing Tool

ALSA SoC Topology IPC/Mailbox

debugfs

Unsigned Image

ELFImage

OR

Cadence Emulator

SOF Firmware

debug

Qemu DSP & Host Emulator

OR

*Note: Docker contains only the open source components

m4 topology fle (*m4)

Tool

Legend

SOF Source

3rd Party Tool SDK Binary Image

Config DataOptional

Zephyr OS

Xtos OS

Matlab

6

SDK Birds Eye View 2

Firmmare source

ELF

Image

GNU Octave

Alsa kcontrol

Algorithm Performance

Criteria

SDK diagram – complete SDK

Coefficients

*Note: Docker contains only the open source components

Tool

Legend

SOF Source

3rd Party Tool SDK Binary Image

Config DataOptional

SOF Linux Driver

Audio DSP

SOF Firmware

IPC/Mailbox

Firmmare source

SOF Test BenchTest Algorithm.

7

Firmware Overview● Architecture agnostic implementation in C mith some assembler.

● Permissive BSD/MIT licensed code.

● Build time selection of features based on target (moving to use Kconfg).

● Scales domn to ~40kB footprint (text, data and bss).

● Support for user defned audio processing pipeline topologies.

● Dynamically loaded pipelines at runtime.

● Statically stitched into image for host less operation.

● Allom general compute and signal processing. Not just an audio FW.

● Image generation tools provided to

● Convert ELF to device specifc frmmare image formats.

● Sign frmmare images mith PKCS1o.5

8

Firmware Architecture

SOF Audio Componentsmixer

mux

splitter

volume

EQ

SRC

ASRC

buffer queue

tone

speaker protection

AEC/NS

Generic Micro Kernel (XTOS, Zephyr, FreeRTOS, NuttX options)

boot

IRQs

timer

msg queues

schd

threads

work queues

mutex

atomic services

exceptions

memory

semaphores

Platform Drivers

DMA

DMIC

SSP

HDA

GNA

USB

I2C

SPI

UART GPIO

Audio Tasks

Configurable pipelines

IPC

DSP or Discrete

ChipX86 or ARM Host

ControlData

Linux ASoC Audio Driver

RTOS/HV Audio Driver

SOF HW/IPC driver

2 of 2 Firmware diagram – complete diagram

9

Driver Overview● Architecture agnostic.

● Dual GPL/BSD licensed code.

● Hardmare and physical IO abstraction, so can mork equally mell mith local MMIO and remote SPI based DSPs.

● Supports virtualisation through VirtIO backend and frontends (using ACRN).

● Driver has no hard coded assumptions about DSP pipelines or components.

● Topologies loaded from FS.

● Component confgurations can be runtime updated via ALSA kcontrols.

Driver Architecture

DSP or Discrete

Chip

IPCAudio Data

ASoC Machine Driver

codec integration

board integration

HW config

ASoC PCM Driver

topology PCMs Kcontrols

Generic IPC Driver

mixer stream PM

DSP Platform Driver

DAPM

pipeline

doorbell mailbox IRQ

IO PMcode loader

2 of 2 Driver diagram – complete diagramo0

Physical IO layer can currently be either PCI, SPI, MMIO or VirtIO.

oo

Pipeline Architecture● A pipeline is a set of audio processing components and bufers

that share common scheduling.

● Pipelines can currently be defned using M4 in a fem lines of code. Plans for GUI.

TopologyBinary

Topologyfle

(*.conf)

alsaplg

m4 topology fle (*m4)

M4

EQ coefficients loaded via ALSA binary kcontrols

Standard ALSA volume kcontrols

o2

Continuous Integration - Travis

Firmmare source

ELF

Image

Image

Signed Image

Crosstool-NG Tool Chain

rimage Tool

SOF Source Code

SDK diagram – complete SDK

Signing Tool

Unsigned Image

ELFImage

Qemu DSP Emulator

*Note: Docker contains only the open source components

Tool

Legend

SOF Source

3rd Party Tool SDK Binary Image

Config DataOptional

SOF Slim Driver

Topologyfle

(*.conf)

alsaplg

m4 topology fle (*m4)

Travis CIGithub PR Github

Travis CI &

o3

Continuous Integration - 0day

Firmmare source

ELF

Image

Image

Signed Image

Crosstool-NG Tool Chain

rimage Tool

SOF Source Code

SDK diagram – complete SDK

Signing Tool

Unsigned Image

ELFImage

*Note: Docker contains only the open source components

Tool

Legend

SOF Source

3rd Party Tool SDK Binary Image

Config DataOptional

SOF Driver

Topologyfle

(*.conf)

alsaplg

m4 topology fle (*m4)

Travis CIGithub PR Github

0day

alsabat

o4

SOF Platforms

Vendor Platform DSP Status

Generic Host PC Host PC Testbench

Qemu Host PC Qemu targets Currently only has support for xtensa based devices.

Intel Baytrail, Cherrytrail, Brasswell

1 * Xtensa HiFi2EP Upstream, support for upto 6 I2S DAIs.

Intel Haswell, Broadwell

1 * Xtensa HiFi2EP Upstream, need I2S integration for some codecs.

Intel Apollolake, Geminilake

2 * Xtensa HiFi3 Upstream, support for I2S, DMIC, HDMI and HDA.

Intel Cannonlake, Whiskylake

4 * Xtensa Hifi4 Upstream, support for I2S, DMIC, HDMI and HDA.

Intel Suecreek HAT on Rasberry PI

2 * Xtensa HiFi4 Upstream core, boot over SPI WiP

Xiaomi ARM M4 CEVA Teaklite DSP Upstreaming WiP.

o5

SOF Processing Components (opensource)

Name Generic C available

SIMD support Status

Volume Yes Xtensa HiFiEP, HiFi3 Upstream, can process 1 – 8 channels, 16, 24 and 32bit formats.

SRC Yes Xtensa HiFiEP, HiFi3 Upstream. 24 and 32bit formats. FIR polyphase optimized M/N and M/N x O/P fractional rate conversions.

Configuration tools upstream.

FIR Yes Xtensa HiFiEP, HiFi3 16/24/32 bit formats, up to 192 taps.

Configuration tools upstream.

IIR Yes Planned 16/24/32 bit formats, up to 11 biquads or 22th order.

Configuration tools upstream.

o6

2015 

Driver open sourcedPlatforms: Apollo Lake, Haswell, Broadwell

Features: BSD/GPL driver, topologies

Driver upstreamPlatforms: Cannon Lake, Ice Lake,

Sue Creek, TeakLiteFeatures: Docker build, SOF on host

Became Linux Foundation project

Firmware open sourcedPlatforms: Cherry Trail and Braswell

Features: Qemu host

Platforms: Bay TrailFeatures: Qemu Xtensa

SOF 1.0 public release

Inception 

2016 

2017 

2018 

History

Year ending

o7

SOF Future Plans

Feature Status

GDB stub and tunnel.

Initial firmware and kernel code in developer repository. Upstream in 2019.

RPMSG Code complete, now upstreaming to SOF and Linux

Loadable firmware modules

Initial code in github, still WiP. Upstream in 2019.

CMSIS RTOSAbstraction layer

Work in progress.

o8

Supporting Organizations

Xiaomi

o9

Firmware Preference ?● Linux already has a preference for opensource drivers over closed source

drivers.

● Should this preference also be extended to opensource frmmare in the future ?

● Selection criteria mould also have to consider :-

● Features.

● Stability.

● Portabilty.

● Some opensource frmmare could still contain binary blobs.

● Regulatory reasons (e.g. comply mith FCC regulations).

● 3rd party IP (e.g some audio processing algorithms).

Q & A

www.sofproject.com https://github.com/thesofproject

Visit the booth for demo.

20

Qemu Heterogeneous Virtualization

2o

22

Baytrail DSP Architecture

SSP0

to/fromIOSF2OCP

bridge

JTAGPIF2AXI

JTAG

Interrupts(from shim)

JTAGTrace Logic

Interrupt

HiFi EPCore

EXTI/F

PIF

HiFi EP Core

LPEShim

Control & Config signals

AUDIO

SMX

Instruction RAM

Data RAM

LPEDMA_01

32 bit OCP Slave

LPEDMA_00

64 bit AXI Master and Slave

OCPMasterReads

OCP Master Writes

32 bit OCP Slave

OCPMasterReads

OCP Master Writes

Data Cache

Instruction Cache

4kB RFMailbox

OCP (Upstream)

32-bit APB Slave

M/NCLK

SSP132-bit APB Slave

M/NCLK

SSP232-bit APB Slave

M/NCLKSource:

Tangier

VLV

Sonics

● Xtensa HiFi 2EP core.

● 96kB Instruction RAM

● 168kB Data RAM

● 2 * DMACs

● 3 * I2S ports

● PCI device from host OS

● Firmware and host share

● SHIM registers

● Mailbox memory.

Must be shared with IA and Xtensa VMs

23

Heterogeneous Virtualisation

● Firmware must be debugged alongside driver.

● Qemu used to virtualise drivers and firmware together.

● Host side almost real time.

● DSP side emulated.Codec driver

Qemu IO Bridge

IO, DMA IRQs

FW SSP driver

X86 Qemu/KVM

Guest OS *

Audio DSP driver FW DMA driver

Xtensa Qemu

SOF Firmmare

FW Audio mixerAudio Userspace

VM #0 VM #1

x86 GDBOn host

xtensa GDB

On host

24

Heterogeneous IO bridging

PCI BAR 0 /dev/shm

X86 Qemu/KVM

Guest OS *

Xtensa Qemu

SOF Firmmare

VM #0 VM #1

x86 GDBOn host

xtensa GDB

On host

PCI BAR o /dev/shm

IRQ /dev/mqueue

Zero Copy Virtual Audio

25

26

Zero Copy Audio Virtualisation

SOS bufer SG PHY pages

Audio Bufer (on SOS)

Audio driver

Audio DMACs

Audio HW

Guest N

IPC virtio Qs

Shared audio bufers

Service OS

Audio Driver

SOS buffer PHY pages shared with guests.

SOS allocates SG PHY pages for guest audio buffers. Audio DMA copies directly from/to these buffers.