Implementing a World of GUIs Using the ARM CortexM3.

Dave Wilson

Luminary Micro, Inc.

GUIs for Embedded Systems

We are not talking about running Linux or Windows on a low-cost consumer device!

We are talking about using CPU-rendered, on-screen graphical controls in conjunction with touch or button input to control your application.

GUI Benefits

Customizability Product features can be defined by the software. Offer new features without needing hardware

changes.

Visually appealing and intuitive Show only the information the user needs at that

time. Less display clutter.

The Good Old Days

In yesteryears, fewer UI options were available to embedded device developers.

• Graphics-capable displays were expensive.

• Microcontrollers were too slow to run an application and handle graphics processing.

• Graphics took too much storage for use on a small device.

• Writing graphical interface code was time consuming and difficult.

LCD Displays Are Too Expensive

$0

$5

$10

$15

$20

$25

$30

$35

$40

2004 2005 2006 2007 2008

TFT LCD

CSTN LCD

MSTN LCD

AM OLED

PM OLED

Source: iSuppli/Stanford Resources, http://www.usdc.org/resources/DisplayTrends_summer2004/biggrowthahead.htm

• High demand for mobile phones and the availability of older-generation LCD manufacturing capacity has driven small panel LCD display prices sharply downwards.

• 128x64, 4bpp OLED~$4.00

• 128x128, color CSTN~$3.60

• QVGA color touchscreen ~$14

Graphics Need Too Many MIPs

320x240x16bpp using an 8bit parallel interface, 2 cycle GPIO operations, 15Hz refresh.

This requires about 14MIPS. 28% of the CortexM3 CPU bandwidth at 50MHz. 72% of the CPU is left for application use. BUT - we are not in the video business!

Updates are seldom at 15Hz and seldom full screen. Interface is often 16bits wide with single cycle GPIOs 15Hz overhead drops to about 9.5% in this case.

Consider a worst case:

To offer a different example…

Full software rendering.

Comparable resolution to today’s small displays.

“DOOM” ran on a 25MHz 80486.

Less than half the power of a CortexM3.

“DOOM” from id Software

State of the art 3D rendering on a PC in 1994.

Bitmapped images are huge!

A 320x240 16bpp image takes 150KB! But… The display panels we are looking at include

integrated frame buffers. Draw directly into the display panel buffer.

But what about application images? Use the CPU power to your advantage:

Store images in a compressed format. Saves about 80% of the space for typical buttons and

text.

Use palletized image formats and convert on-the-fly. Describe the image using 16 colors rather than 65536.

Graphics library software is big

True if you want Bezier curves, non-rectangular clipping regions, support for 15 image formats, pie charts, metafile generation, complex pens, customizable line endcaps, double byte fonts…..

You don’t need the Windows GDI or OpenGL to implement a very usable, compelling graphical user interface.

Luminary Stellaris Graphics Library compiles to about 10KB of code and needs about 400bytes of RAM workspace.

ASCII fonts are between 1.5KB and 6KB depending upon glyph size.

GUI Software Aims

Develop a minimum subset of graphics primitives that you will need for your interface. Keeps code size to a minimum. Keeps work to a minimum!

Isolate the control definitions and appearance from the code that manages them. Allows rapid prototyping. Get the application working with basic controls then spend time

making them pretty.

Be consistent in how controls are defined and used. Learn one set of concepts and use them in different areas.

A widget-based architecture makes a lot of sense.

Widget (noun) “A coil-like device placed in the bottom of cans and bottles of

beer to aid in the generation of froth.” “A component of a graphical user interface with which a user

interacts.”

A widget is a control – something on the screen that you read, press, slide, click, toggle or otherwise fiddle with to set or get application information.

Widgets are instantiated as instances of a widget class (don’t worry – C++ is not needed!).

Widgets encapsulate the drawing of the control and its interaction with the user. Interaction with the application is via messages sent to an application callback function.

Widget Types Container

A convenient way to group other widgets together and, optionally, place them on a background rectangle.

Canvas A “static” control used to display text, image or color.

Canvas widgets ignore user input. Pushbutton

A circular or rectangular area that the user can press to initiate some action. May be outlined color or contain an image.

Radio Button A clickable control which is grouped with its siblings to

allow “one-of-many” selections to be made. Checkbox

A Boolean indicator that can be clicked to toggle its state.

Slider A draggable horizontal or vertical control that reports its

position. ListBox

A scrollable collection of text strings optionally allowing the user to select and highlight one.

Input Driver(e.g. buttons ortouchscreen).

Display Driver

Layered Architecture

A layered UI management architecture allows maximum flexibility and reuse potential.

Graphics Functions

Widget Manager

Widget Classes

Input H/W

Application

Widget ClassesWidget ClassesWidget Classes

Display Driver(s)

Display H/W

Display Driver

A minimal interface to abstract the display hardware from the graphics rendering software.

• Initialize the display hardware.• Report display dimensions.• Translate colors from 24bpp RGB to a display-specific value.• Plot a pixel.• Draw horizontal and vertical lines.• Copy a line of pixels from CPU memory to the display.• Fill a rectangle with a given color.• Flush any outstanding operations to the screen.

Display Drivers Without Hardware

Display drivers may also be written for offscreen rendering. Allows double-buffering for smooth animation. Allows composition of images before display.

Luminary Stellaris Graphics Library includes offscreen display drivers for 1bpp, 4bpp and 8bpp formats, for example.

These drivers offer the same interface as their hardware-supporting siblings (apart from being supplied a frame buffer image on their Init() call).

A Graphics Library for GUIs

A very visually appealing GUI can be generated with a small set of graphics primitives:

Straight lines Filled and outlined rectangles Filled and outlined circles 1, 4 & 8bpp images Bitmap font rendering Clipping rectangle

The Luminary Stellaris Graphics Library offers this support and is free for use on Stellaris microcontrollers.

Graphics Context All graphics functions take a context pointer as first parameter. Graphics operations can take a large number of parameters:

Background and Foreground Colors Fonts Clipping rectangle Display driver to use

These parameters are used frequently but seldom changed. Rather than passing them as parameters on each call, they are

stored in a “graphics context” structure. Context parameters are changed via access functions:

GrContextBackgroundSet() GrContextClipRegionSet()

Context parameters remain active until the application changes them.

“Hello World”// Initialize the display driver and turn on the backlight.Formike240x320x16_ILI9320Init();Formike240x320x16_ILI9320BacklightOn(); // Initialize the graphics context.GrContextInit(&sContext, &g_sFormike240x320x16_ILI9320);

// Fill the screen with dark blue then outline in whitesRect.sXMin = 0; sRect.sYMin = 0;sRect.sXMax = GrContextDpyWidthGet(&sContext) - 1;sRect.sYMax = GrContextDpyHeightGet(&sContext) - 1;GrContextForegroundSet(&sContext, ClrDarkBlue);GrRectFill(&sContext, &sRect);GrContextForegroundSet(&sContext, ClrWhite);GrRectDraw(&sContext, &sRect);

// Say hello using the Computer Modern 40 point font.GrContextFontSet(&sContext, &g_sFontCm40);GrStringDrawCentered(&sContext, "Hello World!", -1, GrContextDpyWidthGet(&sContext) / 2, GrContextDpyHeightGet(&sContext) / 2, 0);

I can draw stuff. Now what?

Graphics primitives offer the tools needed to draw the user interface. Now we need to generate usable controls.

The Widget Manager and Widget Classes turn graphics primitives and user input into working controls.

Each widget is defined in terms of a structure defining properties of the control and a message callback function.

The structure contains a standard widget header used by the Widget Manager and a class-specific section.

Defining Widgets

Each widget is defined using a C structure:

typedef struct __Widget{

long lSize;struct __Widget *pParent;struct __Widget *pNext;struct __Widget *pChild;const tDisplay *pDisplay;tRectangle sPosition;long (*pfnMsgProc)( struct __Widget *pWidget,

unsigned long ulMessage,unsigned long ulParam1,unsigned long ulParam2);

} tWidget;

typedef struct{

tWidget sBase; // Header structure common to all widgetsunsigned long ulStyle; // TrivialButton-specific style informationvoid (*pfnOnClick)(tWidget *pWidget); // Click handler for this TrivialButton

} tTrivialButton;

Widget Tree Structure

GGRANDCHILD3GGRANDCHILD2GGRANDCHILD1

WIDGET_ROOT

CHILD1

GRANDCHILD2GRANDCHILD1 NULL

NULL

NULL

NULL

pParent/pChild

pParent/pChild

pParent/pChild pParent/pChild

pParent

pNext

pNext pNext

pNext

pNext

pNext

NULL NULL NULL

pParent

An Example

• 2 Radio buttons

• “Choice 1” & “Choice 2”

• 1 Push button

• “Done”

• 3 Containers

• “What will it be?” border

• Blue border behind “Done”

• Overall black background

How do we implement this simple interface as a widget tree?

Determining Parent/Child Relationships

Using the right parent/child relationship is vital to ensure correct repainting and handling of user input.

The widget tree is painted starting at WIDGET_ROOT and working downwards – parents are painted before children.

Parents can be thought of as being below their children on the screen.

User input is handled in the opposite direction – clicks are first passed to children and, if not processed, passed up to the parents.

Widgets “on top” get a chance to process input before those underneath.

Think in terms of planes - the further back the widget, the closer it is to WIDGET_ROOT in the widget tree.

Drawing Planes

Great-Grandchildren

Grandchildren Child

Draw Later Draw Earlier

Example Widget Tree

“Done”Pushbutton

“Choice 2”Radio Button

“Choice 1”Radio Button

WIDGET_ROOT

Background Container

“Done” ContainerRadio Container NULL

NULL

NULL

NULL

NULL

NULL NULL NULL

Handling Repaint

“Done”Pushbutton

“Choice 2”Radio Button

“Choice 1”Radio Button

WIDGET_ROOT

Background Container

“Done” ContainerRadio Container NULL

NULL

NULL

NULL

NULL

NULL NULL NULL

WidgetPaint(WIDGET_ROOT);

5

6

4

3

2

1

7

Handling User Input Pointer drivers generate up, down and (x,y) position messages in

response to touchscreen or pointing device input. A touchscreen or pointer driver is attached to the Widget Manager by

setting WidgetPointerMessage as the driver callback function. When a pointer down message (WIDGET_MSG_PTR_DOWN) is received,

the Widget Manager passes it to all widgets, children first. Each widget checks to see if the pointer is within its area and, if so,

claims the message by returning “true” from the message procedure. All further pointer move messages (WIDGET_MSG_PTR_MOVE) are sent

to this widget until pointer up (WIDGET_MSG_PTR_UP) is received or the widget is removed.

Note that the order that pointer messages are passed to widgets is different from the order in which the widgets are painted. The widgets “on top” (children) need to see the input first before we offer it to those which lie behind (parents).

User Input Processing Order

“Done”Pushbutton

“Choice 2”Radio Button

“Choice 1”Radio Button

WIDGET_ROOT

Background Container

“Done” ContainerRadio Container NULL

NULL

NULL

NULL

NULL

NULL NULL NULL

2

5

1

3

6

7

4

Search skips to the bottom of the tree

What does the application have to do?

Define the widget tree. Provide handler functions for each widget.

void OnDoneBtnPress(tWidget *pWidget);

Initialize the display driver. Formike240x320x16_ILI9320Init();

Initialize the touch screen/input driver. TouchScreenInit();

Pipe input events to the widget manager. TouchScreenCallbackSet(WidgetPointerMessage);

Process messages. while(1) { WidgetMessageQueueProcess(); }

Intelligent Display Module

240x320 16bpp touchscreen display with LED backlight

10/100 Ethernet Relay microSD slot 4 ADC inputs Speaker

IDM Example

Security keypad application

Buttons are arranged randomly when the keypad is activated.

Entry of the correct code closes a relay connected to the door lock mechanism.

A web server allows the code to be changed.

Oscilloscope

• Uses a text- and menu-based widget system built on top of the Luminary Stellaris Graphics Library.

• Runs on an EK-LM3S3748 evaluation board with 128x128 CSTN display.

• User input via 4 directional rocker switch with “press to select.”

Conclusion

Using the CortexM3, today’s low cost color displays and software such as Luminary Micro’s Stellaris Graphics Library, visually appealing, high function graphical user interfaces are both affordable, usable and straightforward to develop.

Getting hold of the code

Software

Documentation

Look under C:\DriverLib\boards\rdk-idm for main examples.

www.luminarymicro.com/products/software_updates.html

Questions?