Lab 3

Dual Core System with Interrupts, Mutex, Mailbox and Message Queue

Introduction

In Lab1, we implemented a uniprocessor system in order to have an overview of how to use the Xilinx

EDK tool. Lab1 can be used as base system for the controller of your project (e.g. soccer project group).

In this lab, we will develop a dual core system with two microblaze processors. This lab will be

particularly useful for the development of the server. Furthermore, all our future labs will be based on this

dual core system. Therefore, DO NOT DELETE THIS PROJECT and keep a back up copy of this project

so as to avoid going through the hassle of building and synthesizing the system all over from scratch.

Then we will address on how to enable, configure and use the peripheral interrupts of Xilkernel by

including the necessary APIs and sample code.

In this Lab, we will also introduce the concept of mutex and work on exercises to implement mutex in our

system. As you would have learned, MUTEX (acronym for MUTUAL EXCLUSION) is used to ensure

that a shared resource is not simultaneously used by two different functions. There are two ways of

implementing mutex in the system: Software and hardware mutex.

Then, we will deal with a new scenario. Consider a situation where there are multiple threads running on

different cores of the multiprocessor system. A need may arise where some of these threads require data

that is generated by execution of other threads in the system. Now these threads can either be running on

the same processor core or on different processor cores. Techniques are hence needed so that data can be

sent from one thread to another, whether both the threads are running on the same processor or on

different processors. There are two techniques to achieve this purpose:

Message queues: This is used for passing data between threads running on the same processor core.

Message queues are implemented in software and is available as a feature of the xilkernel OS.

Mailboxes: This is used for inter process communications running on different processor cores. In other

words, this performs the same function as that of message queues but for threads running on different

cores. This is implemented in hardware IP and is available from the Xilinx IP library.

Requirements

Working installation of Xilinx ISE Design Suite 13.1

Objectives

In this lab you will learn:

• How to develop a dual processor system based on Microblaze soft processor systems with peripherals

such as UART, MUTEX, MAILBOX , a PLB to PLB bridge and a TFT controller.

• Work on an embedded OS called XILKERNEL, and thread management techniques using Xilkernel

API.

• Generating a linker script to use DDR SDRAM for code and data.

• Enabling and configuring peripheral interrupts for Xilkernel.

• How to implement Software mutex for threads running on the same core.

• How to use the hardware mutex IP from the Xilinx IP cores library in order to implement mutex

between threads running on two different cores.

• How to implement message queues for threads running on the same core.

• How to use the Mailbox hardware IP from the Xilinx IP cores library in order to implement inter

processor communication.

LAB 3A

BASE SYSTEM BUILDER for DUAL CORE SYSTEM

Procedure

� Launch the Xilinx EDK by navigating to Xilinx ISE Design Suite 13.1 -> EDK -> Xilinx Platform

Studio. After the software is launched, you will see the following screen. Make sure the Base System

Builder is selected. Click OK.

� Browse to an appropriate location where you would like to save your project file and give a name to

your project. The project will be saved as an “.xmp” file. Next time, when you want to open your

project, you can select "Open Recent Project" and point to your project file (.xmp file).

� After that select the PLB system as the bus standard which is shown below right.

� When you click OK, the Base System Builder will appear as shown below.

� Since we are creating a new design, there we will select the first option and click NEXT to continue.

� Select the board Spartan 3E 1600E Microblaze Dev Board (which corresponds to the board we are

using) and Board Revision as RevA. Click NEXT to continue.

� You will be provided with an option to create a single processor system or dual processor system as

shown. For this lab, we are developing a dual processor system. Select "Dual-Processor System".

Click NEXT to continue.

� In the next screen, you will configure the system clock frequency and local memory of each

processor. Select the system clock frequency as 50 MHz and local memory as 32 KB for both

processors. Click NEXT to continue. NOTE: Remember, that the local memory is implemented using

Block RAMs resources of the FPGA. Each Block RAM has a size of 2KB. The Spartan XS3S1600E

FPGA has 36 Block RAMs in total.

QUESTION: How many Block RAMs have we allocated for local memory of both the processors?

� In this screen, you are given the option to add or remove external peripherals to/ from the system for

each microblaze. The mailbox and mutex are the shared peripherals. Configure the peripheral system

as follows:

� Processor 1 (Microblaze_0) Peripherals

DIP_Switches_4Bit, LEDs_6Bit, RS232_DTE (Baud Rate 9600, Data Bits 8, Parity :

None, Use Interrupt checked), dlmb_cntlr, ilmb_cntlr, xps_timer_0 (Interrupts Enabled) **note

that timer is not by default included, you need to add from the left Available Peripherals)

� Shared Peripherals

DDR_SDRAM, xps_mailbox_0, xps_mutex_0

� Processor 2 (Microblaze_1) Peripherals

Button_3Bit, RS232_DCE (same config as DCE), dlmb_cntlr_1, ilmb_cntlr_1,

xps_timer_1 (Interrupts Enabled)

� Also you can choose to enable different peripheral interrupts (e.g. RS232_DTE, GPIOs like push

button) as shown in the following figure. Make sure that the interrupts are enabled

� Once you are finished with configuring all peripherals that you need with proper interrupt enable,

click NEXT to continue.

� We do not require caches for this time being. Therefore, we will not enable caches in this system.

Keep clicking NEXT till we reach the summary screen.

** Note that, the soccer project group may require cache memory (e.g. instruction cache for speed up

their system if the code size fits in the cache memory)

� Finally, we are at the summary screen. We can see the names of various peripherals and the memory

addresses to which each peripheral is mapped.

� Click FINISH to generate your system as shown below.

*OPTIONAL: It is recommended to follow some conventions in multiple microblaze system for the sake

of clarity. Now, we will rename all the peripheral buses to match the suffix (_0 or _1) of their respective

microblaze processors. For example, we will rename dlmb to dlmb_0. Similarly, rename ilmb and

mb_plb. Likewise, we rename the peripherals lmb_bram, dlmb_cntlr, ilmb_cntlr and xps_intc_2.

� Now, we need to add peripherals from the IP Catalog. Add the following peripherals.

1. Bus and Bridge\Processor Local Bus (PLB) 4.6. Rename it to shared_plb

2. Bus and Bridge\PLBV46 to PLBV46 Bridge. Rename it to plb_bridge_0

3. Bus and Bridge\PLBV46 to PLBV46 Bridge. Rename it to plb_bridge_1

� Now, connect the peripherals to the correct buses as follows:

1. SPLB of mdm_0, RS232_DCE, RS232_DTE, Button_3Bit, DIP_Switches_4Bit, LEDs_6Bit to

shared_plb

2. MPLB of plb_bridge_0 and plb_bridge_1 to shared_plb

3. SPLB of plb_bridge_0 to mb_plb_0

4. SPLB of plb_bridge_1 to mb_plb_1

� Now to add interrupts to the interrupt controller corresponding to a processor (for example,

xps_intc_0 for microblaze_0), go to system assembly view > ports, expand xps_intc_0, and do a

single click on the value right to Intr as shown in the figure. Select all the interrupts we need, and set

priorities.

**Please note that priorities are from low(top) to high(bottom). Normally system timer should have

the highest priority.

� Repeat the same thing for xps_intc_1, for all interrupts microblaze_1 needs to handle.

� GPIO (which controls LEDs, pushbuttons and DIP switches) interrupts can also be enabled using a

configure IP in system assembly view. Select Buttons_3Bit from the ‘Ports’ tab of ‘system assembly

view’, right click on it and select ‘Configure IP’ and enable ‘GPIO support interrupts’ as shown

below:

� If you correctly, did the above step, then you will have the main windows as shown above. Now, we

can generate the addresses for our system by clicking the "Generate Address" button from the

"Addressed Tab" as shown below.

IMPORTANT! There is a BUG in the Xilinx Platform Studio. If you miss this step, you will not be able

to communicate with the shared peripherals. To fix this bug, go the projects tab and open

<YOUR_PROJECT_NAME>.MHS file as shown above. Find the "plb_v46" with INSTANCE

"mb_plb_0". Copy the two lines starting with "PORT". Now find the "plb_v46" with INSTANCE

"shared_plb". You will see that the PORT entries are missing. Paste the two lines before END (similar to

the one shown below).

IMPORTANT: The DDR_SDRAM peripheral uses a lot of Block RAM resources. As mentioned earlier,

we have limited BRAM resources. If we attempt to build the system now, it will fail because the number

of BRAMs required by our design will be larger than what is available on the FPGA. To instruct

DDR_SDRAM peripheral NOT to use BRAM, we configure it and navigate to the "Advanced" tab and

then to the "Data Path Configuration" sub-tab. You will need change "Read FIFO Config" and "Write

FIFO Config" from BRAM to SRL as shown.

� Now, you have to add the TFT_Controller to the PLB bus of microblaze_0 processor which is

mb_plb_0. Consult lab 1C for detail help on adding TFT controller.

**Note: As you can remember from Lab 1C on display configuration, to specify the video memory to the

TFT Controller, we used the on board DDR memory for this purpose. Identify the High address of the

DDR_SDRAM from the Addresses tab and subtract 2 MB from this address as TFT controller requires 2

MB of memory. Now open the controller instance for configuration and specify the resultant address of

the DDR_SDRAM in the location of "Base address of PLB attached Video Memory". For example: if the

High address of DDR_SDRAM is 0xC7FFFFFF, then the base address of video memory for TFT

controller will be: 0xC7FFFFFF – (hex of (2*1024*1024) + 1. (Since the address starts from 0 and will

be 0 to 1023 for 1 KB of memory which is equivalent to 1024)

� To generate the hardware, we can select the Generate Bitstream from the menu-bar or the toolbar.

This will take some time to complete.

� CONGRATULATIONS. YOU HAVE SUCCESSFULLY COMPLETED YOUR FIRST DUAL

MICROBLAZE HARDWARE DESIGN ON FPGA.

� Use the correct STDIO settings. Now, you can try out the Test Applications by downloading to FPGA

and run to see the output at console.

**Note: Remember to export your hardware platform from XPS to Xilinx SDK and create a BSP before

creating a C application: follow steps from Lab 1B)

� Before running the test program, you need to make some changes in the SDK platform as discussed

below:

**Note: Now the base address of DDR_SDRAM memory (shared memory) for both the processors i.e.

Microblaze_0 & Microblaze_1 is same which by default 0xC4000000 with a length of 0x04000000

(which is equal to 64 MB of size) and can be seen from the lscript.ld file under src folder of any

application project (i.e. hello_world_0) as shown in figure below. Hence the program memory address

has been overlapped by different programs from two different processors in a dual-core system. To avoid

this problem, we can assign different base address for different microblaze processors. Since very likely

most of the cases the program don’t need more than 16 MB, we can make the length as 16 MB for each

processor’s application program memory which equals 0x01000000. So if the base address of

DDR_SDRAM for microblaze_0 processor is 0xC4000000 then the base address of DDR_SDRAM for

microblaze_1 will be 0xC4000000 + 0x01000000 = 0xC5000000. So before you run any program we

need to change the base address of SDRAM according to under which processor the program will be run.

� If you are successful to run any test program on dual core platform, then we can move into the next

step on how to use interrupt and mtex, mailbox, message queue.

LAB 3B

MUTEX

Procedure

� There are two source codes "Ball.c" and "Players.c" in the Lab3 folder. We will create software

applications for both Microblaze_0 and Microblaze_1 using these two source codes.

� Create a Xilinx C project for Microblaze 0 using the "Players.c" as the source code. From the code,

identify which scheduling policy is to be used and make the corresponding settings in the "Software

Platform Settings" for Microblaze 0.

� Create another Xilinx C project for Microblaze 1 using "Ball.c" as the source code. Again identify

which is the scheduling policy used in the code and make the corresponding changes in the Software

Platform Settings for Microblaze 1.

� Configure the STDIN and STDOUT of both the microblazes to be mdm_0.

� Download the bitstream of the 1st application project to the FPGA board and notice the output to be

similar to given below

� You will find that since the threads in both the processors are using the same STDOUT peripheral,

the outputs from all the threads are completely jumbled and out of sync.

� In order to resolve this issue, we can use the concept of mutex so that we can ensure that only one

thread at a time has access to the UART. We will use software mutex for all threads running on the

same processor and the hardware mutex for threads running on different processors to solve the issue

of UART access.

ENABLING SOFTWARE MUTEX

� The source Code "Players.c" creates four threads that will be run in microblaze 0. So we will enable

software mutex in microblaze 0 to resolve the UART access issue among the 4 threads. For this

purpose, go to the "Board Support Package Settings" of microblaze0 and set the value of parameter

"config_pthread_mutex" to true as shown below. Click OK to continue.

� Now, in the source code "Players.c", we have to make the corresponding changes so that the API for

software mutex can be accessed. Prior to using the APIs, a mutex lock identifier of type

"pthread_mutex_t" is to be created. The APIs to be used for software mutex are given below:

� int pthread_mutex_init(

pthread_mutex_t* mutex,

const pthread_mutexattr_t* attr)

� int pthread_mutex_lock(

pthread_mutex_t* mutex)

� int pthread_mutex_unlock(

pthread_mutex_t* mutex)

� The details are given on page 29 on API documentation of Xilkernel (xilkernel_v5_00_a.pdf). We

will discuss the above API's in lab (look it up!).

� Once the software mutex is initialized, you have to lock the resource to be used using the

pthread_mutex_lock API. Once the thread is ready to release the resource, the pthread_mutex_unlock

API can be called to unlock the resource. This resource is now free and can now be locked by another

thread.

ENABLING THE HARDWARE MUTEX

� Enabling the Hardware Mutex is just like how any other peripherals are initialized in the system.

There are a number of APIs available for the hardware mutex. We require only four of these APIs

for implementing mutex between the two processors. The APIs to be used for hardware mutex are

given below (look them up!):

� XMutex_Config* XMutex_LookupConfig(

DEVICE ID)

� int XMutex_CfgInitialize(

XMutex* instance pointer,

XMutex_Config *ConfigPtr,

ConfigPtr->BaseAddress)

� void XMutex_Lock(

XMutex* InstancePtr,

u8 MutexNumber)

� void XMutex_Unlock(

XMutex* InstancePtr,

u8 MutexNumber)

� The header file to be included in your source code for accessing the hardware mutex API is

"xmutex.h". You can read more about these API and the hardware configuration properties of the

hardware mutex by right clicking the hardware mutex instance and selecting the appropriate option.

� Also make the Heap size as 1 KB from the “Generate Linker Script” and select

DDR_SDRAM_MPMC_BASEADDR in the option from Place Code Sections in the window as

shown below. Then click Generate to continue.

� Now select the base address of DDR_SDRAM memory (shared memory) for Microblaze_0 to

0xC4000000 as origin with a length of 0x01000000 (which is equal to 16 MB of size) from the

lscript.ld file under src folder and for Microblaze_1, select base address to 0xC5000000 as origin

with a length of 0x01000000. Then build the application projects.

LAB 3C

MESSAGE QUEUES & MAILBOXES

MESSAGE QUEUES

� To use message queues, we need to configure xilkernel to support message queues. Please refer to

figure below and make the corresponding changes under the "Board Support Package Settings" menu:

� Set the following parameters as given below:

� config_sema = true

� config_named_sema = false

� config_msgq = true

� num_msgqs = 1

� msgq_capacity = 50

� use_malloc = true

� Message queues use malloc() and free() functions or their xilinx counterparts (bufmalloc and

buffree). Therefore, it is important to increase the heap from 0x0000 to 0x0400 through the

"Generate Linker Script".

QUESTION: Why do we need HEAP for malloc() and free() (or their xilinx counterparts)?

� The source code to be used for message queues is msgqueue.c.

� Create a new software application project using this C code as source. You may use Microblaze 0

as the associated processor for this.

� In this program, three separate threads are created, that sends data to a common display thread

called display. The three sending threads use a common function send that has a structure "msg"

consisting of three variables: id, x and y.

� Each thread has a specific value of id, x and y that it sends to the send function. These values are

modifiable by you to suit your requirements. The send function then passes it to the display thread

using the message queue APIs.

� The message queue APIs to be used are

� int msgget(

key_t key,

int msgflg)

� int msgsnd(

int msqid,

const void* msgp,

size_t msgsz,

int msgflg)

� ssize_t msgrcv(

int msqid,

void* msgp,

size_t nbytes,

long msgtyp,

int msgflg)

� The details are given on page 22 on API documentation of Xilkernel (xilkernel_v5_00_a.pdf). We

will discuss the above API's in lab.

MAILBOXES

� As described earlier, mailboxes are used for communication between threads running on different

processor cores. For this section, there are two source codes, mailbox_ball.C and mailbox_players.c.

MAILBOX_BALL.C: This source code contains a structure ball_msg, which contains the entities dir,

speed, x and y. This source code is to be executed in one microblaze Core, say microblaze1. The program

creates a thread send() that transmits a message packet consisting of the dir, speed, x and y to the

hardware mailbox IP using the corresponding API calls.

MAILBOX_PLAYERS.C: This code contains creates two data structures: ball_msg with entities dir,

speed, x and y. and player_msg with entities id, x and y. There are five threads created in this program.

The thread display receives the data send by other threads in the program and sends it to the UART

terminal.

� The thread thread_func_4 receives the ball data structure from the hardware mailbox (sent from

microblaze 1) and then sends it again to the display thread.

� The other three threads deal with player_msg structure and continuously send data to the display

thread.

� This program is run on microblaze 0 and uses message queues to pass data from one thread to

another, and uses the mailbox to receive data from threads running on another processor.

� The mailbox IP can be configured just like other hardware IPs that we dealt with in earlier labs. The

APIs to be used for hardware mutex are given below:

� XMbox_Config* XMbox_LookupConfig(

DEVICE ID)

� void XMbox_WriteBlocking(

XMboxx* InstancePtr,

u32* BufferPtr,

u32 RequestedBytes)

� int XMbox_Read(

XMbox* InstancePtr,

u32* BufferPtr,

u32 RequestedBytes,

u32* BytesRecvdPtr)

� int XMbox_CfgInitialize(

XMbox *InstancePtr,

XMbox_Config *ConfigPtr,

u32 EffectiveAddress)

� The header file to be used is xmbox.h. Try to understand the functioning of APIs from the driver

documentation of mailbox.

LAB 3D

Interrupts

Peripheral Interrupts in Xilkernel

� The following code snippets explain how UARTLITE interrupts are enabled to receive data

through RS232_DTE port:

XUartLite uartLiteRecvInst;

//The Receive Interrupt Handler

static void Recv_Handler (void* CallBackRef, unsigned int EventData)

{

xil_printf("Inside the receive handler\r\n");

}

main()

{

// Initialize UARTLITE

status = XUartLite_Initialize (&uartLiteRecvInst, XPAR_RS232_DTE_DEVICE_ID);

if (status != XST_SUCCESS)

{

return -1; //XST_FAILURE;

}

// Register and enable interrupt handler for UartLite RS232_DTE

ret = register_int_handler(XPAR_XPS_INTC_0_RS232_DTE_INTERRUPT_INTR, XUartLite_InterruptHandler, (void

*)&uartLiteRecvInst);

// Registering the receive Xuartlite interrupt handler

XuartLite_SetRecvHandler(&uartLiteRecvInst, Recv_Handler, &uartLiteRecvInst);

// Enabling interrupt in the XILKERNEL

enable_interrupt(XPAR_XPS_INTC_0_RS232_DTE_INTERRUPT_INTR);

// Enabling interrupt in the Xuartlite hardware i.e. RS232_DTE XuartLite_EnableInterrupt(&uartLiteRecvInst);

///////////////////////////////////////////////////////

}

** Note: register_int_handler() and enable_interrupt() are XILKERNEL APIs.

XuartLite_SetRecvHandler() and XuartLite_EnableInterrupt() are UARTLITE driver APIs.

**This code is only for understanding purpose.

Reading GPIO

The following code snippets show the relevant driver API’s to enable and read push button:

XGpio gpPB;

XGpio_Initialize(&gpPB, XPAR_BUTTONS_3BIT_DEVICE_ID);

//Set the Push Button peripheral to inputs

XGpio_SetDataDirection(&gpPB, 1, 0x0000000F);

//Read the state of the push buttons

val = XGpio_DiscreteRead(&gpPB, 1);

** Note: Interrupts for GPIO: As done in Lab 3A part while building base system, to enable

interrupts for GPIO, you need to enable them first by configuring the IP through XPS. Also you

need to add the interrupt to the interrupt controller for the respective microblaze. Later, you need

to use the Xilkernel's API as well as the GPIO's API's to enable interrupts in a similar way as

shown for UARTLITE.

� Find the source code sender_0.c and receiver_1.c in the Lab3 folder. Create two BSP namely

xilkernel_0 and xilkernel_1 for two microblazes microblaze_0 and microblaze_1 respectively. Then

create two new C applications associated with these two BSP settings. Copy the source code

sender_0.c under src folder of the application that belongs to microblaze_0 and similarly copy

receiver_1.c for microblaze_1.

� The sample interrupt based system has two parts

1) Sender, running on microblaze_0:

The three pushbuttons will generate interrupts when pressed or released. With each interrupt,

the current 3-bit value is sent out through DTE (which can initially be DCE itself for testing).

2) Receiver, running on microblaze_1:

When a byte is received by DCE, it interrupts microblaze_1. The interrupt service routine

prints the received value.

� Try to understand the two sample codes on how we implement the interrupt. Take a good look at the

comments in the code. E.g. read and understand the xilkernel system calls as well as the driver APIs

that are used in the program.

** Note: Only 3 push buttons out of the 4 can be used. Pressing BTN_South will reset the FPGA. Most

peripheral interrupts work similar to GPIO interrupts. However, UARTLite interrupt handling is done a

bit differently.

� Once we start running both, and connect DCE to DTE together by the serial cable, the whole system

works as expected.

HOMEWORK

PART 1A: SOFTWARE MUTEX

This will use one microblaze processor.

• Modify the "Players.c" and use the software mutex API at suitable locations so that conflicts

between the threads of "Players.c" are resolved.

PART 1B: HARDWARE MUTEX

This program will use two microblaze.

• Modify Players.c and Ball.c source code such that the threads in both the programs use the

hardware mutex to resolve the UART access issues.

PART 2A: MESSAGE QUEUES

There are four threads running in the program on

the same microblaze core. Use an appropriate

technique to ensure that the UART access issue

among the threads while displaying the output is

resolved.

• Complete the code "msgqueue.c" to

implement the message queue and view the

output.

• Vary the parameter msgq_capacity and

see how the output varies.

Output will be similar to the one shown in figure

on the right.

PART 2B: MAILBOXES

This program will use two microblazes. Use the techniques required to resolve the UART conflicts for the

multiple threads running on both the cores.

• Complete the code for MAILBOX_BALL.C and MAILBOX_PLAYERS.C as per the requirements.

• The MAILBOX_BALL.C sends the message containing data structure ball_msg from Microblaze 1

to Microblaze0.

� Use the APIs of Mailbox IP to configure the Mailbox IP

� Complete the send() function so that it calls the correct Mailbox APIs to send messages to

Microblaze 0.

• Complete the MAILBOX_PLAYERS.C so as to

� Use the API of Mailbox IP to configure the IP.

� Complete the thread_func_4() such that it receives data from Microblaze1 through the

Mailbox and then send the received data to the thread display.

� Complete the thread display such that it receives data from all the four threads and then

display it through the UART terminal.

PART 2C: INTERRUPTS

Modify the code in PART 2B to use interrupts wherever possible (hint : mailbox interrupts, UART sent

interrupts etc.)