rsu over pcie design example - intel · 2020-01-20 · with different transfer sizes, and source...

Altera Asia Pacific Regional Support Center

1


Reference Design Document


RSU over PCIe design example


2




TABLE OF CONTENTS

INTRODUCTION .......................................................................................................................................................................... 4

1.1 OVERVIEW .......................................................................................................................................................................... 4 1.1.1 Brief Functional Description ...................................................................................................................................... 4 1.1.2 Target Application ...................................................................................................................................................... 4 1.1.3 Features (Benefits) ...................................................................................................................................................... 4 1.1.4 Project Deliverables .................................................................................................................................................... 4

1.2 TOP LEVEL ARCHITECTURE ............................................................................................................................................... 4 1.2.1 User Application ......................................................................................................................................................... 5 1.2.2 Driver ......................................................................................................................................................................... 5 1.2.3 Qsys ............................................................................................................................................................................ 5

2 SYSTEM REQUIREMENT .................................................................................................................................................... 7

2.1 HARDWARE REQUIREMENTS ............................................................................................................................................. 7 2.2 SOFTWARE REQUIREMENTS ............................................................................................................................................... 7

3 FUNCTIONAL BLOCK DESCRIPTION ............................................................................................................................ 8

3.1 AVMM HARD IP FOR PCI EXPRESS ..................................................................................................................................... 8 3.2 MODULAR SCATTER-GATHER DMA ................................................................................................................................. 8 3.3 ALTERA SERIAL FLASH CONTROLLER ............................................................................................................................... 8 3.4 ALTERA REMOTE UPDATE ................................................................................................................................................. 9 3.5 RU_SM_BLOCK ................................................................................................................................................................... 9

4 EXTERNAL INTERFACE .................................................................................................................................................... 12

4.1 I/O DESCRIPTION ............................................................................................................................................................. 12

5 REGISTERS INTERFACE ................................................................................................................................................... 14

5.1.1 PCIE hard IP CRA interface(bar0 address range 0x02000000 – 0x02003fff).......................................................... 14 5.1.1 Remote Update CSR interface(bar0 address range 0x02020000 – 0x0202001f) ...................................................... 16 5.1.2 Modular SGDMA descriptor slave interface(bar0 address range 0x03000000 – 0x0300000f) ............................... 17 5.1.3 Modular SGDMA CSR interface(bar0 address range 0x03010000 – 0x0301001f) ............................................... 19 5.1.4 Altera Serial Flash Controller CSR interface(bar0 address range 0x06000000 – 0x0600001f) .............................. 21

6 DATA FLOW AND CLOCK DISTRIBUTION ............................................................................................................... 28

6.1 DATA FLOW ..................................................................................................................................................................... 28 6.1.1 Software download program file to EPCQ ............................................................................................................... 28 6.1.2 Software control Remote Update to reconfig FPGA ................................................................................................. 28 6.1.3 Hardware control Remote Update to reconfig FPGA ............................................................................................... 28

6.2 CLOCK DISTRIBUTION ...................................................................................................................................................... 28

7 RUNNING THE EXAMPLE DESIGN ............................................................................................................................... 29

7.1 SET UP THE HARDWARE ENVIRONMENT ......................................................................................................................... 29 7.2 COMPILE THE FACTORY AND APPLICATION DESIGN ....................................................................................................... 31

7.2.1 Open the Example design ......................................................................................................................................... 31 7.2.2 Compile the factory design ........................................................................................................................................ 32


3



Reference Design Document 7.2.3 Compile the application_0 design ............................................................................................................................. 32 7.2.4 Compile the application_1 design ............................................................................................................................. 33

7.3 GENERATE FACTORY JIC FILE ........................................................................................................................................... 34 7.4 DOWNLOAD FACTORY IMAGE TO EPCQ ........................................................................................................................ 36

7.4.1 Power up Stratix V GX FPGA Development Kit ..................................................................................................... 36 7.4.2 Changing the TCK Frequency of USB Blaster II cable ............................................................................................. 36 7.4.3 Download factory.jic file via Programmer ................................................................................................................ 37 7.4.4 Configure FPGA using Factory Image. .................................................................................................................... 37

7.5 GENERATE APPLICATION RPD FILE .................................................................................................................................. 38 7.6 INSTALL LINUX DRIVER ................................................................................................................................................... 41 7.7 SOFTWARE FLOW TO DOWNLOAD APPLICATION_0 IMAGE AND RECONFIGURE FPGA TO APPLICATION_0 IMAGE ... 42 7.8 SOFTWARE FLOW TO RETURN FACTORY IMAGE .............................................................................................................. 43 7.9 SOFTWARE FLOW TO DOWNLOAD APPLICATION_1 IMAGE AND RECONFIGURE FPGA TO APPLICATION_1 IMAGE ... 45 7.10 HARDWARE FLOW TO RECONFIGURE FPGA ................................................................................................................... 46 7.11 UNINSTALL LINUX DRIVER .............................................................................................................................................. 48

8 CONCLUSION ...................................................................................................................................................................... 50

9 REVISION HISTORY .......................................................................................................................................................... 51


4




Introduction

1.1 Overview

The RSU over PCIe design example is target to demonstrate a method to use PC to download program files to

EPCQ, and control Remote Update IP to achieve FPGA reconfiguration with the downloaded program files.

1.1.1 Brief Functional Description

PCI Express used to update the flash through EPCQ controller and also as a driver of the RSU block. Essentially this

makes a software based flow to RSU through program files on external machines and allows for autonomous

movement from Factory to Application images. Better touchless over the air update capabilities.

1.1.2 Target Application

All customers could use this as a what to do Over the Air Updates and push out new image releases for bug fixes

1.1.3 Features (Benefits)

Field technicians don’t have to go out and flash the EPCQ images to the cards, can be done through software via

PCIe and all the RSU registers would be able to be configured through software and an autonomous hardware feature

1.1.4 Project Deliverables

Qsys system solution for a Cyclone V develop kit or Stratix V develop kit to start. This should be portable to all dev

kits. More compelling since it adds in multiple IPs in a Qsys based system and would use AVMM to communicate

between IPs.

1.2 Top Level Architecture

The reference design is made up of three parts: User application, Driver, Qsys; user application and driver are

running at PC which has linux system installed. And Qsys is running in FPGA.


5



Reference Design Document Figure 0-1 Top Level Block Diagram

application

driver

Avmm hard ip for PCI express

ru_sm altera serial flash controller

EPCQ

PC

QSYS

avl_csr avl_memavl_csr

modular scatter-gather dmaTxs

csr

read/w rite/open/release function

Rxm_bar0

mm_read

mm_w rite

FPGA

factory design or

application design

altera remote updateavl_csravl_master

app_page_select

reconfig_enable_n

reconfig_status

factory_select

1.2.1 User Application

User Application could access PCIe BAR space register, and read program file locally and write it to PCIe Card via Driver. Also could read the program file back from PCIe Card via Driver and compare with local file for checking.

1.2.2 Driver

Using read/write functions of file_operations to offer PCIE read/write interface to application. Add parameter on write function, the application could select DMA for large dataq transmission.

1.2.3 Qsys

Build a Qsys which will be a sub-module of the customer’s design to achieve RSU for both factory design and application design. It’s the key part of this reference design. 1.2.3.1 Altera serial flash controller(EPCQ controller):

Altera serial flash controller uses avl_mem interface to write data to EPCQ devices. And use avl_csr interface could access control and status registers.

The Maximum operation frequency of this module is 25MHZ, and the external operation to EPCQ is serial, so the bandwidth of program file download is 25Mbps. 1.2.3.2 Altera remote update:

Altera remote update is used to reconfig FPGA with specify image by controlling the avl_csr interface. It could also report some status of reconfiguration after reconfiguration. 1.2.3.3 Avmm hard ip for PCI express

Using PCIE Gen1 x4 to compatible with all the dev kit, the bandwidth of EPCQ controller is 25Mbps, the bandwidth of PCIE Gen1 x4 is 10Gbps, So the bottleneck is still on EPCQ controller.


6



Reference Design Document 1.2.3.4 Modular scatter-gather DMA:

Though the bandwidth of PCIe is enough for this reference design, but since the program file(.rpd) is usually about 25M B, may occupy CPU bandwidth when data transmission, so it’s better to use DMA for program file transform. 1.2.3.5 RU state machine

Provide the hardware way to control FPGA reconfiguration among Factory and application images.


7




2 System Requirement

This section describes the hardware and software requirements to run the RSU over PCIe design example.

2.1 Hardware Requirements

The RSU over PCIe Design Example requires the following hardware components:

Stratix V GX FPGA Development Kit

Altera USB-Blaster™ II cable

Computer I with PCIe x8 Slot and CentOS7 installed

Computer II with Quartus II 15.0.2 installed

2.2 Software Requirements

You must install CentOS Linux 7 64bit, on Computer I, the kernel version is 3.10.0-123.el7 .

You must install the Quartus II software, version 15.0.2, which includes QSYS on Computer II.

This application note assumes that you install the software into the default locations.

Ensure that you can extract rsu_over_pcie_design_example.qar and open rsu_over_pcie_design_example.qpf in your Quartus II software.


8




3 Functional block description The blocks outline, introduce the major block talked in this chapter, and the relationship between them.

3.1 Avmm hard IP for PCI express

Altera Stratix V FPGAs include a configurable, hardened protocol stack for PCI ExpressR that is compliant with PCI

Express Base Specification 2.1 or 3.0.The Hard IP for PCI Express IP core using the Avalon R Memory-Mapped

(Avalon-MM) interface removes some of the complexities associated with the PCIe protocol. For example, it handles

all of the Transaction Layer Protocol (TLP) encoding and decoding. Consequently, you can complete your design

more quickly. The Avalon-MM interface is implemented as a bridge in soft logic. It is available in Qsys. Related Information: https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/ug/ug_s5_pcie_avmm.pdf

3.2 Modular scatter-gather DMA

In a processor subsystem, data transfers between two memory spaces can happen frequently. In order to

offload the processor from moving data around a system, a Direct Memory Access (DMA) engine can be

introduced to perform this function instead. The Modular Scatter-Gather DMA (mSGDMA) is capable of

performing data movement operations with preloaded instructions, called descriptors. Multiple descriptors

with different transfer sizes, and source and destination addresses are supported with the option to

trigger interrupts.

The mSGDMA has a modular design that facilitates easy integration with the FPGA fabric. It consists of a

dispatcher block with optional read master and write master blocks. The descriptor block receives and

decodes the descriptor and dispatches instructions to the read master and write master blocks for further

operation. It can also be configured to transfer additional information to the host. In this context, the read

master block reads data via its Avalon-MM master interface and channels it into Avalon-ST source

interface based on instruction given by dispatcher block. On the other hand, the write master block

receives data from its Avalon-ST sink interface and write it to the destination address via its Avalon-MM

master interface.

Altera mSGDMA provides three configuration structures for handling data transfers between Avalon-

MM to Avalon-MM, Avalon-MM to Avalon-ST, and Avalon-ST to Avalon-MM modes. Sub-core of

mSGDMA are instantiated automatically according to the structure configured for mSGDMA use model.

Related Information:

www.altera.com/en_US/pdfs/literature/ug/ug_embedded_ip.pdf

3.3 Altera Serial Flash Controller

The Altera Serial Flash Controller wraps around the Altera ASMI PARALLEL IP, and consists of some

conversion logic which converts the ASMI PARALLEL conduit interface to Avalon interface.


www.altera.com/en_US/pdfs/literature/ug/ug_embedded_ip.pdf

https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/ug/ug_s5_pcie_avmm.pdf

http://www.altera.com/en_US/pdfs/literature/ug/ug_embedded_ip.pdf

http://www.altera.com/en_US/pdfs/literature/ug/ug_embedded_ip.pdf


9




3.4 Altera Remote Update

The Altera Remote Update IP core implements a device reconfiguration using dedicated remote system upgrade

circuitry available in supported devices. Remote system update helps you deliver feature enhancements and bug fixes

without recalling your product, and reduces time-to-market and extends product life.

The Altera Remote Update IP core commands the configuration circuitry to start a reconfiguration cycle. The

dedicated circuitry performs error detection during and after the configuration process. When the dedicated circuitry

detects errors, the circuitry facilitates system recovery by reverting back to a safe, default factory configuration image

and then provides error status information.


https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/ug/ug_altremote.pdf

3.5 Ru_sm_block

Figure 3-2 Factory user RSU state machine

Power-up/Reconfiguration

Read the reconfiguration source conditionreconfig_status

Read the reconfiguration modereconfig_mode

waiting for user triggerreconfig_enable

Set the watchdog timer time-out value

enable the watchdog timer

Set the application image start address

Set anf bit 1

enable ru reconfig

https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/ug/ug_altremote.pdf


10



Reference Design Document Factory user RSU code

Factory State Machine

1. Wait for Reset

2. Read reconfig mode in RSU core

3. Read reconfig trigger condition in RSU core

4. If reconfig source=5’b0_0000 (indicates factory image loaded after power up) move to state 5 else stay

here in state 4

5. Waiting for user reconfigure trigger

6. Set AnF=1 in RSU core // indicates intent to move to application image

7. Set Wdog timeout value to ~4sec in RSU core

8. Set Wdog enable in RSU core

9. Set Application Image Boot Address to 32’haa0_0000(or 32’h1540000) in RSU core // divide 256Mbit flash

to 3 pages

10. Set reconfig=1 in RSU core to trigger reconfig to application image

Figure 3-3 Application user RSU state machine

Power-up/Reconfiguration

Read the reconfiguration source conditionreconfig_status

Read the reconfiguration modereconfig_mode

trigger reset timer

reconfig_enable

clear the watchdog timer time-out value

disable the watchdog timer

Set the factory image start address

Set anf to 0

enable ru reconfig

trigger interleave delay


11




Application user RSU code

Application State Machine

1. Wait for Reset

2. Read reconfig mode in RSU core

3. Read reconfig trigger condition in RSU core

4. kick the Wdog by issuing a falling edge to the reset_timer of Cyclone V RSU core and move to state 5

5. Delay 3 sec， if user trigger reconfiguration go to state 6, else go back to 4.

6. Set AnF=0 in RSU core // indicates intent to move to application image

7. Set Wdog timeout value to 0 in RSU core

8. Set Wdog disable in RSU core

9. Set Application Image Boot Address to 32’h0 in RSU core

10. Set reconfig=1 in RSU core to trigger reconfig to application image


12




4 External Interface

The external interface of this reference design are mainly PCIE interface and EPCQ interface. Below is showing the IO

information on Stratix V GX FPGA Development Kit.

4.1 I/O Description

Table 2-1 Global Signals

Name Pin Location I/O standard Description

clkin_50 AN6 1.8 V Input reference clock

Table 2-2 PCIe interface

Name Pin Location I/O standard Description

pcie_refclk_p AF34 1.4-V PCML PCIe reference clock

pcie_perstn AV38 2.5 V Reset input

pcie_rx_p[0] AT38 1.4-V PCML Pcie RX pin

pcie_rx_p[1] AP38 1.4-V PCML Pcie RX pin

pcie_rx_p[2] AM38 1.4-V PCML Pcie RX pin

pcie_rx_p[3] AH38 1.4-V PCML Pcie RX pin

pcie_tx_p[0] AU36 1.4-V PCML Pcie TX pin

pcie_tx_p[1] AR36 1.4-V PCML Pcie TX pin

pcie_tx_p[2] AN36 1.4-V PCML Pcie TX pin

pcie_tx_p[3] AL36 1.4-V PCML Pcie TX pin

Table 2-3 EPCQ interface

Name Pin Location Description

DCLK AC31

Dedicated configuration clock pin. DCLK is used to clock configuration data from an external source into the FPGA. In AS mode, DCLK is an output from the FPGA that provides timing for the configuration interface.

NCSO AD32 Dedicated output control signal from the FPGA to the serial configuration device in AS mode that enables the configuration device.

AS_DATA0 AB31

Dedicated AS configuration pin. When using an EPCS device (x1 mode) this is the ASDO pin used to send address and control signals between the FPGA and the EPCS/EPCQ.


13




AS_DATA1 Y31 Dedicated AS configuration data pins. Configuration data is transported on these pins when connected to the EPCQ devices.

AS_DATA2 AC32 Dedicated AS configuration data pins. Configuration data is transported on these pins when connected to the EPCQ devices.

AS_DATA3 AG32 Dedicated AS configuration data pins. Configuration data is transported on these pins when connected to the EPCQ devices.


14




5 Registers Interface Below is the address map in QSYS, there are 3 avalon master in it.

Figure 2-1 Address Map

msgdma_0.mm_read and msgdma_0.mm_write are charging for data transfer Between pcie TXS slave port and

epcq controller avl mem slave port. The address range of epcq controller.avl_mem is 0x4000000~0x5ffffff. Will be used

for DMA transfer from PC to EPCQ.

Pcie_sv_hip_avmm_0.Rxm_BAR0 is used to access register of each module made up of this system. PC could use

PCIe BAR0 space to read and write the registers in each module based on this interconnect. There is no registers in

pcie_sv_hip_avmm_0.Txs slave port and epcq_controller_0.avl_mem. These two interfaces is connected to Rxm_BAR0

for data testing and checking.

In this chapter, the registers in each module used in this reference design will be listed .

5.1.1 PCIE hard IP CRA interface(bar0 address range 0x02000000 – 0x02003fff)

5.1.1.1 Registers Map

Table 2-4 PCIE hard IP CRA interface registers map

Address

Offset

Name Description

0x0050 Avalon-MM to PCI Express Interrupt

Enable Register

A PCI Express interrupt can be asserted for any of the

conditions registered in the Avalon-MM to PCI

Express Interrupt Status register by setting the corresponding

bits in the Avalon-MM-to-PCI Express

Interrupt Enable register.

0x1000-

0x1FFF

Avalon-MM-to-PCI Express Address

Translation Table

The Avalon-MM-to-PCI Express address translation table is

writable using the CRA slave port. Each entry in the PCI

Express address translation table is 8 bytes wide, regardless of

the value in the current

PCI Express address width parameter. Therefore, register

addresses are always the same width, regardless of PCI

Express address width.


15



Reference Design Document 5.1.1.2 Registers description

Table 2-5 Avalon-MM to PCI Express Interrupt Enable Register

Bits Name Access Description

[31:24] Reserved N/A N/A

[23:16] A2P_MB_IRQ RW

Enables generation of PCI Express interrupts when a specified mailbox is written to by an external Avalon-MM master.

[4:0] AVL_IRQ[15:0] RW

Enables generation of PCI Express interrupts when a specified Avalon-MM interrupt signal is asserted. Your Qsys system may have as many as 16 individual input interrupt signals.

Table 2-6 Avalon-MM-to-PCI Express Address Translation Table(0x1000)


[1:0] A2P_ADDR_ SPACE0 N/A

Address space indication for entry 0. Refer to Table 9– 31 for the definition of these bits.

[31:2] A2P_ADDR_ MAP_LO0 RW

Lower bits of Avalon-MM-to-PCI Express address map entry 0.

Table 2-6 Avalon-MM-to-PCI Express Address Translation Table(0x1004)


[31:0] A2P_ADDR_ MAP_HI0 RW

Upper bits of Avalon-MM-to-PCI Express address map entry 0.


16



Reference Design Document 5.1.1 Remote Update CSR interface(bar0 address range 0x02020000 – 0x0202001f)


The address offset is byte address type, which you see in Altera Remote Update Userguide is word address type.

Table 2-7 Remote Update CSR interface registers map

Address

Offset Register Name Width R/W Description

0x0 RU_RECONFIG_

TRIGGER_CONDITIONS

5 Read Read configuration trigger

conditions.

• Bit 4—wdtimer_source: Users

Watchdog Timer timeout

• Bit 3—nconfig_source: External

configuration reset (nCONFIG)

assertion.

• Bit 2—runconfig_source:

Configuration reset triggered from

logic array

• Bit 1—nstatus_source:

nSTATUS asserted by an external

device as the result of an error

• Bit 0—crcerror_source: CRC

error during application configuration.

0x4 RU_WATCHDOG_TIMEOUT 12 Read/Write Read or write Watchdog Timeout

value

0x8 RU_WATCHDOG_ENABLE 1 Read/Write Enable or disable Watchdog Timeout.

• 0: Disable

• 1: Enable

0xc RU_PAGE_SELECT 24 or 32 Read/Write Read or write start address of

configuration image.

0x10 RU_CONFIGURATION_

MODE

1 Read/Write Write configuration mode set to 1 in

application page and 0 in factory page

0x14 RU_RESET_TIMER 1 Write Write a value of 1 to this register to

trigger reset timer of the remote

update. The IP will automatically

trigger a reset pulse to reset timer pin

of the remote update.

0x18 RU_RECONFIG 1 Write Write to this address with value of 1

to trigger reconfiguration from a new

image. The IP will set 1 to reconfig

pin of the remote update and hold

this value until the process done.


17



Reference Design Document 5.1.2 Modular SGDMA descriptor slave interface(bar0 address range 0x03000000 – 0x0300000f)


The address offset is byte address type. This reference design is using Standard Descriptor Format.

Table 2-8 Modular SGDMA descriptor slave interface registers map

Address


0x0 Read Address[31:0] 32 Write DMA source address

0x4 Read Address[31:0] 32 Write DMA destination address

0x8 Length[31:0] 32 Write The length field is used to specify the number of

bytes

to transfer per descriptor.0xC Control[31:0] 32 Write The control field is available for both the standard and

extended descriptor formats. This field can be

programmed to configure parked descriptors, error

handling and interrupt masks. The interrupt masks

are programmed into the descriptor so that interrupt

enables can be unique for each transfer. 5.1.2.2 Registers description

Table 2-9 Descriptor Control Field Bit Definition

Bit Sub-Field Name Definition

31 Go Used to commit all the descriptor information into the descriptor FIFO. As the host writes different fields in the descriptor, FIFO byte enables are asserted to transfer the write data to appropriate byte locations in the FIFO. However, the data written is not committed until the go bit has been written. As a result, ensure that the go bit is the last bit written for each descriptor. Writing '1' to the go bit commits the entire descriptor into the descriptor FIFO(s).

30:25 <reserved>

24 Early done enable Used to hide the latency between read descriptors. When the read master is set, it does not wait for pending reads to return before requesting another descriptor. Typically this bit is set for all descriptors except the last one. This bit is most effective for hiding high read latency. For example, it reads from SDRAM, PCIe, and SRIO.


18



Reference Design Document 23:16 Transmit Error / Error

IRQ Enable For MM->ST transfers, this field is used to specify a transmit error. This field is commonly used for transmitting error information downstream to streaming components such as an Ethernet MAC. In this mode, it controls the error bits on the streaming output of the read master. For ST->MM transfers, this field is used as an error interrupt mask. As errors arrive at the write master streaming sink port, they are held persistently; and when the transfer completes, if any error bits were set at any time during the transfer and the error interrupt mask bits are set, then the host receives an interrupt. In this mode, it is used as an error encountered interrupt enable.

15 Early Termination IRQ Enable

Used to signal an interrupt to the host when a ST->MM transfer completes early. For example, if you set this bit and set the length field to 1MB for ST->MM transfers, this interrupt asserts when more than 1MB of data arrives to the write master without the end of packet being seen.

14 Transfer Complete IRQ Enable

Used to signal an interrupt to the host when a transfer completes. In the case of MM->ST transfers, this interrupt is based on the read master completing a transfer. In the case of ST->MM or MM->MM transfers, this interrupt is based on the write master completing a transfer.

13 <reserved>

12 End on EOP End on end of packet allows the write master to continuously transfer data during ST->MM transfers without knowing how much data is arriving ahead of time. This bit is commonly set for packetbased traffic such as Ethernet.

11 Park Writes When set, the dispatcher continues to reissue the same escriptor to the write master when no other descriptors are buffered.

10 Park Reads When set, the dispatcher continues to reissue the same descriptor to the read master when no other descriptors are buffered. This is commonly used for video frame buffering.

9 Generate EOP Used to emit an end of packet on last beat of a MM->ST transfer

8 Generate SOP Used to emit a start of packet on the first beat of a MM->ST transfer


19



Reference Design Document 7:0 Transmit Channel Used to emit a channel number during MM->ST transfers

5.1.3 Modular SGDMA CSR interface(bar0 address range 0x03010000 – 0x0301001f)


The address offset is byte address type.

Table 2-10 Modular SGDMA CSR interface registers map

Address


0x0 Status 32 Read/Clear mSGDMA STATUS register

0x4 Control 32 Read/Write mSGDMA CONTROL register 5.1.3.2 Registers description

Table 2-11 STATUS register bit definition


31：10 <reserved> N/A N/A

9 IRQ RW Set when an interrupt condition occurs. Write 1 clear

8 Stopped on Early Termination

R Set when the dispatcher is programmed to stop on early termination and when the write master is performing a packet transfer and does not receive EOP before the pre-determined amount of bytes are transferred which is set in the descriptor length field. If you do not wish to use early termination you should set the transfer length of the descriptor to 0xFFFFFFFF which will give you the maximum packet based transfer possible (early termination is always enabled for packet transfers).

7 Stopped on Error R Set when the dispatcher is programmed to stop errors and an error beat enters the write master.

6 Resetting R Set when you write to the software reset register and the SGDMA is in the middle of a reset cycle. This reset cycle is necessary to make sure there are no transfers in flight on the fabric. When this bit de-asserts you may start using the SGDMA again.


20



Reference Design Document 5 Stopped R Set when you either manually stop the SGDMA or you setup

the dispatcher to stop on errors or early termination and one of those conditions occurred. If you manually stop the SGDMA this bit will be asserted after the master completes any read or write operations that were already commencing.

4 Response Buffer Full R Set when the response buffer is full.

3 Response Buffer Empty R Set when the response buffer is empty.

2 Descriptor Buffer Full R Set when either the read or write command buffers are full.

1 Descriptor Buffer Empty

R Set when both the read and write command buffers are empty.

0 Busy R Set when the dispatcher still has commands buffered or one of the masters is still transferring data.

Table 2-12 CONTROL register bit definition


31：10 <reserved> N/A N/A

5 Stop Descriptors RW Setting this bit will stop the SGDMA dispatcher from issuing more descriptors to the read or write masters. Read the stopped status register to determine when the dispatcher has stopped issuing commands and the read and write masters are idle.

4 Global Interrupt Enable Mask

RW Setting this bit will allow interrupts to ropagate to the interrupt sender port. This mask occurs after the register logic so that interrupts are not missed when the mask is disabled.


21



Reference Design Document 3 Stop on Early

Termination RW Setting this bit will stop the SGDMA from issuing out more

read/write commands to the master modules if the write master attempts to write more data than the user specifies in the length field for packet transactions. The length field is used to limit how much data can be sent and is always enabled for packet based writes.

2 Stop on Error RW Setting this bit will stop the SGDMA from issuing more read/write commands to the master modules if an error enters the write master module sink port.

1 Reset Dispatcher RW Setting this bit will reset the registers and FIFOs of the dispatcher and master modules. Since resets can take multiple clock cycles to complete due to transfers being in flight on the fabric you should read the resetting status register to determine when a full reset cycle has completed.

0 Stop Dispatcher RW Setting this bit will stop the SGDMA in the middle of a transaction. If a read or write operation is occurring then the access will be allowed to complete. Read the stopped status register to determine when the SGDMA has stopped. After reset the dispatcher core defaults to a start mode of operation.

5.1.4 Altera Serial Flash Controller CSR interface(bar0 address range 0x06000000 – 0x0600001f)


The address offset is byte address type.

Table 2-13 Altera Serial Flash Controller CSR interface registers map

Address Offset Register Name Width R/W Description

0x0 FLASH_RD_STATUS 8 R Perform read operation on flash device status register and store the read back data.


22



Reference Design Document 0x4 FLASH_RD_SID 8 R Perform read operation to extract flash

device silicon ID and store the read back data. Only support in EPCS16 and EPCS64 flash devices.

0x8 FLASH_RD_RDID 8 R Perform read operation to extract flash device memory capacity and store the read back data.

0xc FLASH_MEM_OP 24 W To protect and erase memory

0x10 FLASH_ISR 2 RW Interrupt status register

0x14 FLASH_IMR 2 RW To mask of interrupt status register

0x18 FLASH_CHIP_SELECT 3 W Chip select values: • B’000/b’001 -chip 1 • B'010 - chip 2 • B'100 - chip 3

5.1.4.2 Registers description

5.1.4.2.1 FLASH_RD_STATUS

Table 2-14 FLASH_RD_STATUS


31:8 Reserved N/A Reserved

7:0 Read_status R

This 8 bits data contain the information from read status register operation. It keeps the information from the flash status register.

Figure 2-2 EPCQ16 and EPCQ32 Status Register Status Bits


23



Reference Design Document Figure 2-2 EPCQ64, EPCQ128, and EPCQ256 Status Register Status Bits

5.1.4.2.2 FLASH_MEM_OP

Table 2-15 FLASH_MEM_OP



23:8 Sector value W

Set the sector value of the flash device so that a particular memory sector can be erasing or protecting from erase or written. Please refer to the "Valid Sector Combination for Sector Protect and Sector Erase Command" section for more detail.

7:2 Reserved Reserved


24




1:0

Memory protect/erase operation W

• 2’b11 – Sector protect: Active-high port that executes the sector protect operation. If asserted, the IP takes the value of FLASH_MEM_OP[23:8] and writes to the FLASH status register. The status register contains the block protection bits that represent the memory sector to be protected from write or erase. • 2’b10 – Sector erase: Active-high port that executes the sector erase operation. If asserted, the IP starts erasing the memory sector on the flash device based on FLASH_MEM _OP[23:8] value. • 2’b01 – Bulk erase Active-high port that executes the bulk erase operation. If asserted, the IP performs a full-erase operation that sets all memory bits of the flash device to ‘1’, which includes the general purpose memory of the flash device. (Bulk erase is not supported in stack-die such as EPCQ512-L and EPCQ1024-L) • 2’b00 – N/A

5.1.4.2.2.1 Sector Protect

For the sector protect command, you are allowed to perform the operation on more than one sector by giving the

valid sector combination value to FLASH_MEM_OP[23:8] .

There are only 5 bits needed to provide the sector combination value. Bit 13 to bit 23 are reserved and should be set to

zero.

Table 2-16 FLASH_MEM_OP bits for Sector Value

Table 2-17 Block Protection Bits in EPCQ256 when TB Bit is Set to 0


25




Table 2-18 Block Protection Bits in EPCQ256 when TB Bit is Set to 1


26




5.1.4.2.2.2 Sector Erase

For the sector erase command, you are allowed to perform the operation on one sector at a time. Each

sector contains of 65536 bytes of data, which is equivalent to 65536 address locations. You need to provide

one sector value if you wish to erase to FLASH_MEM_OP[23:8] . For example, if you want to erase sector

127 in flash 256, you will need to assign ’b0000 0000 0111 1111 to FLASH_MEM_OP[23:8] .

Table 2-19 Number of sectors for different Flash Devices

5.1.4.2.3 FLASH_ISR

Table 2-20 FLASH_ISR


27






1 Illegal write RW 1C Indicates that a write instruction is targeting a protected sector on the flash memory. This bit is set to indicate that the IP has cancelled a write instruction.

0 Illegal erase RW 1C Indicates that an erase instruction has been set to a protected sector on the flash memory. This bit is set to indicate that the IP has cancelled the erase instruction.

5.1.4.2.4 FLASH_IMR

Table 2-21 FLASH_IMR



1 M_illegal_write RW Mask bit for illegal write interrupt • 0: The corresponding interrupt is disabled • 1: The corresponding interrupt is enabled

0 M_illegal_erase RW Mask bit for illegal erase interrupt • 0: The corresponding interrupt is disabled • 1: The corresponding interrupt is enabled


28




6 Data Flow and Clock Distribution Document the dataflow and clock distribution under this chapter. Descript the whole picture of the project here.

6.1 Data Flow

Show and descript the hardware data flow here. If necessary, can make the data flow into parts according to the

blocks and functions.

6.1.1 Software download program file to EPCQ

6.1.2 Software control Remote Update to reconfig FPGA

6.1.3 Hardware control Remote Update to reconfig FPGA

6.2 Clock Distribution

The clock distribution description, the clock domain map.


29




7 Running the example design

7.1 Set up the Hardware environment

1, Connect Altera USB-Blaster™ II cable to Stratix V GX FPGA Development Kit.

Figure 7-1 Stratix V GX FPGA Development Kit with Altera USB-Blaster™ II cable connected

2, Connect USB port of Altera USB-Blaster™ II cable to Computer II which has installed Quartus II 15.0.2 .

Figure 7-2 to Computer II with Altera USB-Blaster™ II cable connected


30




3, Power down Computer I which has installed CentOS7, (it’s better to unplug the power line of computer for safely

plugging in the Stratix V GX FPGA Development Kit to PCIe slot.) Plug in Stratix V GX FPGA Development Kit to

PCIe slot.

Figure 7-3 Computer I with Stratix V GX FPGA Development Kit Plug in

The total hardware environment is show as Figure 7-3

Figure 7-4 RSU over PCIe design example hardware environment


31



Reference Design Document 7.2 Compile the factory and application design

7.2.1 Open the Example design

1， Launch Quartus II 15.0.2 software.

2， On the File menu, click Open Project, browse to <design example install directory>, and select the rsu_over_pcie_design_example.qar Quartus II project file. Click Open.

Figure 7-5 select rsu_over_pcie_design_example.qar to open

3， Click OK to Restore Archived Project to rsu_over_pcie_design_example_restored folder.

Figure 7-6 Restore Archived Project

4， Click OK to Restore Archived Project to rsu_over_pcie_design_example_restored folder.


32



Reference Design Document 7.2.2 Compile the factory design

1， On the File menu, click Open, browse to rsu_over_pcie_design_example_restored folder, and select the top.v top-level Verilog HDL design file. Click Open.

2， On the top of top.v file, uncomment FACTORY definition, and comment APPLICATION_0, APPLICATION_1 definition.

Figure 7-7 Modify top-level file to compile factory image

3， On the File menu, click Save, to save the modification in top.v.

4， On the Processing menu, click Start Compilation.

5， After compilation, browse to rsu_over_pcie_design_example_restored\output_files folder, change the name of rsu_over_pcie_design_example.sof to factory.sof.

7.2.3 Compile the application_0 design


2， On the top of top.v file, uncomment APPLICATION_0 definition, and comment FACTORY, APPLICATION_1 definition.

Figure 7-8 Modify top-level file to compile application_0 image


33






5， After compilation, browse to rsu_over_pcie_design_example_restored\output_files folder, change the name of rsu_over_pcie_design_example.sof to application_0.sof.

7.2.4 Compile the application_1 design


2， On the top of top.v file, uncomment APPLICATION_1 definition, and comment FACTORY, APPLICATION_0 definition.

Figure 7-9 Modify top-level file to compile application_1 image




34



Reference Design Document 5， After compilation, browse to rsu_over_pcie_design_example_restored\output_files folder, change the name

of rsu_over_pcie_design_example.sof to application_1.sof.

7.3 Generate factory jic file

1， On the File menu, click Convert Programming Files.

2， Select Programming file type as JTAG Indirect Configuration File (.jic).

3， Select Configuration device as EPCQ256， select Mode as Active Serial x4.

4， Type output_files/factory.jic into the File name bar.

5， Select Create Memory Map File.

6， Select Flash Loader, Click Add Device on the right side, Select Stratix V as Device family, select 5SGXEA7K2 as Device name. Click OK.

Figure 7-10 Select Devices Dialog Box

7， Select SOF data, Click Add File on the right side, browse to output_files folder, Select factory.sof, Click Open.

Figure 7-11 Select Input Files Dialog Box


35




8， Select factory.sof, Click Properties on the right side, select Compression, Click OK.

Figure 7-12 Sof File Properties Dialog Box

9， Click Generate on bottom right of Dialog Box to generate factory.jic file.

Figure 7-13 Convert Programming File Dialog Box


36




7.4 Download Factory image to EPCQ

7.4.1 Power up Stratix V GX FPGA Development Kit

Press power button of Computer I to power up it, then Stratix V GX FPGA Development Kit would be

automatically power up via PCIe slot.

7.4.2 Changing the TCK Frequency of USB Blaster II cable

1， Click Start of Windows, type cmd into the command line.

2， Browse to the Quartus II bin directory in your path (for example, c:\altera\15.0\quartus\bin64).

3， Type the following command to change the TCK frequency:

jtagconfig --setparam 1 JtagClock 16M

Figure 7-14 Windows Command Line


37




7.4.3 Download factory.jic file via Programmer

1， Open Tools menu in Quartus II, Click Programmer.

2， Click Hardware Setup, select USB-BlasterII as currently selected hardware. Click Close.

3， Click Auto Detect， select 5SGXEA7K2, click OK.

4， Double click <none> in File column of 5SGXEA7K2, browse to output_files folder, select factory.jic,

Click Open.

5， Both select Program/Configure for 5SGXEA7K2 and EPCQ256.

6， Click Start to Program factory.jic to EPCQ256.

Figure 7-15 Download factory.jic file via Programmer

7.4.4 Configure FPGA using Factory Image.

1， After finishing programming(the progress displays “100%(Successful)”), the program file has been

downloaded to EPCQ256. Power off Computer I to power off Stratix V GX FPGA Development Kit.

2， Re-power up Computer I. Then you could see 4 red LEDs turn on on Stratix V GX FPGA

Development Kit. It indicates that the factory image has been successful configure into FPGA.

By now, when each time you power up Stratix V GX FPGA Development Kit, factory image will be load into FPGA.


38




7.5 Generate application rpd file

1， On the File menu, click Convert Programming Files.

2， Select Programming file type as JTAG Indirect Configuration File (.jic).

3， Select Configuration device as EPCQ256， select Mode as Active Serial x4.

4， Type output_files/ru.jic into the File name bar.

5， Select Create Memory Map File.

6， Select Create config data RPD(Generate ru_auto.rpd).

7， Select Flash Loader, Click Add Device on the right side, Select Stratix V as Device family, select 5SGXEA7K2 as Device name. Click OK.

Figure 7-16 Select Devices Dialog Box


39




8， Select SOF data, Click Add File on the right side, browse to output_files folder, Select factory.sof, Click Open.

9， Select factory.sof, Click Properties on the right side, select Compression, Click OK.

10， Click Add Sof Page on the right side.

11， Select SOF data of Page_2, Click Add File on the right side, browse to output_files folder, Select application_1.sof, Click Open.

12， Select application_1.sof, Click Properties on the right side, select Compression, Click OK.

13， Select SOF Data of Page_0, Click Properties on the right side, select Start of Address mode for selected pages, Type 0x0 to Start address(32-bit hexadecimal), Click OK.

Figure 7-17 SOF Data Properties Dialog Box


40




14， Select SOF Data of Page_1, Click Properties on the right side, select Start of Address mode for selected pages, Type 0xAA0000 to Start address(32-bit hexadecimal), Click OK.

15， Select SOF Data of Page_2, Click Properties on the right side, select Start of Address mode for selected pages, Type 0x1540000 to Start address(32-bit hexadecimal), Click OK.

Figure 7-18 Convert Programming File Dialog Box

16， Click Generate on bottom right of window to generate ru_auto.rpd file.(also generate ru.jic and ru.map)


41



Reference Design Document 17， Open the ru.map file from rsu_over_pcie_design_example_restored\output_files folder, you will see

the start/end address for page_0/1/2, these addresses are the index in ru_auto.rpd file, also the store addresses in EPCQ256 for each program file.

7.6 Install Linux driver

Below operations are on Computer I.

1， Copy rsu_over_pcie_linux.tar to home directory.

2， Open a Terminal for operation.

3， Type su to get root authority. And enter Password.

4， Type cd /home to browse to home directory.

5， Type tar –xvf rsu_over_pcie_linux.tar to extract the driver from .tar file.

Figure 7-19 Extract user application and driver files

6， Type cd rsu_over_pcie to enter the driver install directory.

7， Type ./install to install linux driver for this example design.

Figure 7-20 Install Linux Driver Operation


42




So far, you have successfully installed the driver for this example design, if you meet some error information when

installing driver, please check if the factory image has been successfully load into FPGA by observing if there are 4

red leds turn on. If not, Please redo the steps from Download Factory image to EPCQ.

7.7 Software flow to download application_0 image and reconfigure FPGA to

application_0 image

Before this operation, Please make sure that FPGA is running with factory image confiugraiton. If not, Please re-

power up the FPGA.

1， Using USB disk or network to copy ru_auto.rpd and ru.map files to /home/rsu_over_pcie/user in Computer I

from Computer II .

ru.map was generated together when generate rpd file at chapter 7.5.

2， In command shell, type ./run to run the user application for this example design.

The software will print out operation steps for user.

3， The software will ask “Step1 : Do you want to download the application image to FLASH”, type ‘y’ and press

Enter key.

4， Input Start address of the application image: Open ru.map, find the address information of Page_1, type

aa0000 into command shell, and press Enter key.

5， Input end address of the application image: find the address information of Page_1 from ru.map, type

1377322 into command shell, and press Enter key.

6， Input program file directory: type /home/rsu_over_pcie/user/ru_auto.rpd into command shell, and press

Enter key.

The software will auto download the data from 0xaa0000 to 0x1377322 in ru_auto.rpd into

corresponded address in EPCQ256.

7， The software will ask “Step4 : Do you want to check the application image in FLASH”, Type ‘y’ and press

Enter key.

8， The software will ask “Step5 : Do you want to reconfig FPGA”, Type ‘y’ and press Enter key.

9， After these steps, there will be 1 red leds turn on on Stratix V GX FPGA Development Kit, it indicates that

application_0 image has been configured into FPGA.

Figure 7-21 Application_0 image download and reconfigure


43




Figure 7-22 FPGA is running with Application_0 image

7.8 Software flow to return factory image


2， The software will ask “Step1 : Do you want to download the application image to FLASH”, type ‘n’ and press

Enter key.



44



Reference Design Document 4， Input Start address of the factory image: type 0 into command shell, and press Enter key.


Factory image has been configured into FPGA.

Figure 7-23 Factory Image reconfigure

Figure 7-24 FPGA is running with factory image


45



Reference Design Document 7.9 Software flow to download application_1 image and reconfigure FPGA to

application_1 image

Before this operation, Please make sure that FPGA is running with factory image confiugraiton. If not, Please re-

power up the FPGA.


The software will print out operation steps for user.

2， The software will ask “Step1 : Do you want to download the application image to FLASH”, type ‘y’ and press

Enter key.

3， Input Start address of the application image: Open ru.map, find the address information of Page_1, type

1540000 into command shell, and press Enter key.

4， Input end address of the application image: find the address information of Page_1 from ru.map, type

1e17322 into command shell, and press Enter key.

5， Input program file directory: type /home/rsu_over_pcie/user/ru_auto.rpd into command shell, and press

Enter key.

The software will auto download the data from 0xaa0000 to 0x1377322 in ru_auto.rpd into

corresponded address in EPCQ256.

6， The software will ask “Step4 : Do you want to check the application image in FLASH”, Type ‘y’ and press

Enter key.



application_0 image has been configured into FPGA.

Figure 7-25 Application_1 image download and reconfigure


46





7.10 Hardware flow to reconfigure FPGA

1. Power up Stratix V GX FPGA Development Kit. The development kit could be plugged into PCIe slot, and

using PCIe slot to supply power, or not plug into PCIe slot, use separate power supply.


47



Reference Design Document Figure 7-27 Stratix V GX FPGA Development Kit uses separate power supply

2. Reconfigure FPGA to application_0 image, Set User DIP Switch 0 to 0, User DIP switch 1 to 0, then Press

PB2, there are 1 red LEDs turn on, indicates that current FPGA is running with application_0 image.


3. Return back to factory image: Set User DIP Switch 0 to 1, User DIP switch 1 to 1, then Press PB2, there are 4

red LEDs turn on, indicates that current FPGA is running with factory image.


48



Reference Design Document Figure 7-29 FPGA is running with factory image

4. Reconfigure FPGA to application_1 image, Set User DIP Switch 0 to 1, User DIP switch 1 to 0, then Press

PB2, there are 2 red LEDs turn on, indicates that current FPGA is running with application_1 image.


7.11 Uninstall Linux driver

1,In command shell, type cd /home/rsu_over_pcie to browse to /home/rsu_over_pcie.


49



Reference Design Document 2,type ./unload to uninstall the linux driver for this example design.


50




8 Conclusion

The RSU over PCIe Design Example demonstrates a way to use PCIe transfer program file to EPCQ, and use PCIe to control Remote Update IP to achieve FPGA reconfiguration function. This reference design is running at Stratix V GX FPGA Development Kit, with a little modification on it, it could also be migrated to Arria V and Cyclone V based development kit. At this situation, the linux driver and user application don’t need to be modified. And also you could easily modify the design to accommodate your application. It will greatly reduce your development cycle and your development cost, also it is easier for your to update the FPGA application image in your product.


51




9 Revision History

shows the revision history for the Reference Design Document: RSU over PCIe Design Example.

Table 9-1 Reference Design Document Revision History

Version Date Change Summary

1.1 Jan 2016 Update RU state machine figure.

1.0 Dec 2015 First release of this Reference Design Document.

rsu over pcie design example - intel · 2020-01-20 · with different transfer sizes, and source...

Documents