improvements in functional simulation addressing...

Improvements in Functional Simulation Addressing Challenges in Large, Distributed Industry Projects

Klaus-Dieter Schubert IBM

Schoenaicher Str. 220 71032 Boeblingen

Phone: +49-7031-162689 [email protected]

ABSTRACT The development of large servers is facing multiple challenges. The system combines a mix of design styles from custom VLSI chips to ASIC and SoC designs. The integration of hardware and firmware accumulates further challenges to the functional simulation effort. By adding more and more specialized verification solutions additional constraints are generated and the amount of required resources is increasing. To gain efficiency and to keep staffing requirements reasonable, improvements have to be put in place to integrate and standardize the different environments and tools. This paper talks about some of the enhancements that have been introduced for IBM’s server technology.

Categories and Subject Descriptors B.5.2 [Register-Transfer-Level Implementation]: Design Aids – simulation, verification

General Terms Verification, Management, Performance

Keywords Server verification, Code reusability, Simulator, Simulation Speed, Co-Simulation, Emulation, Acceleration, System Simulation,

1. SERVERS AND SOC A high end compute server is in many ways similar to a system on chip (SoC) design. This statement seems to be odd in the first place, but looking into some of the details might explain it. A SoC consists basically of a variety of design blocks, often in form of intellectual property (IP) designs, interconnects between those blocks and quite some software to achieve the full functionality. In many cases one or even more of the design blocks are embedded microprocessors or microcontroller. A high-end compute server consists of a variety of chips, interconnects to model the system structure and quite an amount of software or, as it is often called, firmware. And, by the way, some of the chips happen to be microprocessor chips. So there is a clear correlation between SoC and high-end servers as defined in table 1.

Table 1: Correlation between SoC designs and high-end

server designs

MicroprocesserChips

Embedded Mircoprocessor

Boards, Cards, …On chip BussesFirmwareEmbedded Software

SystemChip

Chip (Custom Design, ASIC, SoC)

IP Block

Compute ServerSoC Design

MicroprocesserChips

Embedded Mircoprocessor

Boards, Cards, …On chip BussesFirmwareEmbedded Software

SystemChip

Chip (Custom Design, ASIC, SoC)

IP Block

Compute ServerSoC Design

From a methodology standpoint there is also quite some similarity. In both cases the blocks respectively the chips will be verified first and independent from the complete product. The verification of interconnects requires a different set of tools and methods compared to other work. And from a software perspective it is required to verify that the hardware-software interaction is handled correctly and not just the software by itself. While this similarity holds and many ideas can be exchanged between the verification efforts of SoC and compute servers, there are of course some factors that distinguish high end server development from the SoC space. First the size and complexity is higher for the server parts. This is best illustrated by the fact that chips in a server which are corresponding to IP blocks in the SoC world often are SoC designs by itself. And while for an SoC the interconnects consist of one or more bus systems on a chip, in an high end server this could be a combination of multiple boards, cards and modules with each of them containing multiple bus systems. In the SoC world the building blocks are typically reused for multiple products. On the other side in the server world almost all chips are designed exclusively for a certain system. This leads to certain differences in the verification methodology. For servers there is a big push to use a consistent methodology from the low level verification efforts within a chip, the so called unit level, up to the overall system verification. The fact that all the pieces are designed with a certain, well-defined system in mind helps to drive a common methodology. By achieving this synergy and commonality, the movement of work from the system work to earlier levels of simulation and vice versa is enabled. However,

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. To copyotherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee. DAC 2003, June 2-6, 2003, Anaheim, California, USA. Copyright 2003 ACM 1-58113-688-9/03/0006…$5.00.

11

1.3

the fact that some of the chips are VLSI designs and others are ASIC designs leads to difficulties in the simulation environments of the server space as well. Similar as in the SoC space any kind of commonality and uniformity is very difficult to achieve, when certain designs are based on unique and incompatible methodologies. One last difference between server and SoC designs is the fact, that in a server environment all the chips are developed almost on the same schedule as the system. But this is the consequence of having designs targeted for a specific system. As a consequence each individual chip is still changing at a point in time where system simulation work has already started. Therefore a lot of the methodology and concepts used for the verification of high-end servers can be applied to SoC designs as well. Methodologies however, well suited for SoC don’t necessarily deal with the complexity and size issues of a server design point. With teams distributed globally the verification challenge is to integrate and coordinate all efforts to finally ensure that the overall system is working. The next three chapters deal with the main aspects mentioned above: reusability aspects in functional verification, simulation speed and model size and system aspects including the hardware/firmware co-simulation. 2. REUSABILITY Custom design chips like microprocessors have typically the highest complexity. To deal with the problem and the size of the designs the verification of such chips is done hierarchically, splitting the design into multiple units, starting with simulation executed by the logic design team on small chunks of logic, followed by simulation on unit, core and finally chip level. Formal verification techniques are used and are maturing slowly, but simulation is still the dominant process and requires a lot of object oriented software development to build sophisticated test benches. Today the same concepts are being used for ASIC designs as well. To judge the quality and efficiency of the simulation effort the collection and analysis of coverage data is applied. To reduce the workload explosion that is generated by the hierarchical approach, strategies are required to reuse the test benches across those hierarchy levels [1]. Each simulation environment consist of mainly 3 pieces, the driver portion to generate and apply test stimuli, a reference model to predict the expected result and some checking code to search for problems and pre-analyze them. When combining multiple environments to build larger environments the driver will be eventually replaced by hardware models. The checking code and most of the reference model will be ported over to the next level. This approach allows the effective generation of complex environments when thoroughly planned upfront. This methodology has some disadvantages for system simulation. The simulation environment code is slowing down the simulation of the large and already slow system models. The next chapter will describe this in more detail and highlight some of the tradeoffs between ease of analysis, checking capabilities, simulation speed and problem turnaround time.

The fact that a large team is starting with multiple unit environments that have to fit together later in the design cycle creates the need to have a well defined software architecture for the simulation environment in place. Within an environment where all pieces are developed in parallel using the same guidelines this is manageable. ASIC designs and especially SoC designs present a different set of problems, due to the need to combine a variety of designs, delivered by different groups, designed using different tools sets. Without being able to influence these parameters the challenge is mainly integrating all the pieces and making sure that on the interfaces everything is tested. Code reuse in such an environment is extremely difficult. This is comparable to the experience in system simulation where multiple chip environments are coming together. To solve this commonality in the design and verification methodology is required. This has lead to the concept of design for verification. Hereby, the target is to identify problems on the system level and solve or mitigate those by driving design changes or adding additional requirements and constraints to the unit simulation activities. 3. SIMULATION SPEED Simulation speed is an important factor that influences all verification plans as well. For large designs especially when considering system simulation a decent speed can only be achieved by a cycle simulation approach. While this is adding restrictions on the design style and the type of designs that can be simulated the resulting speed advantage is worth the additional effort to develop abstracted behaviors for analog designs. To improve the simulation performance even further a conscious decision has been made to establish design guidelines which allow only a restricted subset of the high level design language (HDL). This restriction enables the development of a specialized software simulator tuned for simulation speed. The disadvantage is, however, that the selected HDL subset excludes the integration of general purpose designs without significant rework. All these tricks finally lead to a raw simulation speed of up to 100 cycles per second for a full microprocessor chip (see figure 1).

��

��

��

��System

Element

Unit

SoftwareSimulator(Cycle Sim)

Co-SimHyperaccelerator /

Emulator

SystemHardware

Accelerator

4-way NAND Gates

Cyc

les

/ sec

1

10

100

1000

10000

100000

1000000

1,0E+05 1,0E+06 1,0E+07 1,0E+08

��

��

��

��System

Element

Unit

SoftwareSimulator(Cycle Sim)

Co-SimHyperaccelerator /

Emulator

SystemHardware

Accelerator

4-way NAND Gates

Cyc

les

/ sec

1

10

100

1000

10000

100000

1000000

1,0E+05 1,0E+06 1,0E+07 1,0E+08

Figure 1: Simulation speed depending on model size and simulation platform

The growing design complexity requires the generation and execution of an increasing number of test patterns. The availability of such a fast simulator is therefore a way to control the project costs, as it reduces the requirements on workstations in the batch pool.

12

However, the speed with this simulator technology drops for larger system models including multiple chips and multimillion gates down to only a few cycles per second. To overcome this issue accelerators have been used that can provide raw simulation speeds of up to 200 Kcyc/sec even for these large models. The usage of these fast accelerators generates new issues and problems that need to be solved. The benefit of multi-level simulation is the potential of reusing all the checking code from previous simulation levels. This code is now the bottleneck when improving the simulation speed further. Simulation speed is defined by the number of simulation cycles executed by the simulator in a given time period. This is reciprocal to the sum of time spent in the simulator executing the model (tmodel) and time spent in the simulation environment code (tcode) controlling it. The following formula describes the improvement in simulation speed compared to a given base. tcode,base + tmodel,base Improvement Imp = ------------------------- tcode,new + tmodel,new

In an ideal situation with an indefinite fast simulator with no time spent there (tmodel,new =0) the improvement to an existing simulation environment will be tcode,base + tmodel,base Improvement Imp = ------------------------- tcode,new

However the increase of the raw simulator speed has diminishing return if the time spent in the simulation code is not reduced as well. With no change in the simulation code the maximal improvement that can be achieved with a faster simulator for a typical chip simulation environment, where about 70% of the time is spent in the model and 30% is spent in the code, is only 3.33 based on the following formula. tmodel,base Improvement Imp = 1 + ------------- tcode,base

This is frustrating when having a raw simulation speed improvement from the simulator of up to factor 100000. There are multiple ways to solve this issue but still maintain a reasonable checking capability. The most obvious one is reducing the time spent in code by eliminating some of the checking. To recover from the reduced coverage the resulting simulation speed-up can be used to execute more tests. Another way is to move code from the software environment into the model, which is equivalent to writing pieces of the test bench in HDL. Because of the complexity of the software environment this can only be achieved for small, but highly model interactive portions of the environment. A third possibility that can be used whenever microprocessors are part of the design under test is the usage of self-checking programs. These programs can be loaded into the model at time zero and are then executed within the simulator, requiring only minimal interaction with a simulation environment.

FormalVerification

On-the-flyGen/Check

ExpectedResult check.

ApplicationExecution

Formal Tool

SoftwareSimulatorHardware

AcceleratorHyperaccelerator

EmulatorCode Simulator

��

��

��

��

��Firmware

Simulation

��

��HW/FWCosimulation

��

��

HW (Sub) System Sim

��

��

Unit/ElementSimulation

��Possible Optimal

Sim

ulat

or

Simulation Methodology

Combinations:

Figure 2: Usage of platform and checking methodology for different verification levels

One of the challenges in industrial projects is to find the best tradeoff between

1. a fast simulation speed paired with additional overhead in providing HDL environments and resulting in less coverage per cycle on one side and

2. a relatively slow simulation speed combined with extensive reuse of simulation environment code including intensive checking and therefore a high coverage per cycle on the other side.

As figure 2 illustrates most chip environments in real life tend to focus on intense checking while the simulation speed is usually good enough even without the usage of an accelerator. For system simulation applications speed is the real issue and because most of the checking is redundant to the chip environments less checking is required there. 4. HW/FW CO-SIMULATION While hardware verification has traditionally a huge focus due to the associated costs in case of a design re-spin, it is getting more and more important to ensure that also the firmware is working together with the hardware right in the beginning. In the server

FW Simulation

HW Verification Fabric. Bringup & Test

TapeOut

PowerOn

HW/FW Cosim

HW Verification Fabric. Bringup & Test

FW SimulationTime to MarketImprovement

Figure 3: Reduction in time to market by introducing

hardware / firmware co-simulation

business the main reason for this is not so much to add just another test for the hardware verification, but to guarantee that at

13

power-on time of the hardware the firmware is available with an excellent quality. This is necessary to make fast progress in bring-up and to reduce the time to market of that product (see figure 3).

With the target to verify interaction between hardware and firmware execution speed is getting the dominant factor. The first approach to solve this problem is often to invest in hardware prototyping using e.g. FPGA based solutions. Unfortunately this is very difficult in a server environment where all the chips are developed on the same schedule. Or using the analogy to SoC here, the whole system would need to be modeled as a prototype, which due to the size of the designs is just not feasible. As an alternative the usage of simulation accelerators has been chosen [2]. This can be in-house solutions as well as publicly available emulator systems, which can be used in simulation mode only. Table 2 shows some characteristics of the model build involved for this kind of simulation. Table 2: Key characteristics for system model build using an

emulator system

Simulation Speed

~ 8 hours (on fastest available IBM server)Netlist to emulator model

~ 40 GB main memoryMachine requirement

750 MBFile size

160 kcyc / sec Max. sim speed

~ 1 hour (using precompiled chips)Chips to system netlist

Compile time

35.7 Mio. 4-way NAND GatesAfter optimization

~100 Mio. 4-way NAND GatesBefore optimization

Modelsize

Simulation Speed

~ 8 hours (on fastest available IBM server)Netlist to emulator model

~ 40 GB main memoryMachine requirement

750 MBFile size

160 kcyc / sec Max. sim speed

~ 1 hour (using precompiled chips)Chips to system netlist

Compile time

35.7 Mio. 4-way NAND GatesAfter optimization

~100 Mio. 4-way NAND GatesBefore optimization

Modelsize

For these activities it is assumed that the hardware and the firmware have been verified upfront as much as possible. Therefore, almost all of the associated in-depth checking from previous simulation activities can be omitted, leading to the required simulation speed. With this environment it is possible to simulate not just the first steps of initializing the hardware and the execution of first applications, but it is also possible to start bring-up activities in simulation, debugging any hardware dependent software applications long before chips have been delivered, further improving the overall process. In this context the firmware verification activities are playing an important role. While this article is not talking about this specific topic, it has to be highlighted; without a decent firmware quality as input to the co-simulation process only very limited progress can be made. Even with a speed greater than 100000 cyc/sec, the turnaround time of problems restricts the number of problems that can be found in such an environment. So, all the “easy” problems have to be removed from the code before, which of course is true for the hardware as well. But in parallel to the co-simulation effort design for verification activities as mentioned earlier can help to reduce the verification task. The main contribution in this context will derive from the architecture of narrow hardware / software interfaces and from using a small well-defined subset of standardized interfaces.

5. SUMMARY Individual solutions exist today for most of the problems mentioned above. But adding more and more specialized solutions in parallel increases the amount of required resources which is in most cases not affordable. To gain more efficiency and to keep staffing requirements flat, effort has to be put in place to integrate the different environments and tools. This needs to be done not just on every simulation level by itself, but also across the various levels. For activities early in the development process this is usually not a problem, because the scope is narrower and the environments are owned by only a few people. For activities later in the process the complexity is increasing and more and more things need to be integrated. Unfortunately, to reduce the effort there, work has to be done already in the early stages of the process, to avoid duplication of work. To get these requirements defined early enough and to force the whole community to obey these defined rules is the real challenge in today large projects. 6. REFERENCES 1. K.-D. Schubert, "A staged verification process with inherent

test bench code reusability" at DVE 2001, Munich (http://www.eda-expos.co.uk/dve/papers-munich.htm)

2. J. Kayser, S. Koerner, and K.-D. Schubert, “Hyper-Acceleration and HW/SW Co-verification as an Essential Part of IBM eServer z900 Verification,” IBM Journal of Res. & Dev. 46, No. 4/5, 597–605 (2002)

14

improvements in functional simulation addressing...

Documents