code coverage formal unreachability analysis
DESCRIPTION
Code Coverage Formal Unreachability AnalysisTRANSCRIPT
…personal…portable…connected
Code Coverage Formal Unreachability AnalysisRicardo DantasDialog Semiconductor
Metric-Driven Verification
Formal Unreachability Analysis: the Challenge
Divide & Conquer: the Solution
Configuration Script
Results
Conclusions
Dialog Semiconductor © 2015 2
Agenda
Metric-Driven Verification
Dialog Semiconductor © 2015 3
• Modern digital verification methodologies, especially constrained random ones (e.g. UVM), strongly rely on feedback metrics, which indicate how thoroughly a design has been verified. Two popular metrics are:
• Functional Coverage:• Captures the occurrence of complex scenarios during simulation, based on
coverage items provided by the verification engineer, e.g. SystemVerilogcovergroups;
• Code Coverage:• Captures parts of the design that have been exercised during simulation, i.e.
block, expression, toggle, fsm;• Requires no user inputs but is not aware of complex scenarios;• 100% code coverage is not an indication that the verification is done (complex
scenarios may be missing).
• So, why do we still care about code coverage?
Why Code Coverage?
Dialog Semiconductor © 2015 4
• 100% code coverage is typically hard to achieve and it is not an indication that the verification is done, but low code coverage figures indicate that:
• more stimuli are needed (new tests, more runs with different seeds) and/or• part of the design is unreachable.
• The so called unreachable code can be due to several reasons, including defensive code, improved readability, parameterization, legacy or dead code. For instance, a perfectly reachable code might become unreachable due to a bug upstream, therefore the importance of analysing unreachable code.
• Manual identification of unreachable code can be extremely laborious and tedious, not to mention error-prone. Therefore the use of a formal tool to automated this process is very welcome, although visual analysis of the identified unreachable code is still necessary.
A tool might pinpoint the location of unreachable code but it is still necessary to understand why it is not exercisable.
UNR Analysis with Cadence IEV
Dialog Semiconductor © 2015 5
• Cadence Incisive Enterprise Verifier (IEV) was used for the formal unreachability analysis (UNR), which, starting from an existing code coverage database, inspects the coverage holes and adds automatic properties to be formally proven. If a property proof succeeds (passes) the code is reachable and extra stimuli
needs to be provided; If a property proof fails the code is unreachable and can be waived after
review; The tool might not be able to formally prove (in reasonable time) complex logic,
thus resulting in ‘explored’ properties (neither passed nor failed).
• A database with all unreachable holes is created, which can then be merged with the original coverage db.
• Unreachable coverage holes can be inspected inside IMC before being waived.
UNR Analysis in simple steps (I)
Dialog Semiconductor © 2015 6
The following requirements are needed by IEV for the unreachability analysis:
1. the RTL design;
2. a code coverage database• the higher the coverage, the faster the unreachability analysis runs;• depending on the design complexity, a number of properties might be explored
(neither proven nor failed);
3. the hierarchical level definition where to start the analysis from• the top-level DUT or any internal block instance;
4. list of modules to be black-boxed, if any.
No constraints were added for the results presented in the next slides but there is the possibility to load them from a Tcl script. The goal was to create a robust and simple to use flow to be adopted by all digital design groups.
UNR Analysis in simple steps (II)
Dialog Semiconductor © 2015 7
irun -iev -sv -64bit -f <compilation options>-top $top-inst_top $blk_inst -covdut $top-bb_list $blk_bblist -covtest ${unr_results}_{blk_name}-covoverwrite -coverage all -covfile cov.cf -covdb $covdb -l ${unr_results}_{blk_name}.log -licqueue -input unr.tcl
IEV arguments:• -top: entity to be considered top-level design unit for formal verifier, e.g. testbench;• -inst_top: hierarchical name to be considered as the top-level for the formal build
(top-level DUT or any internal block instance);• -bb_list: file containing the design units and libraries to be blackboxed;• -covdb: simulation coverage database for coverage unreachability flow;
set engine auto_distdefine auto_dist_max 3define effort lowproveexit
Constraints or additional IEV options can be loaded via an
external script
UNR Analysis: a practical approach
Dialog Semiconductor © 2015 8
• Pros:• Quick set-up and straightforward flow;• Good results (no explored properties) can be achieved from a few minutes to a
couple of hours;• Unreachable code is highlighted when visualized with IMC and, if needed, can
be completely waived;
• Cons:• UNR analysis of a complex design, e.g. top-level, can be extremely slow
(several days), ending up in poor results due to the increasing amount of explored properties;
• As an alternative, it is possible to divide the design in smaller parts and tackle them individually. The problem with this approach is the reduced awareness of an isolated block, i.e. other blocks in its cone of influence are not considered during UNR analysis.
• The challenge:• How to maximized the results (all unreachable code found and no explored
properties) whilst keeping reasonable running times (hours, not days)?
UNR Analysis: Divide & Conquer (I)
Dialog Semiconductor © 2015 9
A divide & conquer strategy consists of breaking up the design for quicker runs and merging the results afterwards. The way the design is split will influence the results and can be divided into the following strategies:
1. Standalone blocks UNR analysis is launched for all internal blocks individually (big internal blocks can be
divided further), i.e. IEV –inst_top set to the block instance under analysis; a “hollow” top-level run is executed with all internal blocks black-boxed; running times are reasonable but individual blocks are not influenced by their neighbours
(low awareness).
2. Hierarchical blocks (sub-systems) UNR analysis is executed multiples times from the top-level, each time with one internal
block visible, whilst the others are black-boxed (top-block awareness); sub-systems can be created by letting a few blocks visible (intra-block awareness); typically slower than the previous method, as the top-level is re-analysed for every run; depending on the complexity and size of the sub-systems, running times can be kept
short or can increase drastically, ending up several explored properties. Sub-system definition requires design knowledge.
UNR Analysis: Divide & Conquer (II)
Dialog Semiconductor © 2015 10
TOP
Wrapper
Block BBlock A Block C
Block EBlock D Block F
TOP
Wrapper
Block BBlock A Block C
Block EBlock D Block F
Standalone blocks1) Individual block-level UNR analysis of all internal blocks:
A, B, C, D, E, and F;-inst_top: TOP.Wrapper.Block_<x>-bb_list: [ ]
2) Top-level analysis (starting from TOP) with everything under Wrapper black-boxed;-inst_top: TOP-bb_list: Block_A, Block_B, Block_C,
Block_D, Block_E, Block_F]
3) Merge of all results from (1) and (2).
Hierarchical blocks1) Top-level analysis (starting from TOP) with all blocks but one
under Wrapper black-boxed;-inst_top: TOP-bb_list: Block_A, Block_B, Block_C,
Block_D, Block_E, Block_F]2) Re-run (1) for all other internal blocks: A, B, C, D and E;3) Merge of all results from (1) and (2).
UNR Analysis: Divide & Conquer (III)
Dialog Semiconductor © 2015 11
Hierarchical blocks with sub-systems1) UNR analysis can run on sub-systems made of several blocks;2) Key blocks, such as a central register bank, fsm or clock generator, can be included in all sub-systems;3) Running times can increase drastically, hence different configurations of the same blocks are needed;4) Single blocks can still be analysed individually and the results merged with other runs.
TOP
Wrapper
Block_B
GlueDFT
Block_C
Block_D
Block_E
Block_F
Block_GBlock_H
Block_I
Block_A
Example: sub-system ‘green’ (blocks D/H/I):-inst_top: TOP-bb_list: [blk_A, blk_B, blk_C,
blk_D, blk_E, blk_F,blk_G, blk_H, blk_I]
Configuration script
Dialog Semiconductor © 2015 12
%ANALYSE_LIST = ( "$TOP.u_block_A" => [()],"$TOP.u_block_B" => [()],"$TOP.u_block_C" => [()],"01$TOP“ => [(@major_modules)],"02$TOP" => [(rm_element(\@all_modules,[('block_A')]))],"03$TOP" => [(rm_element(\@all_modules,[('block_B','block_C')]))]);
no black-boxes
As will be shown in the following slides, the divide & conquer approach is able to generate good results in reasonable times, but the chief difficulty is to configure all the runs, collect the results in an organized way and merge everything into a final database.
A script was developed to ease the set-up and handle all different configurations in a single run. Each individual UNR analysis is entered as an element of a hash, which contains the hierarchical level where it should start from, as well as an array of black-box modules.
Auxiliary array of modules can be created to be shared amongst the runs.
@all_modules array without modules block_B and block_Chierarchical level + optional ID(hash keys have to be unique)
Top-level results
Dialog Semiconductor © 2015 13
0,00%
0,20%
0,40%
0,60%
0,80%
1,00%
1,20%
1,40%
Standalone blocks Sub-systems Whole designTop-level 1,23% 1,37% 0,38%
Unr
each
able
cod
e (a
vera
ge o
f all
met
rics)
Top-level unreachable code
The chart illustrates the amount of unreachable code identified in the whole design using 3 different approaches:
1. Divide & conquer of standalone blocks(~8 hours);
2. Divide & conquer of sub-systems(~12 hours);
3. Single run for the whole design(~10 days).
Block by block results: linear scale
Dialog Semiconductor © 2015 14
0,00%
0,50%
1,00%
1,50%
2,00%
2,50%
3,00%
3,50%
4,00%
4,50%
Unr
each
able
cod
e (a
vera
ge o
f all
met
rics)
From the biggest (left) to the smallest (right) block
Unreachable code relative to the block sizeStandalone blocks
Sub-systems
Whole design
Block by block results: log scale
Dialog Semiconductor © 2015 15
1
2
4
8
16
32
64
128
256
512
1024
Unr
each
able
cod
e (a
vera
ge o
f all
met
rics)
From the biggest (left) to the smallest (right) block
Unreachable code relative to the block size(normalized log scale)
Standalone blocks
Sub-systems
Whole design
Running times and explored properties
Dialog Semiconductor © 2015 16
0
200
400
600
800
1000
1200
0:00:00
3:00:00
6:00:00
9:00:00
12:00:00
15:00:00
18:00:00
21:00:00
24:00:00
27:00:00
30:00:00
33:00:00
36:00:00
39:00:00
42:00:00
45:00:00
48:00:00
0 2000 4000 6000 8000 10000 12000 14000
Num
ber o
f exp
lore
d pr
oper
ties
Tota
l tim
e
Total number of holes per block/sub-system
Performance results
Time
Explored PropertiesThe number of explored properties seems
to increase drastically after a certain threshold, leading to sub-systems to be revisited and big blocks to be divided.
Conclusions
Dialog Semiconductor © 2015 17
• The advantage of doing formal unreachability analysis is to reduce the time spent investigating whether a given hole is reachable or not. The tool findings need to be reviewed by the verification team before being waived. However, the tool indication helps to focus on the right holes.
• Information about unreachable code can be useful for the block designer, as it can potentially show areas with dead code that could be removed or parts of the code that are unreachable due to a bug.
• The Divide & Conquer approach showed itself manageable and capable of identifying a good deal of unreachable code. Unless a design is so small that could be analysed thoroughly by IEV, the D&C approach represents a good compromise.
• Knowledge of the design is necessary in order to define the sub-systems with some reasonable criteria, which need to be revisited in case of several explored properties.
• Effort was set to low (10s) but the total amount of properties to be proven can be so big that an increase in the effort would increase even more the running times. It is preferable to review the sub-systems and try to break them up.
…personal…portable…connected
The power to be . . .
Dialog Semiconductor © 2015 18
Thank you