interaction techniques in medical volume visualization

Interaction Techniques in Medical Volume Visualization

Bernhard Preim

Interaction Tasks and Techniques

Interaction Tasks• Exploration of original data• Data reduction• Manipulation of transfer functions• Multiplanar reformatting (MPR)

Bernhard Preim 2/68

Interaction Tasks and Techniques: Exploration of Original Data

Bernhard Preim 3/68

• “Browsing” through the slice data• Simple contrast and brightness

control via mouse movement (windowing)

• Flexible definition of slices in a corresponding visualization

• Cine mode for animation impression

• Opening and closing of a legend in the viewer

Patient information (name, date of birth, gender, Id)Image information (modality, voxel size, recording date)Coordinate and value of the selected voxelOption: more or less detailed legend

• Synchronized display of two data sets

Example: Liver CT; first data set without contrast agent, second data set with CASynchronization related to windowing parameters and the displayed layer

• Selection of the viewing direction (coronary, sagittal, axial)

Bernhard Preim 4/68


Example for legends, data: Univ. Hospital Leipzig

Bernhard Preim 5/68



Change of contrast and brightness, data: Univ. Hospital Leipzig

Bernhard Preim 6/68


Browsing through the slices (interactive or as movie)

Bernhard Preim 7/68


Synchronized illustration. Left: original data, right: filtered data

Bernhard Preim 8/68

Moving of a cross line in communicated views (Peter Hastreiter, Uni Erlangen)

Bernhard Preim 9/68

Historical model:Drawings by Dürer


Why?

Focus on certain problemsReduction of the data volume (memory requirements, rendering speed)

How?

Data selection in a certain interval (e.g. iso-surface)Definition of a volume of interest (cuboid partial volume)Subsampling of data (e.g. reduction by factor 2 in x and y direction)

Bernhard Preim 10/68

Interaction Tasks and Techniques: Data Reduction

Definition of a VOI in orthogonal views


Interaction Tasks and Techniques: Data Reduction

Transfer functions: Mapping of data onto presentation parameters (colors, gray values, transparency)

• Determine the visibility and perceptibility of structures• Parametrization of TFs is an essential interaction for the

exploration of volume data.

Challenges:• Exploration of data sets with unknown structures• Exploration of data sets with different structures of similar

intensity


Interaction Tasks and Techniques: Transfer Functions

Three volume visualizations of one CT data set with different opacity transfer functions.


Skin Bones Teeth


Requirements• Selection of predefined TFs (e.g. liver CT, lung CT)• Targeted search for suitable TFs• Correlation between adjustable parameters and characteristics

of the resulting images• Definition flexibility• Fast preview



Typical transfer functions:• Windowing• Bi-/trilevel windowing• Inverse windowing• Piecewise linear functions• Polynoms of higher degree/splines

Problem: No recognizable relation between TF characteristics and visualization



Thorax CT data set, emphasis of skeletal structures




Thorax CT data set, emphasis of blood vessels


Representation and application of TFs• Discrete representation in lookup tables• Size: e.g. 4096 entries with 32 bit (8 bit each for RGB and

alpha)• Hardware support for Lookup tables

Problem: hardware dependency w.r.t. size of color tables



Interaction Tasks and Techniques: Manipulation of Transfer Functions

Sophisticated concepts:

• Stochastic generation of TFs that may be selected by the user (multilevel iterative search), presentation as thumbnails (He et al. [1996], König et al. [2001])

• Image-based TF design (Fang et al. [1998])

• Enhanced TF

Integration of image processing filters (e.g. edge recognition)Local TF

Multidimensional TF (illustration of derived data, e.g. gradient fields, Levoy [1988])



Stochastic generation of TFs:

Iterative search process (He et al. [1996]):

1. Use of an initial TF library

2. "Mutation" of this function through a genetic algorithm (25 generations)3. Direct volume rendering (back then with VolVis 100x100 pixel, 10s)4. Subjective result analysis by the user

20/68Bernhard Preim

21/68Bernhard Preim

Source:

König et al. [2001]



Image-based TF design• Idea: Definition of the transfer function, image information

serve as context (Castro et al. [1998])

• Global histogram• Histogram along a layer• Histogram along a ray



Histogram along the orange ray as context for TF specification


Image: Dirk Bartz, Univ. Leipzig

eye ball (light)muscles (dark)


RGBAlpha and gray value Alpha TF (Peter Hastreiter, Uni Erlangen)



Composition of a TF as weighted sum of component functions

Parameters of component functions:

Sb, Sc - inner sampling points, Sa, Sd - outer sampling points



Adaptation of a trapezoid template to the local histogram of a rectangular region.


Source: Castro et al. [1998]

• “Implicit” segmentation of the white brain substance through suitable transfer functions

• Emphasis of the histogram area between the maxima of gray and white brain substance



Histogram TF (purple: opacity values, green: gray values)

Intersection of gray and white substance

• “Implicit” segmentation of the white brain substance through suitable transfer functions

• Emphasis of the histogram area between the maxima of gray and white brain substance



The Transfer Function Bake-Off, Data: Sheep heart (IEEE CG&Application 5/6 2001)• Comparison of different TF specification techniques1. ISO rendering of the segmented raw data (sheep heart) 2. Trial&Error - (20 min) with VolumePro3. Without data model - ISO automatically selected according to the maximum

gradient magnitude4. 2D TF with data model – automatic distance map, semiautom. opacity, manual color

map




Data-based TechniquesSelection of a transfer function that emphasizes the edges.Edge model:

Perfect intersection between 2 structures is "blurred" through an error function. Assumption: Blurring through an isotropic Gaussian function. -> fits to CT data well

Source: Kindlman, Durkin [1998]



Data-based Techniques: edge enhancementEdge criteria: strong gradient g, very small second derivative

h (zero crossing):

-h(v)p(v) =

g(v)

Data values along an edge, 1st and 2nd derivative



Determination of g (v) and h (v) via average determination from all first and second derivatives of all voxels with value v.Internal representation:Histogram volume H:

x-axis → f (v)y-axis → f“(v)z-axis → f´(v)

Algorithm:1. Determine min. and max. valuesfor f‘‘(v) and the maximum for f´(v). Minimum for f´(v) is assumed to be 0.

2. Fill H, whereas the values are scaled such that min and max are depicted from f´ and f´´ to 0 and 256.

32/68Bernhard Preim


Data-based Techniques: edge enhancement• What can be determined from the histogram volume?

Edge positions w.r.t. the data• What can be entered by the user?

• A selection of the "peaks" that shall be depicted• Form of the depicted peaks via boundary emphasis

function (bef)• Typical forms of bef()



Data-based Techniques: edge enhancement

Applied 2D opacity functionand volume rendering of the Visible Woman data set (TF automatically determined).• The small image indicates the 2DHistogram (intensity values vs. • Gradient magnitude)Brightness indicates frequency of.

Source: Kindlmann, Durkin (1998)



Data-based Techniques: edge enhancementComparison of edge-enhancing direct volume rendering and iso-

surface rendering

Illustration of a spiny dendrite based on microscopy data

Source: Kindlmann and Durkin 1998



Data-based Techniques: edge enhancement• Preconditions for successful application:

Existence of clear object boundariesHomogenous dataOnly little noise, no "outliers"Medicine: CT data (if CA is applied, it must be equally distributed)



Data-based Techniques• Use of a once specified TF as reference• Goal: "Re-use" of an empirically specified TF

Application: targeted illustration of a structure in a modality (e.g. aneurysms in MR)Procedure:

Selection of a reference data set Dref and a TF Tref(v)Use of the normalized histograms of the data sets H(Dref )

and H(Dstudy)Non-linear transformation t of the intensity values of Dstudy, such that H (Dstudy) ~ H (t(Dref))Hence, Tstudy (v) = Tref (v)



Data-based Techniques: reference TF• Determination of the similarity of the histograms

1. Idea: minimization of the histogram distances2. Better idea: use of the p-function by Kindlmann (considers also f‘(v) and f‘‘(v))In case of comparable data sets the p-values are similar to the histograms

Literature: Rezk-Salama et al., VMV [2000]



Data-based Techniques: Reference TF

• Visualization of blood vessels in the brain with CT angiography, left: no adaptation, middle: illustration of the first idea (histogram transformation), right: adaptation of the p-function Source: Rezk-Salama et al. [2000]



Multidimensional TFs• 1D TF: Map data onto opacity/colors• Multidimensional TFs: Use additionally derived information,

e.g. strength of the gradient or the second derivative

• Typical example: Adaptation of the opacity to the strength of the gradient, emphasis of data

intersections• Advantage: Additional degrees of freedom to generate

high-quality images• Disadvantage:High interaction costs



Multidimensional TFs• Consideration of the 2nd derivative (1st derivative of a scalar

field → vector, 2nd derivative → matrix)

• Hessian Matrix:

• Criterion (scalar value) for the 2nd derivative: largest eigenvalue of the Hessian Matrix and strength of the 2nd derivative, respectively in direction to the gradient (instead of the Hessian Matrix)



Multidimensional TFs• Gradient calculation usually via central differences

• Mapping of the gradient size to the opacity (gradient magnitude weighted transparency)



Multidimensional TFs• Volume visualization with a

gradient-dependent TF for opacity, accord. to Levoy [1988]) (Visible Human CT data set)


Interaction Tasks and Techniques: 2D Transfer Functions

Starting point for a simple specification: gradient intensity histograms. Filtering is important. Goal: accentuation of intersections.




Dense tissue and bone parts with additional gradient emphasis (green marking)Image courtesy: Hoen-Oh Shin and Benjamin King,MH Hannover [2004]



More dense soft tissue (yellow marking)Image courtesy: Hoen-Oh Shin and Benjamin King,MH Hannover [2004]



Regions with high gradients are visualized (red marking)Image courtesy: Hoen-Oh Shin and Benjamin King, MH Hannover [2004]



Source: Stölzl [2004]


Edge detector as input to define arcs


Source: Stölzl [2004]


Local TFs• Motivation: Often, global TFs enable no sufficient

differentiation• Example: Division of a lookup table into 4 segments for 4

different illustrations

Caution: Interpolation beyond segment borders is not allowed!


Source: Rezk-Salama [2002]


Blood vessels in the lung lobes are displayed with separate local TFs.



Template-based specification of 2D TFs1D templates:

2D templates:


Source: Tappenbeck [2006]

Source: Castro et al. [1998]


Template-based specification of 2D TFs:• Simplification of the interaction is even more important than in

the 1D case• Discretization in a Lookup table• Sufficient size required, at least 256x256• Applicable to arbitrary 2D domains (intensity: gradient

strength, intensity: distance to a target structure, …)



Representation of 2D TFs in a rectilinear grid as basis for discretization in an LUT



Distance-dependent TFs:• Additional entries:

Segmented target structure (tagged volume)Distance transformation w.r.t. the target

• Use of an editor for 2D TF• Sample applications:

Fade-in of safety margins around tumorsOpacity control in case of large organs




Target structure: lung surfaceSelection of interesting structures by intensity and distances



Resulting Volume Visualization (quite useless example; but appropriate illustration of the concept). Distance to the lung is used to assign different colors and opacity values.




Useful example: Emphasis of blood vessels in certain distances around a tumor via distance-dependent TFs




Interaction Tasks and Techniques: Multiplanar Reformatting

MPR illustration of an MRT data set of the head.Left: The cutting plane indicates which slice is cut out from the original data.


• Combination of the exploration of slice data with an MPR illustration



• Direct manipulative control of the MPR via Jack Manipulator



Application for vessel diagnostics • MPR is automatically oriented

orthogonal to the vessel centerline • Integrated view of cross section and

3D visualization



Local MPR:Rotation around a locally interesting structure (tumor, vessel centerline)



Anatomical Reformatting:

Idea: Use of segmentation results for a reformatting during which layers with a constant

distance to an anatomical structure (e.g. an organ surface) arise.

Procedure: - Lines of the data set are shifted against each other in such a way that voxels on the surface are located in a layer vertically to the viewing direction.

- The originally curved slices are viewed in layers → organ-specific coordinate system.

Feature: Anatomically reformatted layers show only voxels with same distances to the organ boundary.




Anatomical reformatting based on a lung lobe segmentation.Round lesions in the mediastinum.Source: Dicken et al., BVM 2003, Data: Prof. Günther (RWTH Aachen)


Summary

• Suitable interaction techniques are crucial for the practical application of medical visualization techniques.

• On the one hand, the interaction should be simple and clear. On the other hand, it should be flexible enough.

• Presets, or automatically adapting presets, are often a good basis.

• Transfer functions: Image galleries and gradient-dependent TFs are the standard.


Literature

Silvia Castro, Andreas König, Helwig Löffelmann, and Eduard Gröller, Transfer Function Specification for the Visualization of Medical Data, Technical Report, Institute of Computer Graphics and Algorithms, Vienna University of Technology, 1998, ftp://ftp.cg.tuwien.ac.at/pub/TR/98/TR-186-2-98-12Paper.ps.gz

V. Dicken, B. Wein, H. Schubert et al. Projektionsansichten zur Vereinfachung der Diagnose von multiplen Lungenrundherden in CT-Thorax-Aufnahmen, Bildverarbeitung für die Medizin, Springer, Reihe Informatik aktuell, März 2003

S. Fang, T. Biddlecome, and M. Tuceryan. Image-Based Transfer Function Design for Data Exploration in Volume Visualization. In Proc. IEEE Visualization, 1998, http://www.cs.iupui.edu/~tuceryan/research/Microscopy/vis98.pdf

T. He, L. Hong, A. Kaufman, and H. Pfister, Generation of Transfer Functions with Stochastic Search Techniques, in Proceedings of Visualization '96, October 1996, http://www.merl.com/people/pfister/pubs/vis96.pdf

Gordon Kindlmann and James W. Durkin. Semi-Automatic Generation of Transfer Functions for Direct Volume Rendering, In IEEE Symposium On Volume Visualization, 1998, http://www.cs.utah.edu/~sci/publications/vv98glk-paper.pdf


LiteratureA. König and E. Gröller. Mastering Transfer Function Specification by Using VolumePro

Technology, In Proc. Spring Conference on Computer Graphics, 2001,http://www.cg.tuwien.ac.at/research/TR/00/TR-186-2-00-07Paper.pdf

M. Levoy, Display Surfaces from Volume Data, IEEE Computer Graphics and Applications, 25 (1988) http://www-graphics.stanford.edu/papers/volume-cga88

C. Rezk-Salama, Peter Hastreiter, J. Scherer, G. Greiner: Automatic adjustment of transfer functions for 3d volume rendering. In Proc. of Vision, Modelling and Visualization, S. 357-364, 2000.

C. Rezk-Salama, Volume Rendering Techniques for General Purpose Graphics Hardware, Dissertation, Philipp-Alexander Universität Erlangen-Nürnberg

D. Stölzel. Entwurf gradientenabhängiger 2D-Transferfunktionen für die medizinische Volumenvisualisierung. Master's thesis, Dept. of Computer Science, 2004. http://www.vismd.de/lib/exe/fetch.php?media=files:master_thesis:stoelzel_2004_thesis.pdf

A. Tappenbeck, B. Preim, and V. Dicken.Distance-Based Transfer Function Design: Specification Methods and Applications. In Simulation und Visualisierung, pages 259-274. SCS-Verlag, 2006 http://www.vismd.de/lib/exe/fetch.php?media=files:volume_rendering:tappenbeck_2006_simvis.pdf


interaction techniques in medical volume visualization

Documents

exploration of volume

data selection

interaction techniques

mapping of data

ct data set

data sets example

drerinteraction tasks

axialbernhard preim