Computer Graphics in Medicine:

From Visualization to Surgery Simulation

/Vlarkus H. Gross our vision of fully interactive and immersive have allowed the identification of individual
Swiss Federal Institute of Technology surgery simulation. anatomic substructures.
(ETH) Generally. volume data feature extraction
The Past: InsightThrough and interpretation is a paramount example of
Abstract Visualization the fruitful relationship and convergence of
3D medical imaging systems, such as X-ray graphics and vision. Here, the computer
Medicine is an extremely challenging field of
(CT), magnet resonance (ME) or nuclear vision community developed many different
research, which has been -- more than any
(SPECT) scanners, were revolutionary devel- strategies, which partly belong to the reper-
other discipline- of fundamental imponlance in
opments of modern diagnostics which con- toire of any graphics researcher working in
human existence.The variety and inherent com-
quered the field starting in the early '70s. the medical field (for Instance, John Canny's
plexity of unsolved problems has made it a
These methods gave insight into almost every famous edge detector). Conversely, many
major driving force for many natural and engi-
individual section of the human body and graphics methods are actually integral parts
neering science~.Hence, from the early days of
have saved countless lives by early diagnosis of sophisricared computer vision techniques
computer graphics the medical field has been
of tumors, heart diseases and others. (For instance, parametric polynomial sur-
one of most important application areas with faces).
The data sets, usually defined on equally
an enduring provision of exciting research chal-
spaced 3D grids, have been visualized in Many of the graphics and vision methods
lenges. Conversely.individual graphics tools and
terms of large sequences of individual slices, designed in the early days are presently well
methods have become increasinglyirreplaceable
sometimes with the help of pseudocolor.The established and support advanced applica-
in modern medicine, where medical imaging tions in medicine, such as rediatlon and oper-
search for more sophisticated ways of visual-
systems are only one prominent example.
izing this new type of volume data created an ation planning, prosthesis design, dental
The purpose of the following article is
enormous push in a new subfield of comput- treatment, education and training and many
twofold: Without claiming completeness, the ochers.
er graphics: volume rendering. Based on land-
first part gives a brief retrospective of the
mark works of Blinn and KajJya, many
fruitful relationship between computer graph- •The Present: Interactive
different algorithms have been developed for
ics and individual subareas of the medical
field.We start with early imaging and 3D visu-
the efficient and realistic rendering or vol- Exploration and
umes. The variety of approaches ranges from Telecollaboration
alization and move via interactive, collabora-
simple back-to-front rendering to sophisticat- Whereas, in the early days,graphics and visu-
tive data analysis co the emerging field of
ed lighting and shading models implemented alization algorithms were mostly designed as
surgery simulation.The second parr of the
via ray tracers.The produced images are high- preprocessors producing still images, the
paper presents a more detailed view on the
ly realistic and ubiquitous. emerging high performance computing and
interdisciplinary field of virtual and simulated
Besides direct approaches to volume ren- graphics hardware swiftly conquered the field
surgery which encompasses knowledge from
dering, the geometric reconstruction of and changed the way medical data was ana-
medicine, computer graphics, computer
anatomic structures gained increasing atten- lyzed. Nowadays, hardware-assisted real-time
vision, mechanics, material sciences, robotics
tion, since it enabled the further processing rendering of complex shapes and volumes
and numeric analysis.The author describes
and analysis of medically important features. allows for the interactive exploration and
the leading role of graphics and VR as core
In this context, data segmentation has proven analysis of huge amounts of medical data.
technologies and summarizes his personal
to be a critical preprocessing step. In addition, In addition, upcoming 3D input and output
vision of current and future research prob-
lems, which have to be pursued for realizing
higher-level volume data analysis algorithms devices allow the development of virtual real-

Figere I'A JAVAapplet for distributedcompassiondomain volumerendering F'~nsre2" User fr~rface of ~e telecaopera~onsystemKANIEDIN.Image
(htq)'/Iwww.inf.elhz.ch/personal/lipperrJEVOLVF_).Seepage I01 ~r/maLe in )CuB providedcourtesyar the CarnputerGreph~ Center,D a ~ Germany.
co/or. See page I01 for Imagein lull color.
Computer Graphics February1998 S3
 ity systems for interactive journeys through
the human body. Highly immersive interfaces
generate gorgeous illustrations and are cur-
rently being investigated as enabling tools for
the next generation of medical training and
education systems. In some other application
scenarios, VR systems are envisioned to
replace costly and dangerous invasive meth-
ods.Yet, in most cases the lack of force feed-
back turns out to be a critical issue because it
seriously diminishes the degree of realism.
In times where medical information sys-
tems and databases are commonplace in most
hospitals, networked compression, visualiza-
tion and analysis methods gain more and
more importance. Typical requirements
encompass fast searching and browsing of
large CT or MR databases as well as scalabili-
ty to the network performance and computa-
tional power of the clients. As an example,
Figure I presents a JAVA applet for rendering
remotely located volume data sets instanta-
neously from a highly compressed file format.
Another vast subfield of graphics in medi-
cine is collaborative diagnosis and telepres-
ence. For instance in radiology, physicians
often face the problem of having instanta-
neous access to a specialist who is located
somewhere remote. Whereas in the past
images were exchanged as hardcopies via sur-
face mail, upcoming high-speed digital net-
w o r k s open the d o o r for real-time
distributed and collaborative diagnosis.
Medical telecooperation projects and systems
are currently supported by various telecom-
munications companies, and in some cases
they are already available as commercial prod-
ucts. Figure 2 shows a user interface layout of
a typical telecooperation system for radiolo-
gists.
Figure 3: Physics-based modeling of facial s0~ t/ssue:a) Presurgical facial shape profile, b) Postsurgical profile, c) Postsurgical frontal view.Data source:Visible Human Project,Courtesy
National Library of Medicine. See page 101 for image in full co/or.
54 February1998Computer Graphics
Besides high-end CT and MR data sets,
graphics researchers are discovering the lower
end of medical imaging systems. Here, ultrason-
ic devices dominate the scene and various
approaches for rendering and reconstruction of
ultrasonic data sets can be found in contempo-
rary literature. Although noise and alignment
problems make a robust segmentation and fea-
ture extraction much more difficult, it is
expected that some of those methods will be
part of next generation's ultrasonic systems.
Other research groups design high-end
telepresence systems, which enable the sur-
geon to perform telesurgery using haptic and
robotics interfaces. Additionally, augmented
reality systems have been used for interactive
diagnosis and medical check-ups. Images taken
from a camera on the physician's head are
superimposed with coincidentally recorded
ultrasonic scans and are composed in a head-
mounted display. Real-time projections of 3D
reconstruction of individual anatomic features
allow the location and visual inspection of the
inner part of the human body.
Furthermore, 3D hardcopy devices, such as
stereolithographs, have been widely and suc-
cessfully used in medicine. Current research
activities include the automatic computation
and reproduction of prostheses or missing
bones, based on geometric reconstruction. In
particular, the symmetry of the human anato-
my is exploited to compute missing pieces,
which are then reproduced by milling
machines and implanted by surgeons.
Additionally, 3D hardcopies help the study of
anatomic deviations or malformations in the
approaches of complex surgical procedures,
such as in facial surgery.
Besides mere geometric reconstruction of
anatomic structures, graphics researchers
developed models to simulate the physics of
the human body. Presently, one of the most
challenging and attractive subfields here is the
fast and accurate computation of the mechan-
ical behavior of soft tissue, which is an essen-
tial prerequisite for the vast and mostly
unexplored area of virtual and simulated
surgery. Although we are still far from a
mature computational model, some existing
soft-tissue models perform surprisingly well
and therefore can be considered a first step
towards more sophisticated approaches. An
example from facial surgery simulation is
given in figure 3, where pre- and postsurgical
facial shapes of a case study are presented for
a lower jaw bone repositioning. In this partic-
ular application, highly realistic images of the
post-surgical appearance of the patient's face
are of enormous importance to relax
patient's fear.
The Future:Virtual
Operation and Surgery
Simulation
Undoubtedly, future key applications of graph-
ics in the medical field include operation plan-
ning and surgery simulation. Here, we foresee
fully immersive real-time simulation environ-
ments in which surgeons can learn, plan and
rehearse complex operations using individual
patient models and being supported by
sophisticated input devices, such as virtual
scalpels. The design of these types of simula-
tion environments is a highly interdisciplinary
project which requires input from a variety of
disciplines including graphics, vision, robotics,
numeric analysis, mechanics and material sci-
ences, hardware design and, needless to say,
medicine. However, unlike any other disci-
pline, computer graphics will be paramount,
since it will take the role of the system archi-
tect.
T h e c o n c e p t u a l c o m p o n e n t s of an
advanced surgery simulation environment can
be summarized as follows: In a first step, raw
data sets have co be acquired by highly accu-
rate 3D medical imaging o r vision systems.
Subsequent preprocessing steps e x t r a c t
anatomic substructures and create geometric
models of the patient au~ributed with respec-
tive material parameters, stemming from an
appropriate material database.A sophisticated
modeler allows the surgeon to modify the
geometry and the topology of individual parts
of the derived model while simulating cuts,
bone repositioning, transplants etc. Force
feedback as a function o f the underlying
material has to be computed and interpolated
to meet the high update rates which are nec-
essary to beat the temporal resolution of the
tactile sensory channel. Tissue volume forces
and deformation fields as well as collision
detection must be computed in real time,
since they convey the parameters For visual
and force feedback. In essence, Fast approxi-
mations of the underlying differential equa-
tions have to be Found. Although for some
applications, such as facial surgery simulation,
more expensive and accurate solution strate-
gies could compute the exact deformation
fields in batch mode, the design of appropri-
ate real-time engines remains the most chal-
lenging part of the simulation. As an option,
appropriate rendering algorithms might gen-
erate photorealistic still images of the patient.
D a t a A c q u i s i t i o n and Analysis
Since CT and MR scanners still have limited
resolution, highly accurate 3D surface data
acquisition is fundamental For some applica-
tion scenarios. In this context, fast and robust
3D range finding methods are necessary co
satisfy the resolution constraints imposed by
the simulation environment. Furthermore,
color and texture samples must be recorded
co equip subsequent rendering methods.
Especially when using both volume and sur-
Face data, manifold registration and alignment
problems arise, including surface-surface, sur-
face-volume and volume-volume registration.
Robust (semi)automatic methods are desir-
able.
Since the early days of medical imaging, seg-
mentation and feature extraction have lost
n o n e o f t h e i r i m p o r t a n c e , and despite
decades of research, there is no robust and
fully automatic method in sight- For matching
problems, semiautomatic strategies might be
an appropriate alternative and the collabora-
tive effort of computer graphics and comput-
er vision researchers is necessary t o
accomplish this goal.
T h e H u m a n C o m p u t e r Interface
The design of a sophisticated natural human
computer interface is certainly one of the
keys to successful simulation.The illusion of a
full medical working environment can only be
created with the help of advanced virtual
reality hardware. C o n t e m p o r a r y o u t p u t
devices, such as head-mounted displays,
C a v e s , V i s d o m e s and Workbenches are
promising tools to study the behavior and
acceptance of medical users. However, we
foresee smart, small, lightweight and very high
resolution eyeglass displays as the ultimate
devices for mediating visual information to
the surgeon.We believe that future display
technology will provide a new generation of
sophisticated solutions.
Much more important and fundamental For
any surgeon than the display itself is the pro-
vision of highly accurate tactile and force
feedback information. Therefore, we require
highly accurate haptic interfaces, which cap-
t u r e the responses o f human tissue t o
mechanical stimuli. One of the major prob-
lems, however, is that the haptic device must
be t h o r o u g h l y tailored to the underlying
application. In facial surgery simulation, for
instance, completely different setups are
required than in laparoscopy. Moreover, com-
plex surgical procedures usually e m p l o y
extensive sets of individual mechanical tools.
Although various haptic interfaces are avail-
able in research labs or on the market and
enable first experimental steps into the right
direction, we are still far from convenient
solutions. All in all, the design of individual
force feedback devices is strongly influenced
by the application context and by the under-
lying physics. OptimJ7.arion to users' demands
requires multiple iterations and design cycles.
Therefore, successful solutions can only be
d e v e l o p e d in a t i g h t c o l l a b o r a t i o n of
researchers from medicine, robotics and
computer graphics.
Mechanics and N u m e r i c Analysis
The differential equations governing any volu-
metric tissue deformation have their roots in
mechanics. In the past, linear volumetric strain
and deFormation models have been extensive-
ly investigated and are widely available for
various kinds o f m a t e r i a l simulation. In
surgery simulation, however, large strains and
deformations occur and the material tensors
become non-constant_The resulting phenom-
ena are highly nonlinear and extremely diffi-
cult t o m o d e l . O t h e r effects relate to
compressibility of human soft tissue.The high
percentage of liquid makes it almost incom-
pressible; local tissue forces generated during
surgery force liquid to stream out and to dis-
t o r t the mechanical behavior of the material.
Besides mathematical models, appropriate
material parameter databases are of great
importance in surgery simulation. Moduli of
elasticity or non-linearity are usually functions
of age, sex, e t h n i c g r o u p and o t h e r s .
Unfortunately, it is e x t r e m e l y difficult to
obtain the desired parameters and extensive
experimental research has to be pursued.
Here, we need the help of material scientists
co provide us with appropriate sets of para-
meters and interpolation models For individ-
ual patients and tissue types.
The strategies for optimal solutions of par-
tial differential equations are critically depen-
dent on the complexity of the underlying
mechanical model. In particular, in order to
achieve real-time response we have co bal-
ance mathematical accuracy versus computa-
t i o n a l e f f o r t . N u m e r i c analysts have
developed various classes of solvers, where
FEM is only one prominent example. Upon
simplifying the physics, appropriate solution
strategies have to be designed in close coop-
eration with numeric analysts. Here, hierarchy,
progression and localization might be some of
the key words to success.
T h e P a r ~ n o u n t Role o f C o m p u t e r
Graphics
The particular attractiveness of highly com-
plex surgery simulation environments for us
lies in the versatility and depth of individual
research problems touching o r covering
almost every important subfield of computer
graphics. Therefore, we finally illuminate the
paramount role of graphics in surgery simula-
tion and briefly sketch some of the major
research challenges:
• Modeling of geometry and topolo~. The basis
of any sophisticated surgery simulation sys-
tem is the efficient mathematical description
of the underlying geometry and topology.
Here, most of the model has co be described
in a full volumetric settin~ Sophisti,~ted dis-
cretJzations and approximation models are
essential preprocessing steps For advanced
numeric procedures, which compute tissue
deformation. Conformity and polynomial
degree of individual elements strongly affect
the complexity and accuracy of numeric
solutions. Therefore we have to adapt our
models to subsequent numerical strategies
where piecewise linear approximations are
mostly insufficient.As a consequence, our
challenge is to develop new generations of
higher order volumetric approximations,
which do not impose unnecessary topologi-
cal restrictions on our model.
• Ed/ting of complex structures:With appropri-
ate approximation methods in place, anoth-
er task is to design p o w e r f u l e d i t o r s
allowing the user to modify the geometry
ComputerGraphlcs February1998 SS
 and topology of anatomic structures. Again
mere surface editing is certainly not suffi-
cient for cutting and repositioning of indi-
vidual pieces of soft tissue. Functions such
as a zoom into substructures are highly
desirable and enable a surgeon to operate
on different scales in the full volumetric
setting. Therefore, elaborate data struc-
tures have to be designed giving efficient
access to, and maintaining individual primi-
tives of the underlying approximation.
Furthermore, these data types will have to
maintain all necessary parameters of the
physics-based models, which in turn must
interact with the editor in real time.
• Simplification and adaptation of physical mod-
els: One of our core challenges in surgery
simulation is to understand and simplify the
physics of the simulation process.The quest
for real-time update rates in visual and
mechanical feedback forces us to develop
sophisticated solutions, which have to bal-
ance computational complexity with accu-
racy. To this end, we have to thoroughly
investigate and analyse the governing equa-
tions and to study their error bounds.
Based on a profound knowledge and
observation of the physical phenomena,
new approximate solutions must be
designed. Many different strategies and
parameters have to be investigated and
evaluated against each other, such as finite
element versus finite difference, flat versus
hierarchy, local versus global or polynomial
degree versus spatial resolution. Required
update rates for visual and force feedback
rendering might be decoupled from the
simulation engine and appropriate tempo-
ral i n t e r p o l a t i o n could compute the
desired in-betweens. In addition we will
have to consider next generations graphics
hardware to accelerate the envisioned
algorithms.
56 February1998Computer Graphics
• Image generation: Generally, fast rendering
algorithms are irreplaceable in surgery sim-
ulation. However, w i t h the design of
advanced volumetric approximations new
types of possibly unstructured graphics
primitives will enter the scene and appro-
priate hardware support will be highly
desirable. Thus both surface and volume
rendering of higher order primitives will be
of great interest.The required visual quality,
however, depends highly on the application
context. The mostly lubricant surfaces of
the interior of human bodies, for instance,
are highly specular and will ask for hard-
ware support for more sophisticated shad-
ing models. F u r t h e r m o r e , bump,
displacement and texture databases or
procedural models for human skin and soft
tissue have to be created. Here, multipass
methods might be a promising approach.
Apart from the real-time constraint, some
applications require extremely sophisticat-
ed illumination models. In facial surgery
simulation, for instance, the rendering of
photorealistically looking images might be
computed off-line by raytracing of the high-
er order primitives envisioned earlier. Here
we need advanced reflection and scattering
models for facial skin, which will also vary
as a function of age, sex and other parame-
ters. In addition, high quality rendering of
human hair is still a widely open issue.
• System Design and Layout: Computer graph-
ics researchers will undoubtedly be the
principal designers of future surgery simu-
lation systems. Thus, besides the mere
development, tuning and tailoring of indi-
vidual algorithms graphics research will also
have to cover overall systems layout and
optimisation. In particular, global design
issues, such as the interactions of editor,
rendering, simulation engine and force
feedback, are of fundamental importance in
highly interconnected, complex systems
and must also be considered by graphics
researchers.
Summary
In summary, from the earliest days, the med-
ical field has been one of the most appealing
areas for computer graphics research. Being
the chief engineer of next generation's med-
ical training and simulation systems, computer
graphics will help to improve our present
health system.Thus it will provide a significant
contribution to our society -- and to modern
human life.
Markus H. Gross is full Professor at the
Computer Science Department of the Swiss
Federal Institute of Technology (ETH) in
Z~irich. He received a degree in Electrical and
Computer Engineering in 1986 and a Ph.D. in
Computer Graphics and Image Analysis in
1989, both from the U n i v e r s i t y of
Saarbr~icken, Germany. From 1990 to 1994,
he was with the Computer Graphics Center
in Darmstadt, where he established and
directed the Visual Computing Group. His
current research interests include physics
based and multiresolution methods for graph-
ics with applications in the medical field.
Markus H.Gross
Department of Computer Science
Swiss Federal Insitute of Technology (ETH)
CH 8092 Zurich, Switzerland
Tel: +41-1-632-7114
Fax:+41 -1-632- I 172
Email:grossm@inf.ethz.ch
httpdlwww.inf.ethz.chldepartrnent/ISIcgl
 ,. ...

FROM THE ED IT O R

Gordon Cameron related note, the real-time column is on rata.

SOFTIMA6E, Inc. tion for this issue,but will return in Hay.
Next time around, we investicate an area
When we look at the current state of comput- that has had tremendous impact on society
er graphics it is easy to take so much for and the computer graphics world over the
granted. By doing so, we forget the remarkable past decades - - computer taming.To coincide
flurry of research and development undertak- with the 25th anniversary, an earlyAugust issue
en over the past quarter century (and more) will act as a special history document tied
which has led us to a present where comput- specifically to the conference.
er-generated imagery mesmerizes us at the The world of computer graphics continues
cinema, visualizations aid in our hospitals, to evolve at a stardinI pace. I hope you enjoy
graphical tools aid in the design of our homes this trip down memory lane and look to the
and vehicles and countless other applications future.
of computer graphics impact our daily life. I~e
been involved in computers and graphics in
Gordon Cameron
one way or another for around 16 years (since
Software Development
my I~rst dabblings at the age of 13 on a neigh-
SOFTIMAGE, Inc.
bour's Apple II Europlus, and implementation of
3510 boul.$t-Laurent
a painfully basic drawing program in 0.5K on a
Suite 400
ZXtil). It is amazing to me how Par we have
Montreal,Quebec H2X 2V2
come since '82, and equally amazing how we
Canada
got to that stage from preRy much nothing
Tel:+ I-514-845-1636 ex¢_3445
over the preceding few decades.
Fmc+ I-514-845-5676
This year marks the official 25th birthday
En~l: Iordon_carnemn@s/araph.arg
of the 51GGRAPH conference (Ed/t0r's note:
although the organization is older m see Carl
Machover's column on page 25). To kick off
the celebrations, I asked FranTois 5illion (a
pioneer in graphics, particularly in the fields of
rendering and global illumination) if he would
like to guest edit an issue which would ask a
selection of experts from a cross section of
the community to offer their musings on the
past few computer graphic decades, as well as
look into their personal crystal balls to give
us their opinion on what lies in store m a
'~orward-looking retrospective:' Much to my
delight, Francois decided co rise to the chal.
lenge and collect and collate for us a fantastic
selection of articles written by some of the
leading researchers from a wide range of
computer graphics disciplines. My thanks go
out to all the authors and FranTois for pre-
senting a fascinating view on the world of
computer graphics - - past, present and possi-
ble future!
This February issue also sees the debut of a
new regular education column, as well as a stu-
dent gallery to showcase the works of those
studying in educational establishments around
the globe. Many thanks and best wishes ro the
new additions to the Computer Graphicscolum-
nist family, Rosalee Wolfe, Jodi Giroux-Lang,
Lynn Pocock and Karen Sullivan. On a column-

Computer Graphi,'q February1998 3