Documente Academic
Documente Profesional
Documente Cultură
Semantic
Classification
of Byzantine Icons
Paraskevi Tzouveli, Nikos Simou, Giorgios Stamou, and Stefanos Kollias,
National Technical University of Athens
R
ecent advances in Web technologies that ensure quick, effective infor-
mation distribution have created the ideal terrain for spreading vari-
This system uses nessing the publication of a huge amount of cultural information on the Web.
fuzzy description In the past, people gathered cultural in- To meet this need, researchers have pro-
formation from physical objects (such as posed using image analysis methods (for
logics and patterns books, sculptures, statues, and paintings). some examples, see the sidebar on the next
Now, digital collections have become com- page). However, these methods use domain
to automatically mon practice for most cultural-content knowledge for only low-level analysis. Also,
providers. two main difficulties have arisen in using
determine the sacred However, as the amount of the Web’s cul- image analysis to automate annotation and
tural content grows, search and retrieval classification of cultural digital assets. The
figure depicted procedures for that content become increas- first is the failure of semantic image seg-
ingly difficult. Moreover, Web users need mentation and image analysis algorithms in
in an icon. more efficient ways to access huge amounts some real-world conditions. This is due to
of content. So, researchers have proposed so- the high variability of content and environ-
phisticated browsing and viewing technolo- mental parameters (luminance and so on),
gies, raising the need for detailed metadata which makes the problem complex. The sec-
that effectively describes the cultural con- ond difficulty is the extensive and vague na-
tent. Several annotation standards have been ture of domain knowledge (at least for some
developed and implemented, and Semantic cultural content), which complicates formal
Web technologies provide a solution for the knowledge representation and reasoning.
semantic description of collections on the However, some cultural domains are ap-
Web.1 Unfortunately, the semantic annota- propriate for automatic analysis and classi-
tion of cultural content is time consuming fication methods. Byzantine icon art is one
and expensive, making it one of the main of them. The predefined image content and
difficulties of cultural-content publication. the low variability of the image characteris-
Therefore, the need for automatic or semi tics support the successful application of im-
automatic analysis and classification of cul- age analysis methods. Consequently, we’ve
tural assets has emerged. developed a system that exploits traditional
T
he July 2008 IEEE Signal Processing Magazine pre- of brushwork characteristics that art historians use for au-
sented some particularly interesting applications of thentication. 3 Finally, Howard Leung and his colleagues re-
image analysis to the cultural-heritage domain. Ale- ported on the preservation of ancient Chinese calligraphy. 4
jandro Ribes and his colleagues described problems con-
cerning the design of multispectral cameras as well as the
analysis of the Mona Lisa in the context of the digitization
References
of paintings.1 Anna Pelagotti and her colleagues proposed 1. A. Ribes et al., “Studying That Smile,” IEEE Signal Processing
a novel multispectral digital-imaging technique for clus- Magazine, vol. 25, no. 4, 2008, pp. 14–26.
2. A. Pelagotti et al., “Multispectral Imaging of Paintings,” IEEE
tering image regions in a set of images containing similar
Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 27–37.
characteristics when exposed to electromagnetic radia- 3. C.R. Johnson et al., “Image Processing for Artist Identifica-
tion. 2 Their technique provides material localization and tion,” IEEE Signal Processing Magazine, vol. 25, no. 4, 2008,
identification over a painting’s entire surface. C. Richard pp. 37–48.
Johnson and his colleagues described how three research 4. H. Leung, S.T.S. Wong, and H.H.S. Ip, “In the Name of Art,” IEEE
groups developed systems that identify artists on the basis Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 49–54.
iconographic guidelines to automatically analyze icons. figures of the Christian faith are defined by specific facial
Our system detects a set of concepts and properties de- characteristics. For example, according to Dionysios, Jesus
picted in the icon, according to which it recognizes a sa- is illustrated as a young figure with long, straight, thick,
cred figure’s face, thus providing semantic classification. black hair and a short, straight, thick, black beard.
According to these rules, the area in which the sacred
Byzantine Iconography figure will be painted is separated into equal segments. The
Byzantine art refers to the art of Eastern Orthodox states first segment contains the head, focusing on the eyes and
that were concurrent with the Byzantine Empire. Certain facial expression. This segment contains the sacred fig-
artistic traditions born in the Byzantine Empire, particu- ure’s most distinguishable features—for example, hair and
larly regarding icon painting and church architecture, have beard style. The remaining segments contain the sacred fig-
continued in Greece, Bulgaria, Russia, and other Eastern ure’s body, focusing on the hands and arms. The body can
Orthodox countries up to the present. The icons usually be depicted in different poses—for example, standing, sit-
depict sacred figures from the Christian faith, such as Je- ting, from the head to the waist, and from the head to the
sus, the Virgin Mary, the apostles, and the saints. thorax.
At the beginning of the 17th century, iconography man- The head should be elliptical. The major axis is along
uals represented the most important source of inspiration the vertical part of the head and is equal to four times the
for icon painters. The 16th-century monk Dionysios from height of the nose (H). (According to Dionysios, the nose
Fourna is credited as the author of the prototypical man- serves as the metric for producing harmonious facial and
ual, Interpretation of the Byzantine Art. 2 In it, he defines body proportions.) The minor axis is along the horizontal
the rules and iconographic patterns that painters still use and is equal to 3H. The head segment can be separated into
to create icons. four equal vertical sections of height H. The hair is in the
Byzantine iconography follows a unique convention of first part, the forehead is in the second, the nose is in the
painting. The artistic language of Byzantine painters is third, and the area between the mustache and the chin is in
characterized by apparent simplicity, overemphasized flat- the fourth.
ness, unreal and symbolic colors, lack of perspective, and A Byzantine painter applies four base colors in layers on
strange proportions. The sacred figures are set beyond real the face segment. The darking is a dark color for the base
time and real space through the use of gold backgrounds. of the face segment and the darkest shadows. The proplas-
The most important figure in an icon is depicted frontally, mos is a less-dark color for lighter shadows. The sarkoma
focusing on the eyes, facial expression, and hands. Clarity is for flesh, and the lightening is for highlights, containing
is the rule, not only in describing shapes but also in arrang- a good amount of white and being light enough to show off
ing figures in balanced compositions so that the narrative’s against the base color. The absolute value of these colors
actions are clear. The figures have large eyes, enlarged ears, differs from icon to icon.
long, thin noses, and small mouths, each painted following
specific rules. From Art Manual to Semantic Representation
When depicting various figures, the artist is obliged to Although the knowledge in Dionysios’s manual concerns
observe the manual’s rules regarding posture, hair and vague concepts such as “long hair,” “young face,” and so
beard style, apparel, and other attributes. Specific sacred on, it’s quite strict and formally described. Consequently,
Face detection
extracts the face
and its basic weights and model eyes to deter-
components mine an eye’s presence and position
(nose and eyes). (indicated by the red and green dots
in the second image in the face de-
tection box of Figure 2).
Face component Base color After eye localization, the mod-
detection analysis extracts
detects the most the base color
ule detects the nose. Let H denote the
important components layers (darking, distance between the eyes and con-
of the face (beard, proplasmos. sequently the nose’s length. Accord-
moustache, cheek, sarkoma, and ing to Dionysios, the nose starts on a
forehead, and hair, lightening).
respectively). small horizontal line whose center is
a distance of H/5 below the midpoint
between the eyes. We use a Hough
s1 Image description transform to find this line. Therefore,
Feature extraction s: Face we determine the nose’s ending line
automatically extracts s2 s1: Hair
features such as
by applying a Hough transform at a
s2: Forehead
distance, color, s3 s3: Cheek distance H down vertically from the
and length for s4: Mustache starting line.
each face s5: Beard The computation of H leads to the
component and s4 (s, s1): hasComponent
for some ...
detection of the face’s other main com-
segments s5 Length(s1) = 2.8*H ponents—the hair, forehead, cheek,
of the face ... mustache, and beard—because they are
components. darking(s1) = 88% well defined in Dionysios’s manual.
...
The module detects the four base
colors by applying an Otsu segmen-
Semantic Formal assertions tation algorithm 6 to a restricted area
interpretation s: Face of the face segment (see the base
gives meaning to BelowEarsHair AboveEarsHair s1: Hair color analysis box in Figure 2), de-
Membership
aid of geometric information from the eyes and the sur- Feature extraction. This module automatically ex-
rounding area. 5 We use a training set of sacred-figure tracts features from the segmented parts, 8 providing
eyes to create vector maps, and we use principal com- additional information regarding each part’s length,
ponents analysis to derive eigenvectors for these maps. color, shape, and texture. This information constitutes
For a candidate face segment, the algorithm 5 projects the characteristic values of the features, providing an
the length and its angle maps on the spaces spanned image description set (see the image-description sec-
by the eigenvectors that were evaluated during train- tion of the feature extraction box in Figure 2).
ing. In that way, we use the similarity of the projection Dionysios assigns four features to hair:
Figure 3. A reasoning example. The extracted information from image analysis constitutes the assertional component (ABox) of
the knowledge base; the terminological component (TBox) is defined on the basis of Fourna’s specification. These components
form the input to the fuzzy reasoning engine, which infers information about the icon.
tions, so we use the fuzzy intersection operator for the mini- eyes and nose. The base-color-analysis submodule defined
mum. So, the GLB of segment s (the face) in YoungMansFace the color model used by the icon’s artist, which the fea-
is 0.82, whereas the GLB of segment s in JesusFace is 0.78. ture extraction module later used. Using the nose’s height,
In other words, the extracted features indicate that the face which can be easily determined, and taking into account
depicted is YoungMansFace to a degree of 0.82 and is Je- the manual’s rules, the face-component-detection submod-
susFace to a degree of 0.78. ule estimated the positions of the background and fore-
ground pixels for every part of the face. These positions
Results constituted the input to the graph-cut algorithm. This sub-
We evaluated our system on a database, provided by the module achieved 96 percent accuracy.
Mount Sinai Foundation in Greece, containing 2,000 digi- The feature extraction module extracted features for
tized Byzantine icons dating back to the 13th century. The every icon segment. Then, using the fuzzy partitions, the
icons depict 50 different characters; according to Diony- semantic-interpretation module interpreted each segment’s
sios, each character has specific facial features that makes specific features and properties and the relationship of face
him or her distinguishable. parts. Thus, that module determined each feature’s degree
Evaluation of the Byzantine-icon-analysis subsystem pro- of membership in a specific class, thereby creating an image
duced promising results. The subsystem’s mean response description.
time was approximately 15 seconds on a typical PC. To evaluate overall system performance, we used
In the semantic-segmentation module, the face detection precision and recall. Table 1 (on the next page) pres-
submodule reached 80 percent accuracy. In most cases, the ents results for 20 of the sacred-figure classes. The
failure occurred in icons with a destroyed face area. If the more distinctive facial features the sacred figure in a
submodule detected the face, it almost always detected the class contained, the better our method performed (for
No. of images
Class No. of icons in class No. of images classified correctly classified Precision Recall
example, for the Jesus class). Conversely, our method sify paintings on the basis of unmixed colors and dense
performed worse on figures with similar facial features, textures.
such as women.
Acknowledgments
Our research has been partly supported by the MORFES (extraction
and modeling of figure in Byzantine paintings of Mount Sinai monu-
AdvertiSER Information
March/April 2009 • IEEE Intelligent Systems
Advertising Personnel Advertising Sales Midwest/Southwest Product:
Marion Delaney Representatives Darcy Giovingo
Phone: +1 847 498 4520 US East
IEEE Media,
Fax: +1 847 498 5911 Joseph M. Donnelly
Advertising Dir.
Email: dg.ieeemedia@ieee.org Phone: +1 732 526 7119
Phone: +1 415 863 4717 Recruitment:
Email: jmd.ieeemedia@ieee.org
Email: md.ieeemedia@
Mid Atlantic Northwest/Southern CA
ieee.org
Lisa Rinaldo Tim Matteson US Central
Phone: +1 732 772 0160 Phone: +1 310 836 4064 Darcy Giovingo
Marian Anderson
Fax: +1 732 772 0164 Fax: +1 310 836 4067 Phone: +1 847 498 4520
Sr. Advertising
Email: lr.ieeemedia@ Email: tm.ieeemedia@ Fax: +1 847 498 5911
Coordinator
ieee.org ieee.org Email: dg.ieeemedia@ieee.org
Phone: +1 714 821 8380
Fax: +1 714 821 4010 New England Japan US West
Email: manderson@ John Restchack Tim Matteson Lynne Stickrod
computer.org Phone: +1 212 419 7578 Phone: +1 310 836 4064 Phone: +1 415 931 9782
Fax: +1 212 419 7589 Fax: +1 310 836 4067 Fax: +1 415 931 9782
Sandy Brown Email: j.restchack@ieee.org Email: tm.ieeemedia@ieee.org Email: ls.ieeemedia@ieee.org
Sr. Business
Development Mgr. Southeast Europe Europe
Phone: +1 714 821 8380 Thomas M. Flynn Hilary Turnbull Sven Anacker
Fax: +1 714 821 4010 Phone: +1 770 645 2944 Phone: +44 1875 825700 Phone: +49 202 27169 11
Email: sb.ieeemedia@ Fax: +1 770 993 4423 Fax: +44 1875 825701 Fax: +49 202 27169 20
ieee.org Email: flynntom@ Email: impress@ Email: sanacker@
mindspring.com impressmedia.com intermediapartners.de