università degli studi di palermo - eurasip
TRANSCRIPT
Università degli Studi di Palermo
Dottorato di Ricerca in Ingegneria Informatica
DIPARTIMENTO DI INGEGNERIA INFORMATICA
AUTOMATIC DETECTION, CLASSIFICATION AND RESTORATION OF DEFECTS IN HISTORICAL IMAGES
Dottorando di Ricerca: Ing. Giuseppe Mazzola
Tutor: Ch.mo Prof. Edoardo Ardizzone
Coordinatore: Ch.mo Prof. Salvatore Gaglio
Tesi di Dottorato di Ricerca in Ingegneria Informatica – XIX Ciclo SSD: ING-INF/05
Acknowledgements
I would like to thank my tutor prof. Edoardo Ardizzone, who supervised my work
during these three years and introduced me into the world of digital restoration.
I wish also to thank dr. Haris Dindo, who provided me the guidelines and supported me
to achieve the goals of my research.
I wish to thank Alinari Archives in Florence, which let me use their material for my
study, without which my research could not be possible.
I wish to thank Italian Ministry of Education, University and Research, which funded
my work in the last year.
I wish to thank all the partners of the project, for their discussions on several digital
restoration issues.
Finally, I wish to thank all the co-authors of my papers, for the fruitful collaboration
and their contributes to my scientific publications.
Index Index ................................................................................................................................ 2 Introduction...................................................................................................................... 4 Chapter 1............................................................................................................................ Management and Preservation of Cultural Heritage by using ICT technologies ............ 6
1.1 Cultural Heritage and ICT Techniques ............................................................ 6 1.1.1 Management................................................................................................. 7 1.1.2 Diagnosis...................................................................................................... 9 1.1.3 Restoration ................................................................................................... 9 1.1.4 Safeguard ................................................................................................... 10 1.1.5 Communication-Fruition............................................................................ 12 1.1.6 Education ................................................................................................... 14 1.2 European Projects .......................................................................................... 14 1.2.1 MINERVA................................................................................................. 14 1.2.2 MICHAEL ................................................................................................. 16 1.2.3 Prestospace................................................................................................. 19 1.3 Conclusions.................................................................................................... 21
Chapter 2............................................................................................................................ A Knowledge-Based Model to support Digital Restoration.......................................... 23
2.1 Goals of the Project........................................................................................ 23 2.2 Related Works................................................................................................ 25 2.3 The Approach................................................................................................ 26 2.4 The Proposed Restoration Model vs. Classical Model .................................. 28 2.5 The Knowledge Base ..................................................................................... 30 2.6 The Prototypal Restoration Tool.................................................................... 35 2.6.1 Guiding the User through the Restoration Process .................................... 36 2.6.2 Making Visual Queries to the DB............................................................. 39 2.7 Conclusions.................................................................................................... 40 Acknowledgements.................................................................................................... 41
Chapter 3............................................................................................................................ The Defect Taxonomy ................................................................................................... 42
1.3 Origin-based defect taxonomy....................................................................... 43 1.3.1 Mechanical (physical) damages................................................................. 43 1.3.2 Chemical damages ..................................................................................... 44 1.3.3 Limitations and problems .......................................................................... 45 1.4 MPEG-7 Visual Descriptors .......................................................................... 46 1.4.1 Color Descriptors ....................................................................................... 46 1.4.2 Texture Descriptors.................................................................................... 48 1.4.3 Shape Descriptors ...................................................................................... 49 1.5 Dual taxonomy............................................................................................... 49 1.5.1 Description Ability..................................................................................... 54 1.6 Conclusions.................................................................................................... 55
Chapter 4............................................................................................................................ Classification – A Case Study: Foxing .......................................................................... 56
4.1 Introduction and related works ...................................................................... 56 4.2 Foxing spots ................................................................................................... 57
3
4.3 Foxing detection............................................................................................. 57 4.4 Feature extraction........................................................................................... 58 4.5 Content-based foxing retrieval....................................................................... 60 4.6 Experimental results....................................................................................... 62 4.7 Conclusions and future works........................................................................ 63 Acknowledgements........................................................................................................
Chapter 5........................................................................................................................ 64 Detection and Removal of Quasi-Horizontal Scratches ................................................ 64
5.1 Introduction and related works ...................................................................... 64 5.2 Scratch Features ............................................................................................. 65 5.3 The Proposed Method.................................................................................... 65 5.4 Scratch detection............................................................................................ 66 5.5 Restoration phase ........................................................................................... 67 5.5.1 Direction Estimation .................................................................................. 68 5.5.2 Pixel filling................................................................................................. 69 5.6 Experimental Results ..................................................................................... 70 5.7 Conclusions.................................................................................................... 71 Acknowledgements.................................................................................................... 72
Chapter 6............................................................................................................................ Texture Synthesis Restoration within the Bit-Plane Representation ............................. 73
6.1 Introduction and Related Works .................................................................... 73 6.2 The bit-plane representation .......................................................................... 75 6.3 Restoration methods....................................................................................... 77 6.4 The conditional random generation method .................................................. 77 6.4.1 Information Analysis ................................................................................. 78 6.4.2 Reconstruction ........................................................................................... 78 6.4.3 Computational Cost ................................................................................... 81 6.5 The best matching method ............................................................................. 82 6.6 Experimental results....................................................................................... 82 6.7 Remarks and limitations ................................................................................ 85 6.8 Conclusions and future works........................................................................ 87
Conclusions.................................................................................................................... 88 References...................................................................................................................... 90
Introduction
Historical photos are significant attestations of the inheritance of the past. Since
Photography is an art that is more than 150 years old, more and more diffuse are the
photographic archives all over the world. Nevertheless, time and bad preservation
corrupts physical supports, and many important historical documents risk to be ruined
and their content lost. Therefore solutions must be implemented to preserve their state
and to recover damaged information.
This PhD thesis proposes a general methodology, and several applicative solutions, to
handle these problems, by means of digitization and digital restoration.
The purpose is to create a useful tool to support non-expert users in the restoration
process of damaged historical images.
The content of this thesis is outlined as follows:
Chapter 1 gives an overview on the problems related to management and preservation
of cultural repositories, and discusses about possible technological solutions that can
help cultural institutions in their activities. Some examples of significant European
projects are given.
Chapter 2 presents the key problems related to the purpose of this work. It briefly
describes the Italian scientific project, in the context of which my research work has
been carrying out. A restoration model is proposed, and compared to the classical
model. Then it discusses the methodology that has been proposed, which consists in a
knowledge-based model for image restoration. Finally a software restoration tool. to
guide users through the restoration process, is presented.
Chapter 3 presents a taxonomy of typical defects by which damaged old photos are
affected. A dual taxonomy is proposed, designed to catalogue defects of both old
printed photos and their digitized copies.
The next two chapters present two applications that have been implemented as solutions
for specific damages.
5
Chapter 4 presents a classification application for a particular damage of the digital
defect taxonomy. Foxing spots are analyzed, and a set of low level descriptors, specific
for this damage, is proposed. Then a classification tool, based on these descriptors, is
presented.
Chapter 5 discusses about detection and the removal of quasi-horizontal scratches in
still images. The test dataset is composed by digitized aerial photos of the Sicilian
territory, which has been damaged by manual inspection of the photo negatives with a
mechanical device.
In chapter 6 a new methodology is proposed to handle the problem of restoration of
greyscale textured images. Two texture synthesis based approaches are presented: a
conditional random generation process, which is designed for random textured images,
and a best matching method, that works better with periodic textures.
A final section summarizes obtained results.
Chapter 1
Management and Preservation of Cultural Heritage by using ICT
technologies
This chapter introduces the problems related to the management and the preservation of
Cultural Heritage, and the possible solutions offered by ICT applications. It focuses on
the “Italian case”, both for the richness and the variety of Italian Culture, and for
political and organizational problems of Italian cultural institutions in planning effective
strategies. In the second part of the chapter three European projects, which handle these
problems are presented: MINERVA, MICHAEL and Prestospace. MINERVA and
MICHAEL deal with the problem of promoting cultural repositories by means of
Internet and communication networks. Prestospace aims to preserve old audiovisual
archives by digitization.
1.1 Cultural Heritage and ICT Techniques
The term “Cultural Heritage” includes not only cultural artefacts but all the traditions
and habits we inherited from the past: archaeological sites, museums, handcrafts with
acknowledged historical-artistic value, buildings with historical relevance, monuments,
churches, paintings, furniture, objects of worship, fabrics, but also cities, urban
structures, folk and food customs, traditional recipes, religious customs.
“Cultural” means that the object:
- has a story;
- has been dignified by time;
- its story is correlated with today, and has a value, to discriminate cultural objects
from very old things;
Chapter 1
7
- his value has been acknowledged in some way.
According to the Italian Law, can be also considered as “cultural” objects :
- means of conveyance, if more than 75 years old;
- scientific and technical tools and goods, if more than 50 years old;
- photos and movies, if more than 25 years old.
ICT tools and techniques are essential to preserve this huge heritage of the past. ICT
techniques involves all the technologies implemented with the development of
computer science and communication networks. The following classification of the
possible uses of ICT techniques for the preservation of the Cultural Heritage has been
proposed by Paolini[1]. It is based on the “destination of the use” rather than on the
used techniques, so that same techniques are mentioned for different goals:
- Management: for a more effective and cheaper management of the Cultural
Heritage;
- Diagnosis: to monitor the state of conservation (or degradation) of the objects;
- Restoration: to support the restoration activity;
- Safeguard: to protect Heritage against criminal acts and natural disasters;
- Communication: to communicate to a larger number of people the relevance of
cultural repositories, both to spread knowledge and to promote the tourism;
- Education: to help young students in studying history;
- Fruition: to enjoy with a more effective experience Cultural Heritage
Note that some of these fields are related to each other: a good archive of “cultural
objects” can be used both for the management and as a base for a safeguard application.
However there is no causal relationship between these applications: a good archive is
not a warranty of an effective safeguard of the repository.
1.1.1 Management
Each body (company, government, private or public organization), that has the
responsibility of some cultural objects, must care about their management. It has to
know which are these objects, their general features, location, state of conservation, etc.
Essentially cultural heritage management is based on database applications: all the
objects must be inventoried, and the corresponding files organized in a useful system.
Chapter 1
8
Even if this method is effective for business application, it is more complex to apply to
cultural objects. In an archaeological site, for example, which is rich of thousands of
objects, is difficult to detect which are the object to inventory. One step is to decide the
level of detail for the inventory, that is if objects must be considered separately or in
groups: whole buildings or each wall, if it has a significant fresco painted on it? a
funeral equipment or each single object in the set? A management application for a
typical archaeological site must handle as well a huge temple or an amphora fragment.
Less problem for the management of an art museum, because is simple to identify
which are the objects to catalogue. Therefore the development of the right management
application must be, obviously, content-dependant.
The main goals for cataloguing a cultural heritage repository are:
- the “warehouse”: to know which are the objects in the inventory and if they are
there;
- support to study and research: to help scholars or researchers to study, using the
inventory to know “where” the objects are;
- support to safeguard: to plan the safeguard actions, to know which object could be
missed for by theft or a flood;
- rate the repository: as in a commercial warehouse, giving a value to each object in
the repository.
The management of a cultural heritage repository, even if it seems similar with respect
of commercial warehouses, is much difficult. Every well-organized company make the
inventory of its warehouse, while most part of the cultural heritage repository is not or
bad inventoried, because of the use of obsolete data representation models. Moreover,
even if they are required for bureaucratic attainments, existing inventories are
considered not useful for working applications, and not used in most case. It is
impossible in practice to create applications for rating a cultural repository, because of
the objective difficulty to estimate the value of cultural objects.
In the last years, in Italy, the MiBAC (Ministry for Arts and Culture) tried to
standardize settings (different data for different objects) for a cultural repository[2], but
these directives were not acknowledged by cultural organizations. Furthermore,
Chapter 1
9
inventoried data files are heterogeneous and unreliable. The goal to have a well-
structured inventory for the Italian Cultural Heritage is far away.
An example of a working application for cultural object management is geo-location, a
technology to detect the geographical coordinates of the heritage on earth. This is, at the
moment, a very expensive technique, and it cannot be considered as “the” solution for
the problem of cataloguing objects in cultural repositories.
1.1.2 Diagnosis
The main problem in the diagnosis of cultural heritage is to get accurate information
about the state of preservation of the object (painting, sculpture, building, etc.), without
damaging it. No invasive diagnosis or physical removal is admitted for objects which
have a cultural value. It is not possible to take, for example, a piece of plaster or to
remove some material from the object to analyze it.
Non invasive techniques are based on X-rays (to discriminate different levels of
transparency), laser (to sense the surface of an object), sound waves (to receive different
vibrations emitted, as a reaction to a sound stimulus, by different parts of a surface), etc.
ICT techniques are less relevant than these ones for diagnosis application. Sensed data
can be post-processed, to enhance information, or archived in a database for future
comparisons.
A very interesting possibility is the use of remote sensing techniques for diagnosis:
sensors are spread around the building or the site to analyze, to monitor physical or
environment conditions. Collected data are than sent to a processing center, to be
processed and analysed by expert researchers.
1.1.3 Restoration
Restoration means to modify the state of an object, to bring it to a better or a more
correct conservation state. As in diagnosis applications, the main problem is the using
of invasive techniques.
There are three possible goals for a physical restoration:
- to preserve the actual state in the best way, e.g. preserving the plaster of a fresco;
- to restore the optimum state, e.g. brightening the colors of a painting;
Chapter 1
10
- to recreate original conditions, e.g. reconstructing lacking parts of a painting.
ICT techniques are applied with a twofold goal:
- support with simulations, to show the final result before the object is physically
restored
- virtual reconstruction, to show the original aspect of the object, without modifying
its state.
Virtual restoration of historical images is the topic of this PhD thesis, and will be
analyzed in depth in the next chapters.
Section 1.2.3 discusses about Prestospace, a European project, which aim to preserve
and restore old archives with audiovisual contents.
1.1.4 Safeguard
Cultural heritage safeguard is one of the priority in Italy, due to the richness of its
heritage, but it is a hard task, because of its fragmentary distribution all over the
country. Objects with artistic or historic relevance can be found both in well-known
sites (museums, public palaces, national archaeological sites) and in small or private
sites (churches, private palaces, private collections or country museums).
To protect cultural repositories from natural disasters, the main application based on
ICT techniques is the “Risk Map”, a database which chart all the repositories with a
potential risk to be damaged by calamities, and the corresponding actions to do for a
first intervention. The “Risk Map” is still a work-in-progress: too many repositories, too
many high-risk areas. A hierarchical information system could be the solution. The
Ministry for Arts and Culture would be responsible for the main national cultural
repositories, while other bodies would chart those in districts or in towns. There no
technological problems to make a well-structured information system. Many are the
solutions implemented by software companies. Main problems are political (agreement
between involved bodies, distribution of the costs, etc.) and organizational
(coordination of the actions, standardization of the solutions, unified data representation
models, etc).
The second point is to prevent and punish criminal acts against the cultural heritage.
Problems are the same of those discussed above: repositories are spread all over the
Chapter 1
11
country and an effective monitoring is hard, without well-organized infrastructures. ICT
techniques can help providing solutions to build a national distributed database, to
catalogue data, respecting accessibility and security specifications. At the moment a
database, to catalogue stolen objects, still exists, to help the police force in its work, but
it is still incomplete. In most cases it lacks of the description or the picture of the object,
so it cannot be very helpful to track down the booty.
Video-surveillance networks can be a solution for the security of cultural sites. This is
one of the most active fields in ICT research. The goal is to monitor an environment in
order to detect suspicious behaviours of visiting people.
The third point is copyright protection. Old pictures, movies and songs must be
considered as cultural objects as well archaeological sites. Analogue media inevitably
lose quality with time and with each copy generation, while its digital version can be
duplicated and used without losing quality, and last (almost) forever. The advent of
Internet and file sharing tools had promoted the illegal distribution of copyrighted
digital files (digital piracy). Digital Rights Management (DRM) techniques are
technologies used by copyright holders to limit the unsupervised diffusion of digital
media data. Using DRM, audio and video files are encoded and encrypted to allow:
- a more difficult diffusion of copyrighted data
- limitations for the users:
o time-limitations or purpose limitations
o pre-defined limitations (license, password, etc.)
The earliest example of a DRM application was Content Scrambling System (CSS),
proposed by the DVD Forum on 1996, which used a simple encryption algorithm, and
required a license key provided by device manufacturers. Much and much resources has
been spent in this field. Today, as movies, audio files can be bought at on-line
multimedia shops, with DRM limitations such as the times the file can be played and
the type device into which it is played ( e.g. Apple iPod DRM ).
Digital watermarking is another well-known technique, used to add identification
information into a digital flow. It is used above all for still images and movies.
According to the purposes, watermarking techniques can be divided into:
- visible, e.g a tv logo;
Chapter 1
12
- hidden, more difficult to remove.
Its main applications[3] are:
- Fingerprinting: to trace illegal duplications;
- Copy protection: to control digital playing and recording devices;
- Authentication: to verify data authenticity;
- Invisible annotations: to enrich information of a digital data flow (e.g. author and
date in a picture)
- Copyright protection: copyright data can be inserted in the digital document.
1.1.5 Communication-Fruition
Communication and fruition applications are often developed for the same goals, so in
this section they are considered as synonymous terms.
Communication is probably the most important topic in managing cultural heritage.
There are two opposite opinions about communicating cultural heritage:
- Communicate culture to much people as possible, to find funds to support cultural
repository management;
- Culture is propriety of scientific community, and to make profit with culture
desecrate its value.
Cultural objects have values if they have a story to tell, therefore communication is the
best way to promote culture. On the other hand, promoting culture to support it is often
the excuse to exploit it and to make profit. Moreover there are two different approaches
in the world for cultural heritage communication:
- North European and North American cultural institutions, which are above all
private foundations, claimed themselves as “Culture diffusion sites”, and developed
communication strategies to make culture closer to families, schools, young people, etc.
Therefore their activities are very popular, can attract a lot of funds, can supply services
with very limited costs and with no economic intervention from public institutions;
- In the rest of the world very few institutions planned effective strategies to
communicate culture to every social layers. In Italy, for example, the lack of cultural
communication is due to bad-planned future projects, wrong allocation of economic
Chapter 1
13
resources, and a too bureaucratic structure of cultural institutions. This situation is often
covered with spectacular events, which are not useful for a long-term planning.
Two examples of European projects which deal with the problems of cultural
communication are discussed in section 1.2.1 and 1.2.2.
New technologies can help communication in several ways:
- Interactive multimedia applications: developed in 80’s and 90’s, today are ousted by
Web. They are designed above all for school applications or for entertainment. The
main advantage is direct economic returns but there are many problems in distribution.
- Web sites: the most diffuse solution to communicate and promote culture, because
their easy accessibility from all over the world. Web sites are easy to crate, that often
leads to low-quality products in contents and implementation. There are neither direct
economic returns, nor distribution costs. In North Europe and North America, Web sites
typically evolved from institutional sites (to present organization activities), to
permanent repository sites (to describe inventories), temporary event sites(to attract
visitors), virtual museums, educational games and school applications via Web. In Italy
cultural institutions, as discussed above, don’t trust in communication technologies,
therefore promotion by cultural websites is not often used for a long-term strategy, and
some the few attempts ended with an unhappy end (see the www.italia.it portal case).
- Application for PDA and mobile devices: devices available in more and more
cultural sites in the very last years. These technologies are defeating the diffidence of
cultural organizations towards the introduction of electronic devices in sites with
historical objects (art museums and archaeological sites). Small size and (relatively)
low cost are the strong points of this not invasive technology. They will soon substitute
audio-guide and brochure, at least in larger sites, to supply information and details to
visitors, but also supplying further services as interactivity, chats, educational games,
online shopping, etc.
- Virtual reconstructions: useful also for restoration and diagnosis goals, most of
virtual reconstruction applications has been used for communication and entertainment.
They are used to enrich user fruition of the cultural heritage: to see the original aspect
of damaged buildings, paintings or artefacts; to immerse visitors into a virtual
reconstructed historical environment, etc.
Chapter 1
14
1.1.6 Education
E-learning applications to support people in their studies. Much more diffused in the
open-minded North America than in “Old Europe”, because of the European traditional
view of education. Applications supply educational subjects, documents and images to
students, and educational games or chat-bots can be used by teachers to enrich and to
test students’ learning.
1.2 European Projects
This section discusses about three European projects concerning the use of new ICT
technologies for the preservation and the promotion of Cultural Heritage. MINERVA
and MICHAEL concerns the standardization of digitization of cultural repositories, to
promote and diffuse European Cultural Heritage via Internet. Prestospace deals with the
preservation and restoration of digitized old audiovisual archives.
1.2.1 MINERVA
MINERVA (Ministerial Network for Valorising Activities in Digitisation) is a network
of Member States' Ministries to coordinate activities for the digitisation of cultural and
scientific content, in order to create an European common platform, recommendations
and guidelines about digitisation, metadata, long-term accessibility and preservation.
This network aims to co-ordinate national programmes, and to integrate its work within
national digitisation activities.
It also establishes contacts with other European countries, international organisations,
associations, networks, international and national projects involved in this sector. The
project is supervised by a Committee, to identify and integrate best practices in a pan-
European framework, to follow the Lund action plan.
The Lund Action Plan[4], based on the Lund Principles, was established by European
Commission on 4 April 2001 in Lund(Sweden). It is an agenda of actions to be carried
out by Member States and the EU Commission in order to implement a framework for
digitisation coordination in Europe.
Chapter 1
15
The MINERVA project has been funded by EU in the context of the IST (Information
Society Technology) Programme and the Fifth Framework Programme covering
research and technological development.
The key goals are:
- to coordinate strategies and policies of the partners for digitization of cultural
content;
- to provide a European dimension to policies and programmes of the partners;
- to define, exchange and disseminate good practices across EU;
- to support the development of national and international inventories of cultural and
scientific content:
- to promote accessibility and fruition of cultural Heritage
In its first phase, in 2001, MINERVA has started a collaboration, in the Cultural
Heritage digitization field, between the Ministries of the UE. It worked onto two levels,
political and technological. The political activity has permitted a tight collaboration
between Member States through the participation of high level institutions such Art and
Culture Ministries and the European Committee. From this point of view, MINERVA
has given visibility to national activities, has promoted best practice interchange, and
has spread knowledge of European policies and programmes to national and local
organizations.
Figure 1.1 The MINERVA process model
Chapter 1
16
Fig. 1.1 shows the MINERVA process model. It is a TOP-DOWN approach:
institutions make projects and standards; project partners create digital collections from
physical ones; digital collections have access to services and products.
Technological activity dealt with the creation of a European common platform shared
between the Member States, built using recommendations and guidelines for
digitization, with the goal of communicating and enjoying Cultural and Scientific
Heritage(archives, libraries, museums) by Internet.
The MINERVA main activities concerns cultural web quality, system interoperability
(key point to create portals and digital libraries), the digitization process and its cost,
the diffusion of best practices along Europe. It produced several tools and guidelines,
and many handbooks useful to detect and design tools for a high-quality cultural web
communication[5][6][7].
Digitization is discussed in [8], which provide useful information to develop policies
concerning Cultural Heritage digitization in Europe, according to the content
accessibility rules standardized by EU. What is proposed has been approved by the
NRG (National Representatives Group), made up of experts from the 15 Member States
and 10 new members.
Many satellite projects spread out from MINERVA activity, the most important of
which are:
- MINERVA eC, which will end in 2008, and it has the goal to implement
MINERVA results, to support NRG to realize the Dynamic Action Plan (DAP);
- MICHAEL, which is discussed in the next sub-section.
1.2.2 MICHAEL
MICHAEL (Multilingual Inventory of Cultural Heritage in Europe) is a European
project[9] that aims to make digital collections of Europe’s museums, libraries and
archives accessible from all over the world.
It has been funded by EU Committee in the context of the eTen programme, which have
the goal to promote the development of intra-European services, based on
communication networks. Started in 2004 with three Member States (France, Italy and
UK) it evolved in MICHAELplus in 2007, and it has been extended to 15 European
Chapter 1
17
countries. MICHAEL implemented a multilingual cultural portal to let people find
information and enjoy digital cultural heritage using Internet.
The political and organizational purposes of the project are:
- To make cultural heritage accessible to European citizens: young and old people,
handicapped, researchers, and all the possible users;
- To coordinate and integrate national cultural initiatives;
- To agree and implement common standards and platforms, and technical models;
- To identify best practice and Major centres;
- To achieve goals of the Lund Action Plan and to lead the NRG of the MINERVA
project;
- A European cultural heritage inventory, accessible for everyone and which can
offer to wide set of trans-European resources;
- A growing numbers of national catalogues which will use metadata, models,
services based on a common platform;
- A sustainable management of the project and an effective strategy to have more
funds from national political institutions;
- A methodology, with a flexible technical platform, to add new instances of the
MICHAEL model, to increase contents and user database.
Practical objective is to use Internet, web, broadband and new technologies:
- To Enrich value of European cultural resources;
- To organize interoperable national initiatives;
- To improve the accessibility to public resources;
- To extend national cultural activities to a trans-European dimension;
- To help people in studying, life-long learning, e-Learning, home studying;
- To promote cultural tourism;
- To help the “education for everyone”, for a social and economic inclusion;
- To make available cultural contents to creative companies, artists, designers, etc.
- To increase the numbers of the portals which use best practices and international
standards for metadata, models, etc.
Technical results are:
Chapter 1
18
- National inventories based on a common metadata, data and service model, and a
multilingual thesaurus;
- National portals working on a open-source common technical platform;
- A sustainable, flexible and extensible model, based on XML;
- An open-source solution, based on Apache Tomcat, Cocoon, XtoGen, etc.
- A methodology and a model easy to implement and replicate in other countries.
The MICHAEL platform is made of two modules, which work together to provide data
management and publishing service:
- A production module: to create, modify, import, and manage records describing
features of digitized cultural heritage; these services are available using standard Web
browser. Data is stored using XML database, based on MICHAEL data model;
- A publication module: which provides an intuitive interface for end-users to browse
cultural contents by Web; this module uses a XML search and display engine, which
can be customised to create interfaces
Figure 1.2 the MICHAEL process model
The MICHAEL platform is based on MICHAEL data model, which is derived from
MINERVA project and is relates to RSLP collection description schema and the Dublin
Chapter 1
19
Core1 Metadata Initiative on collection description. The platform can be deployed on
several system based on Java technologies.
Fig. 1.2 shows the MICHAEL process model. It is centred on Digital Collections.
Institutions are responsible both for projects, digital collections, and physical
collections which are the source for digital versions. Digital collections are created by
projects which creates also services, which makes available digital collections.
1.2.3 Prestospace
PrestoSpace (Preservation Towards Storage and Access) project[10] is funded by the
European Union's IST programme. It evolved from the Presto (2000-02) project, which
aimed to develop the technological means to transfer programme material in broadcast
archives to digital media, in order to reduce the cost of the digitisation process.
Prestospace started February 2004, to involve not only broadcasters, but also of all the
other European institutions that store audiovisual materials, such as film museums,
university collections, industry archives and national heritage collections. It is
coordinated by Institut National de l’Audiovisuel (INA), France, and involves three of
the biggest audiovisual archive owners in Europe (INA, RAI and BBC), and many other
technological partners.
The overall goal of the PrestoSpace Project is to develop and launch actual facilities and
services in the following fields (which are summarised in fig.1.3):
- Preservation: a fast, affordable datacine, a contactless playback tool for audio disks,
an automated audio preservation tool, an automated video preservation tool, a manual
tape condition assessment tool, an information system for preservation management
- Restoration: a restoration management tool, a defect analysis and description
infrastructure, a disk-to-disk real-time restoration tool, a digital film restoration
software tool, a set of high-level restoration algorithms
- Storage and Archive Management: a web-guide and software tool for planning
storage for audiovisual preservation, a guide and software tool for business-case
1 Dublin Core defined collections as: any aggregation of physical or digital items; Collections of physical items; collections of digital surrogates of physical items; collections of 'born-digital' items and catalogues of such collections.
Chapter 1
20
planning for audiovisual preservation, a logistics and quality insurance system for
audiovisual preservation.
- Metadata, Delivery and Access: a semi-automatic description tool, an export system
for delivering preservation results to medium and large archives, a Turnkey system for
delivering preservation results to small archives.
Figure 1.3 Prestospace process model
Prestospace proposes the “preservation factory” approach, to provide low cost
standardized services to all kinds of collections owners to manage and allow access to
their assets.
The main motivation for the factory approach to preservation is to minimise loss in
time, money, training and equipment. The choice is between an item-by-item approach
and dealing with the overall content of the collection, which can be a too general
approach. Fig.1.4 shows that ad hoc processes are slow and expensive, while industrial
processes reduce cost and time, but are often too general. The preservation factory is a
systematic approach to the whole problem, which tries to maximize throughput (most
items saved per hour and per Euro), putting resources where they give the best return.
Chapter 1
21
Figure 1.4 Industrialisation process for archives
The basic elements of the approach are:
- knowledge of the whole collection (a collection map);
- automation of the actual workflow;
- trial to keep the automation effective.
Automation applications are the simple use of bar-codes, but also full robotics for tape
handling and signal monitoring. The important result is that a 50% savings can be made
by sensible engineering of a specific process (workflow) to deal with preservation work,
and then advanced technologies can give another 50% savings.
Technical results was shown in the Final Prestospace Workshop in Rome, January 21-
22, 2008 and include:
- audio, film and video scanning tools;
- tools, guidelines, and services to manage the migration process and the storage;
- audio, video, and film restoration tools;
- Publication Platform and the Turnkey System for rendering audiovisual contents
accessible.
1.3 Conclusions
ICT techniques are absolutely necessary tools for organizations, companies and
research unities who aims to work in the field of cultural heritage preservation.
This chapter intended to give an overview of the problems related to this field, and the
possible ICT techniques which can be useful to handle these problems. In most cases
Chapter 1
22
problems are more organizational than technical. A well-made planning of the activities
is the necessary first step to achieve significant results.
Mediterranean culture is richer than that of the rest of the world. On the other hand
North-European and American countries succeed in managing and promoting their
cultural heritage much better than Mediterranean countries.
Some examples of European projects which achieved interesting cultural and scientific
results are discussed in the last part of the chapter.
My research work has been carrying out in the context of an Italian project, dealing
with cultural heritage preservation by means of digital restoration of historical images.
The project involves partners from several Italian universities and some technological
consultants. It will be briefly introduced in the first part of the next chapter.
Chapter 2
A Knowledge-Based Model to support Digital Restoration
Historical photos must be considered as cultural objects as well museums, paintings,
archaeological sites. But time and careless preservation corrupt physical supports,
therefore solutions must be found in order to preserve and recover lost information of
damaged images.
My research work has been developing in the context of the FIRB project entitled “A
knowledge based model for digital restoration and enhancement of images concerning
archaeological and monumental heritage of Mediterranean coast”, which aims to
develop a useful tool to guide users in the digital restoration process of damaged
images. It involves the “Dipartimento di Ingegneria Informatica” dell’Università degli
Studi di Palermo and several Italian research partners, and the Engineering Ingegneria
Informatica s.p.a and Alinari Photographic Archives in Florence as consultant partners.
Furthermore, Alinari provided us useful material for the scientific research, that consists
in a database of high resolution, coloured, black and white historical images since 1840.
The first part of this chapter describes the goals of the project, and the attended results.
In the second part of the chapter the proposed approach to achieve these goals is
discussed, and the obtained results are presented.
2.1 Goals of the Project
The aim of the FIRB project, approved in 2005, is the development of novel
methodologies for the description and restoration of degradation typologies occurring in
digitized copies of historical photos. More precisely, the project aims:
1. to develop methodologies and algorithms which are able to completely recover the
original image attenuating aging effects caused by a bad preservation, or abrasions
Chapter 2
24
of the film material. In particular a limited class of defects is analysed, e.g.
scratches, small missing data regions, blotches, noise and colour rendering
distortion. The choice of a limited class of degradations is supported by the attempt
of tackling some unsolved theoretical problems;
2. to find a defect representation along with its interaction with scene components
using an object oriented description. It implies that a set of descriptors has to be
singled out. These latter are mathematical objects which are able to represent
different level semantic features, considered as representative for the image, such as
colour, texture, scene components, degradation, blotches dimension, scratch
appearance, etc. In other words, descriptors represent an alphabet for image
description. Descriptors connections, like "taking part of" or "lying on", constitute
the description grammar;
3. to detect a framework for formalizing and collecting information concerning the
meta-representation of the defect, the restoration technique and the achieved result.
This way, the right restoration process could be automatically applied if an effective
meta-representation of degradation is defined;
4. to define techniques for chromatic rendering which could be used as best and
practice for digital acquisition and image content presentations in which the
colorimetric point of view is a relevant aspect.
Attended results are methodologies, algorithms, prototype for software and meta-
descriptions along with publications about research results. In particular:
1. models and algorithms for the considered defects, to eliminate the gap between the
processed regions and the original ones and to preserve original information that
could occur in the degraded area;
2. set of descriptors and their connections (mainly spatial) more suitable for describing
image features and degradation typologies;
3. an ontological scheme for formalizing the knowledge driving restoration processes;
4. calibration functions for the application of chromatic rendering concerning
acquisition mechanisms and to measure the monitor response by means of spectro-
radiometer when it is displaying the primary colours.
Chapter 2
25
In my PhD work I dealt with the problems at point 2 and 3 of the list above. Approach
and obtained results are discussed in the next sections.
2.2 Related Works
Image and video content based retrieval is one of the most investigated research fields
of image analysis in the last decade. Images are typical examples of non structured
documents and then they cannot be indexed by means of traditional methods which are
based on key words memory or other textual descriptions. In fact, these latter are
computationally expensive and somewhat subjective. Image content analysis, i.e.
automatic or semi-automatic detection of the main features of regions and objects and
their measurement by means of direct measures (descriptors), represent the main steps
of the indexing process. Nevertheless, with regard to still images, although the
increasing algorithms for features extractions, there are not meaningful methodological
developments and most of the proposed approaches endow low level features such as
colour, shape, texture, etc. (see [11] for a wider review).
More recently, image objects description based approaches have been proposed. In
these cases, content based description necessarily becomes a two level jointed process:
- object low level features extraction;
- high-medium level object representation, by means of automatic or semi-automatic
mapping mechanisms, which allow us to put the intrinsic semantic in low level features
interactions or to render them independently of the observer-user.
Such methodological approaches refer to algorithms and techniques concerning image
analysis, database systems, information retrieval, artificial intelligence, knowledge
representation.
Although suitable, multimedia content representation does not guarantee greater
effectiveness for retrieval or, more in general, for applications based on user-images
interaction [12]. Some data modelling techniques, related to their content, have been
singled out. They allow both to propose sophisticated content representations and to
properly use them in query and navigation operations. In fact, it is known [13] that
Chapter 2
26
different kinds of information can be associated to a given image: a) data not directly
concerning its content but that can be related to it (content independent meta data), such
as names, date, format, etc.; b) data concerning low level features, such as colour,
textures, spatial relations, etc. (content dependent meta data); c) data concerning content
semantic (content descriptors meta data).
A language for representing and mixing extracted features is required. With regard to
this point, both the descriptions and the description scheme of MPEG-7 [14][15] can
represent a good example. The use of automatic or semi-automatic mapping mechanism
usually requires to employ learning algorithms that often are off line [16] but sometimes
can be on line (for example relevance feedback [17]). Moreover a "formal language" for
descriptions is provided, i.e. DDL (Description Definition Language) which helps both
query and navigation. For example, MPEG-7 uses XML (eXtensible Markup Language)
[18]. Usually some similarity measures (or distance [19]) between descriptors are
defined. They are often based on metric models but it is not a fundamental request.
With respect to the restoration step, professional operators often use commercial
software like Adobe Photoshop. This kind of restoration is heavily user-guided because
the defects are subjectively detected and the type of correction is user-selected too. This
task is complex, expensive, and acceptable just for very important pictures. An
automated system based on algorithmic restoration could facilitate the non-professional
restoration.
2.3 The Approach
The adopted approach consists in defining a knowledge-based model for image
restoration exploiting meta-representations of image contents, including degradation
typologies. The contents of degraded images will be stored on conventional relational
DBMS, rather than on a home made system, in order to make it fast and effective for
the users to find the information they needed.
It is assumed that a knowledge based model, exploiting automatic tools of digital
restoration, is able to further increase the enhancement and the use of historical images.
Chapter 2
27
In particular, material from Alinari Archives in Florence is analysed, composed of high
resolution, coloured, black and white images since 1840.
In order to make a complex process, as the restoration one, automatic, fine techniques
are required. These latter, starting from the analysis of the acquired image, have to be
able both to extract main image features, such as colour, texture, scene components,
dimensions, etc., and to detect its degradation, such as scratches along with their
dimensions, blotches and their morphology, etc. The description of these components
and their correlation by means of a formal language allows us to deal with the image
using a semantic level finer than the digital representation, which is commonly denoted
with meta-representation. Moreover, a restoration process can be made automatic by
finding a way of representing knowledge in order to formalize the pair "kind of defect-
recovering algorithm". Hence, it is now necessary to design a scheme which enables to
select the proper restoration methodology starting from the interpretation of the defect
meta-representation. Such a scheme constitutes an ontology in which at the best
satisfying the need of restoration, expressed by the image meta-representation, by
means of experience coming from knowledge. Finally the proposed model aims to free
users, interested in digital restoration, from the annoying task of image analysis, defects
detection, choice of the best restoration algorithms along with the selection of their
optimal parameters.
The proposed approach presents some completely novel tools for digital image
restoration:
- the restoration process is based on knowledge and then it is able to receive
experience about new kinds of defects along with more recent and effective algorithms
for restoration: the model constantly grows while it is used;
- with regard to the classes of defects (see chapter 3 for details about the defect
taxonomy) the algorithms for their restoration account for both their physical-chemical
causes and their human eye perception;
- image meta-representation and degradation can be also used in finer applications as
content retrieval and the automatic definition of pictures degradation typologies;
- a basis of knowledge is defined to automatically select the best restoration method
for an image which is affected by a specific damage.
Chapter 2
28
The next sections propose a new model for a restoration process, compare it to the
classical digital restoration model, and further discusses on the two main objectives of
the approach. Then it illustrates the basis of knowledge that has been designed to
represent the model. At last it describes the prototypal software that has been developed
to guide the user through the restoration process, and as an interface to the database.
2.4 The Proposed Restoration Model vs. Classical Model
This section briefly describes the adopted model, which is based on the real restoration
process of manual photo restorers.
Figure 2.1 Classical restoration model
In the classical approach[20] (see fig. 2.1) the degradation process is modelled as a
function that, with an additive noise term, operates on the ),( yxf input image to
produce the degraded image ),( yxg . Given ),( yxg , some knowledge about the
function H and the noise ),( yxη , the goal of the restoration is to have the closest
estimation ),(ˆ yxf . If H is linear and spatial invariant, the degraded image can be
represented as:
),(),(),(),( yxyxfyxhyxg η+∗=
where ),( yxh is the spatial representation of the degradation function and * is the
convolution operator. In the equivalent frequency domain representation:
),(),(),(),( vuNvuFvuHvuG +=
),( yxg),( yxf
Degradation Function
H
Restoration Filter(s) +
Noise ),( yxη
),(ˆ yxf
DEGRADATION RESTORATION
Chapter 2
29
where G, H, F, N are the Fourier transform of the corresponding functions. Starting
from the knowledge of the degradation function and the model of the additive noise, the
original image can be derived, typically using inverse filtering:
),(),(),(ˆ),( vuH
vuNvuFvuF −=
where ),(),(),(ˆ
vuHvuGvuF = is the Fourier transform of the estimated image.
This approach has several problem to handle. First of all, the degradation function and
the additive noise term are difficult to estimate, so this approach is often inapplicable.
Moreover this approach was designed for typical defects of digital images (blurring,
noise, etc) which are global and diffuse in the whole image. A different approach is
needed to handle defects that come from the digitization of printed documents, which
are in most case local defects, caused by degradation of part of the support of the photo.
To achieve these goals, a new model, which is inspired by the real restoration process of
manual photo restorer, is proposed.
Figure 2.2 The restoration model
Fig. 2.2 shows a simplified scheme of the model:
- Original image is the whole input image;
- Cropped image is the part of the image which is degraded, and it is selected by the
user;
- The crop is described with some descriptors, and descriptor values are the inputs of
the Classification Box;
- The classification step aims to recognize which type of damage the cropped image
is affected by, using information extracted in the description step;
Chapter 2
30
- The appropriate detection method is applied, to locate the position of damaged
pixels into the cropped image. Detection algorithms are also used to support the
description step. Detection is by-passed if a global damage has to be processed;
- The proper restoration algorithm is applied;
- The restored (cropped) image is overlapped with the original image, to reconstruct
the enhanced image.
2.5 The Knowledge Base
The elements of the proposed restoration process are included in a knowledge base,
which is represented by the entity-relationship diagram shown in fig. 2.3. The main
entities and relationships are listed and described in tables 2.1 e 2.2.
The original image, which is the whole digitized photo, can have no, one or more crops,
which are linked with their origin by the selection relationship. Cropped images are
described by one or more descriptors and classified according to a damage taxonomy.
Corruption description relationship connects these three entities. Damage is further
connected to the appropriate descriptors through description ability. Each descriptor is
implemented by one algorithm, to which it is linked by descriptor implementation. As
well, some of the algorithms are implemented to apply some restoration type. The
application of a restoration type is a restoration instance which produces a restored
cropped image. Each restored cropped image is linked to its original and its cropped
through the belonging relationship. Each restored crop can have a residual damage,
which is described in the persistent corruption description. The overlapping operation
is applied to put the restored crop into the enhanced image, which is linked to the
original image through the enhancement.
Chapter 2
31
Figure 2.3 The ERD diagram
Chapter 2
32
Table 2.1 Entities involved in the database ERD
Entity attributes description
Original Image
ID archive path author title year width height DPI print size color depth compression state
The whole image to restore
Cropped Image
ID path width height x_start y_start contours_file_path hits success descr_xxx…
A part of the original image into which there is the defect to restore
Restored Cropped Image
ID path width height x_start y_start contours_file_path hits success descr_xxx…
The crop of the image after restoration
Restoration Type ID type description
The restoration path applied to the crop
Damage ID family description
The defect taxonomy
Algorithm ID path description parameter_number
Algorithms used in the restoration process (description, detection, restoration, etc.)
Descriptor
ID type description threshold algorithm
Parameters used to describe defects
Enhanced Image
ID path original_image state result
The original image after restoration of the crops
Chapter 2
33
Table 2.2 Relationships between entities
Relationship Between Entities Description
Corruption Description
Cropped Image (0,N) Damage (0,N) Descriptor (0,N)
To describe defects of a cropped image using proper descriptors
Restoration Implementation
Restoration Type (1,N) Algorithm (0,N)
To apply algorithm of the chosen restoration type
Restoration Restoration Type (0,N) Restored Cropped Image (1,N)
To apply one or more restoration operation to a cropped image
Overlapping Restored Cropped Image (0,N) Enhanced Image (1,N)
To use a restored cropped image to enhance original image
Description Ability
Descriptor (0,N) Damage (0,N)
To link each type of defects to the appropriate descriptor
Persistent Corruption Description
Restored Cropped Image (0,N) Damage (0,N) Descriptor (0,N)
To describe defects of a restored cropped image using appropriate descriptors, after restoration
Selection Original Image (0,N) Cropped Image (1,1) To select part of an original image
Belonging Original Image (0,N) Cropped Image (0,N) Restored Cropped Image (1,1)
To link the crop, its restored version and the original image
Enhancement Original Image (0,N) Enhanced Image (1,1)
To link original image with their enhanced versions, giving an evaluation
Descriptor Implementation
Descriptor (0,1) Algorithm (0,1)
To apply the algorithm related to the chosen descriptor
The knowledge base has been implemented as a database using a conventional
relational DBMS (MySQL is chosen because it is open source). Fig. 2.4 shows the
database schema.
The implementation of the interface to the DB, which will be discussed in the next
section, showed that this structure, even if well designed for the general purpose of the
projects, has some practical limitations, that will be overshot in the future works. For
example, it doesn’t include the dual taxonomy for digital and real defects (see chapter
3), and doesn’t plan a fully automatic method to insert new descriptors in the
DB(manual intervention in the software code is required at the moment).
Chapter 2
34
Figure 2.4 The database schema
Chapter 2
35
2.6 The Prototypal Restoration Tool
A prototypal version of a restoration software tool has been implemented, using the
Matlab programming language. The tool is used both as an interface of the database and
to support the user during the restoration process.
Figure 2.5 A screenshot of the restoration tool
Figure 2.5 shows a screenshot of the main window of the tool. Its main functionalities
are:
- Opening from the db and saving into the db five different typologies of image:
o Original;
o Cropped;
o Restored;
o Enhanced;
o New;
- Selecting part of an image to create crops;
- Showing information about the state of the current image and, eventually, the
related original;
- Enhancing the image with generic global operations (histogram stretching, color to
greyscale, image user-defined filtering, manually adjusting the contrast, etc.);
Chapter 2
36
- Manually or automatically describing degradation of a cropped image, or the
residual degradation of a restored cropped;
- Applying an existing restoration path;
- Creating a new restoration path, consisting in a sequence of existing algorithms;
- Automatically suggesting to the user the appropriate restoration paths for the current
degraded image;
- Overlapping restored cropped into an enhanced image;
- Managing in the DB
o Inserting new objects in the DB (descriptors, algorithm, damages, etc.);
o modifying and deleting records and tables of the DB;
o Making useful queries to the DB(see subsection 2.8.2);
o Save the output of the queries in a custom XML format, or seen as an
HTML page.
The best way to explain how the restoration tool works is to describe a typical
restoration process of a degraded image.
2.6.1 Guiding the User through the Restoration Process
The user loads an image (original, cropped, restored, enhanced) from the database, or a
new image he wants to restore. Let’s consider the case of a new image. The new image
is saved into the DB, and it is stored in the original image table. Then the user has to
select a part of the image which he recognizes as damaged, with no knowledge about
the type of that damage. The selected part of the image is then saved as cropped image
(fig. 2.6.b). Cropped image can coincide with the original image, in case of global
defects. The next step is to classify the damage of the cropped image. The user can
choice to manually assign a damage, and the related severity, using the appropriate
plug-in tool, or to use the automatic description tool. In both cases descriptors of the
current image are computed, and stored in the DB. Before talking on the following steps
of the restoration process, some remarks must be drawn for the classification step.
Chapter 2
37
a) original image d) enhanced image
b) cropped image
c) restored image
Figure 2.6 Processed images during the restoration process. Selected cropped image is highlighted in red into the original image
The automatic classifier is still an open problem. In the current version of the tool, the
user can choice between a classifier which is based on standard MPEG-7 image
descriptors, and another one which uses damage oriented descriptors. The MPEG-7
based classifier uses 3 color descriptors (Color Coherence Vector, Dominant Color,
Color Structure) and 2 texture descriptors (Gabor and Edge Histogram) (see section 3.3
for an overview about MPEG-7 image descriptors). This classifier compares the current
image with all the cropped image in the DB, using a fixed-weighted distance of the
corresponding descriptors. Then it assigns to the current image the damage of the most
similar image (see the top part of fig. 2.7) , according to this distance, in the DB, and
asks confirmation to the user. The other one classifier is designed, at the moment, to
discriminate only foxed images, and will be described in chapter 4. The definitive
solution of the problem should be a classification box which would combine standard
and specific descriptors. Note that description ability table contains the relationships
Chapter 2
38
between implemented descriptors and damages. That is, which descriptors are able to
describe which damage. The table is filled analyzing tests obtained using the content
retrieval plug-in tool, based on standard descriptors (see section 3.3.1). Further tests are
required but this information cannot be ignored when designing the classifier.
Figure 2.7 From classification to restoration path proposal
Taking back to the restoration process, once the cropped image has been classified, the
proper restoration type has to be applied. The term “restoration type”, or as well
“restoration path”, is used rather than “restoration algorithm” because a “restoration
type” can be made of more “restoration algorithms”.
The current version of the tool let the user to choice between three options:
- Using one of all the existing restoration paths. The user selects one of the restoration
paths in the list and, when needed, manually sets the required input parameters.
- Creating a new restoration path combining existing algorithms. At the end of the
process the new restoration path will be saved as a new restoration type, and its
implementation, as a sequence of applied algorithms, stored in the “restoration
implementation” table.
- Allowing the system to suggest the appropriate restoration paths for the current
image. The tool analyzes all the cropped images which are affected by the same damage
Chapter 2
39
of the current image, and which has been yet restored. Then it proposes to the user those
restoration paths which has been used to correct these images(see bottom part of fig.
2.7). Paths in the sub-list are rated, using the mean vote of the restoration results. The
user can choice one element of this sub-list, and manually set the required input
parameters, when needed. Otherwise, the tool can suggest the best parameters to the
user. It compares, using the appropriate descriptors for the current damage, the current
image to the subset of the images, affected by the same damage, which has been
restored by the chosen path. It can be reasonably supposed that similar images,
according to the proper descriptors, can be processed using the same parameters. Then
the tool proposes these parameters to the user. Further testing is required to evaluate this
solution.
In some cases the chosen restoration type requires a binary mask, into which pixels of
the cropped image are labelled as damaged or uncorrupted. Whenever an automatic
defect detection is not a part of the restoration path, a tool is provided to the user, to
manually select pixels to restore and to create the required binary mask.
After the cropped image is restored (fig. 2.6.c), the tool requires the user to vote the
restoration result, and the new vote will contribute in the restoration path selection for
the next images to process. The restored cropped image is saved and the restoration,
which is a single instance of a restoration type, is stored in the corresponding table with
the given vote.
The last step is to merge the restored cropped image and the original image into the
enhanced image (fig. 2.6.d). Optionally the overlapping step can be done later.
2.6.2 Making Visual Queries to the DB
The tool is designed also to be used as an interface to the database.
In addition to the classical MySQL queries, several useful visual queries can be
selected by users:
- Original, restored, cropped images, enhanced: to extract the list of original, restored,
cropped and enhanced images from the archive;
Chapter 2
40
- Images damaged with a specific damage: to show all the cropped images which are
affected by a specific damage. This can be useful to compare different instances of the
same defect.
- Images enhanced or restored with a specific restoration type: to show visual results
of a specific restoration type;
- Comparing original and enhanced: to compare images before and after the overall
restoration process
- Comparing cropped and restored: to compare visual results of a restoration instance;
Query outputs can be shown directly in the tool window, or seen as a HTML page. As
well, query results are saved in XML format.
2.7 Conclusions
Historical images are essential documents of the recent past. Nevertheless time and bad
preservation corrupt their physical supports. Digitization can be the solution to extend
their “lives”. Furthermore lost information can be recovered with digital techniques.
The Italian scientific project, into which my research work has been developing, aims to
achieve significant results in the field of restoration of damages in digitized old photos.
This chapter presented the key problems related to this field. A restoration model,
which is inspired by the process of manual professional restorers, is proposed, and
compared to the classical model. A basis of knowledge is used to represent the elements
and the relationships of the model. Images, but also damages and descriptions, are
considered as entities involved in the process. Data are stored using a conventional
relational DBMS.
A prototypal restoration software tool is implemented to extract information from the
database, in order to support non-experts user in the restoration process
Chapter 2
41
Acknowledgements
This work has been funded by the MIUR (Italian Ministry of Education, University and
Research) project FIRB 2003 D.D. 2186 - Ric December 12th 2003. I wish to thank
Alinari Archive which has permitted the use of their digitized photo database for my
research. I also acknowledge G. Grecomoro, S. Fabiano, A. Gulli, G. Schillaci, S.
Vicari, C. Maniscalco, and F. Turturici for their implementation work of the restoration
tool and the plug-ins.
Chapter 3
The Defect Taxonomy
Old images may present a huge variety of damages, due to several different factors.
Some defects may lead to a complete loss of information, while other deteriorate the
overall appearance of images. Mostly, the damages are originated by inappropriate
environmental conditions (temperature, humidity, lighting), inaccurate handling (dirt,
image protection, cracks) human intervention (stamps, writings, restorations) and
chemical factors (reactions with micro-organisms). Usually, the same image may
contain many distinct defects.
While the origin of image defects on the physical support (whether positive or negative)
is an important issue for a manual restoration activity, several defects appear similar
once images are digitally scanned and should be described and removed by similar
underlying processes.
A first interesting attempt to propose a possible taxonomy of all the typical defects in
old photos is in[21], but this analysis is incomplete and did not focus on the digital
aspect of the defects.
The first part of the chapter proposed a former defect taxonomy, which is based on the
physical-chemical origin of the defects, and discusses about its limitations.
Then a new dual taxonomy is presented, which distinguishes defects in real photos and
in their digital versions, and then a comparative study to correlate the two defect sets is
discussed.
Chapter 3
43
3.1 Origin-based defect taxonomy
Defects of old photographic prints can be divided in different sets according to their
origin(fig. 3.1 shows the scheme):
Figure 3.1The origin-based defect taxonomy
3.1.1 Mechanical (physical) damages
Originated usually by an inaccurate handling and/or store of the original image; may be
further divided into:
- Deposited matter: different materials adhere to the surface creating small spots that
cover the original image; may be seen as the presence of localized high-frequency
noise; Some examples are:
o Dirt;
o Dust;
o Fingerprints;
- Physical alteration of images: usually originated by an inaccurate handling; often
lead to a complete loss of information and should be removed by specialized
techniques. Typical examples are:
o Cracks: deteriorate the aspect of the image and may be very large; do not
exhibit a dominant orientation; however, each crack has its own
direction; may also appear because of folded or torn scanned images;
Chapter 3
44
o Scratches: thin straight lines without a preferential direction;
o Craquelures: micro-fractures of the support of the photo, usually
branched;
o Abrasions: lack in the emulsion of the photo, cause by friction with other
part of the photo or with some external tool;
- Deformations: originated by an inappropriate conservation of original images;
often caused by excessive humidity and/or temperature and corrupt the way the gelatin
is fixed to the support; the effect is a deformation of the planarity of the support:
o Lifting;
o Bending;
- Human retouches: deliberate human retouches that usually irremediably alter the
image; some examples are:
o Gaps;
o Stamps;
o Writings;
o Presence of adhesive;
o An impropriate restoration.
3.1.2 Chemical damages
Defects originated chemically, may be further divided into:
- Spots:
o Blotches: originated by water or humidity; each pixel preserves the
information about the real data and noise;
o Foxing: originates as the result of chemical reactions between the print
and some microorgan-isms; appears as reddish-brown spots;
o Other: other types of spots in the image;
Chapter 3
45
- Tonal and color balance defects: originated by an excessive exposure of original
photos to light; some examples are:
o Bleaching (Fading): overall whitening of the image;
o Yellowing: alteration in the image chromaticity, which tends to yellow;
o Uniform/Irregular color cast: occurs in images where color balance has
been destroyed.
3.1.3 Limitations and problems
This classification proved to be not-well suited for the purpose. The need of a new dual
taxonomy arises from one of the goal of the project, which aim to implement an
automatic defect classification method, to link to damages the most appropriate
“digital” detection and restoration algorithms. Digital features (shape, color, texture,
etc.) of a defect must be analyzed, rather than considering its origin. For example, an
automatic classifier won’t be able to discriminate an abrasion from a tear, if their digital
versions have similar features (see fig. 3.2)
a) fold
b) abrasion
c) tear
Figure 3.2 A comparison between three defects, which have similar digital aspects. According to their origin they are classified as three different types of defect.
Chapter 3
46
To understand which are the digital features that have to be analyzed for a digital defect
taxonomy, next sub-section presents an overview of the MPEG-7 standard visual
descriptors for digital images.
3.2 MPEG-7 Visual Descriptors
Image features such texture, color, motion, object shape have been investigated during
the past decade as possible descriptors of the image content.
Each one of them may be related to the whole image (global features) or to one or more
image parts (local features). Local features are specially important if related to
meaningful image parts, i.e. regions normally corresponding to objects present in the
scene. For example, an object’s shape may be described in terms of its boundary but
also in terms of geometric properties like area, perimeter, aspect ratio, etc.
The choice of the more appropriate descriptor strongly depends on the application.
A subset of the MPEG-7 visual descriptors can be selected from the proposed standard
descriptors set for the purpose to achieve an effective meta-representation of damages
as objects in the image. The MPEG-7 visual descriptors[22][23] can be classified into
general visual descriptors and domain specific descriptors. The first ones describe the
low-level visual features such as color, texture, shape, motion, and so forth; the second
ones are application dependent and include identification of human faces and face
recognition. Three kinds of low-level visual descriptor, are briefly discussed in this
section: Color Descriptors, Texture Descriptors and Shape Descriptors.
3.2.1 Color Descriptors
There are seven Color Descriptors: Color space, Color Quantization, Dominant Colors,
Scalable Color, Color Lay-out, Color-Structure, and GoF/GoP Color.
- Color space: The feature is the color space that is to be used in other color based
descriptions. The following color spaces are supported: R,G,B ; Y,Cr,Cb ; H,S,V;
HMMD; Linear transformation matrix with reference to R,G,B; Monochrome.
Chapter 3
47
- Color Quantization: This descriptor defines a uniform quantization of a color
space. The number of bins which the quantizer produces is configurable, such that great
flexibility is provided for a wide range of applications.
- Dominant Color(s): Color quantization is used to extract a small number of
representing colors in each region/image. The descriptor consists of the representative
colors, their percentages in a region, spatial coherency of the color, and color variance.
- Scalable Color: The Scalable Color Descriptor is a Color Histogram in HSV Color
Space, which is en-coded by a Haar transform. Its binary representation is scalable in
terms of bin numbers and bit representation accuracy over a broad range of data rates.
- Color Layout: This descriptor effectively represents the spatial distribution of color
of visual signals in in an arbitrarily-shaped region, in a very compact form. Its
compactness allows visual signal matching functionality with high retrieval efficiency
at very small computational cost.
- Color-Structure Descriptor: The Color structure descriptor is a color feature
descriptor that captures both color content (similar to a color histogram) and
information about the structure of this content. To this aim, a 8x8 pixels window slides
over the image. With each shift of the structuring element, the number of times a
particular color is contained in the structure element is counted, and a color histogram is
constructed. Values are represented in the HMMD color space, which is non-uniformly
quantized.
- Group-of-Frames/Group-of-Pictures (GoF/GoP) Color Descriptor: The
GoF/GoP color descriptor defines a structure required for representing color features of
a collection of similar frames or video frames by means of the SCD. It is useful for
retrieval in im-age and video databases, video shot grouping, image-to-segment
matching, and similar applications. It consists of average, median, and intersection
histo-grams of groups of frames calculated on the individual frame histograms.
Another color descriptor, which is very diffuse in literature but that is not included in
the MPEG-7 standard, is the Color Coherence vector (CCV). The color coherence
vector[24] is an indication of how similar colors are oriented in the image. Regions
Chapter 3
48
which have a collection of similar color pixels are termed as coherent regions. The CCV
computes the number of coherent and non-coherent pixels.
3.2.2 Texture Descriptors
There are three texture Descriptors: Homogeneous Texture, Edge Histogram, and
Texture Browsing.
- Homogenous Texture Descriptors: The Homogenous Texture Descriptor describes
directionality, coarseness, and regularity of patterns in images. It is useful for image-to-
image matching for texture image database retrieval. This descriptor is extracted
filtering the image with a bank of orientation and scale tuned filters (modeled using
Gabor functions) using Gabor filters. The computation of this descriptor is based on
filtering using scale and orientation selective kernels.
- Texture Browsing: The computation of this descriptor proceeds similarly as the
Homogeneous Texture Descriptor. First, the image is filtered with a bank of orientation
and scale tuned filters (modeled using Gabor functions); from the filtered outputs, two
dominant texture orientations are identified. This is followed by analyzing the filtered
image projections along the dominant orientations to determine the regularity and
coarseness. The second dominant orientation and second scale feature are optional. This
descriptor, combined with the Homogeneous Texture Descriptor, provide a scalable
solution to representing homogeneous texture regions in images.
- Edge Histogram: The edge histogram descriptor represents the spatial distribution
of five types of edges: vertical, horizontal, 45 , 135 , and non-directional edge. Since
edges play an important role for image perception, it can retrieve images with similar
semantic meaning. Thus, it primarily targets image-to-image matching (by example or
by sketch), especially for natural images with non-uniform edge distribution. In this
context, the image retrieval performance can be significantly improved if the edge
histogram descriptor is combined with other descriptors such as the color histogram
descriptor.
Chapter 3
49
3.2.3 Shape Descriptors
There are three shape Descriptors: Contour-Based Shape, Region-Based Shape and
Shape 3D.
- Contour-Based Shape: The Contour Shape descriptor captures characteristic shape
features of an object or region based on its contour. This descriptor is based on
curvature scale-space (CCS) representations of contours and also includes of
eccentricity and circularity values of the original and filtered contours. A CCS index is
used for matching and indicates the heights of the most prominent peak, and the
horizon-tal and vertical positions on the remaining peaks in the so-called CSS image.
- Region-Based Shape - Art: The MPEG-7 Region-Based Descriptor ART (Angular
Radial Transformation) is suitable for shapes that can be best described by shape
regions rather than contours. The main idea behind moment invariants is to use region-
based moments which are invariant to transformations, as the shape feature. The
MPEG-7 ART descriptor employs a complex Angular Radial Trans-formation defined
on a unit disk in polar coordinates to achieve this goal. Coefficients of ART basis
functions are quantized and used for matching.
- shape 3d: not useful for the purpose.
3.3 Dual taxonomy
This section presents the proposed a dual taxonomy. Table 3.1 Real defect taxonomy
BIOLOGICAL ALTERATIONS PHYSICAL ALTERATIONS CHEMICAL ALTERATIONS 1 infections 4 garbage 12 bending 20 spots 2 infestations 5 dust 13 marks 21 fading 3 other 6 fingerprints 14 Abrasions 22 yellowing 7 stains 15 tears 23 silver mirror 8 folds 16 lacunas 24 sulfuration 9 craquelures 17 cracks 25 foxing 10 lifting 18 presence of adehesive 26 other
11 deformations 19 other
Chapter 3
50
Table 3.2 Digital defect taxonomy
DIGITAL DEFECTS TYPE REAL DEFECTS Spots Local 4,6,14,19,20,25 Semi-Transparent Spots Local 20 Scratches Local 8,11,14,15,16,17 Foxing Local/Diffuse 25 Folds Local 14 Cracks Local 15,11 Deformations Local 11 Blotches Diffuse 7,20,23 Fading Global 21 Yellowing Global 22 Irregular Color Global no images in the DB Lacking Emulsion Local 14,16 Lacking Portions Local 8,11,16,18 Handwritings Local 19
Figure 3.3 An example of an annotation file written (in Italian) by one of the manual restorers of the Alinari Archives (courtesy)
Chapter 3
51
a) spots
b) semi-transparent spot
c) blotches
d) handwritings
e) crack
f) scratch g) deformations (lifting)
h) fold
i) foxing
j) fading
k) yellowing
l) lacking emulsion
m) lacking portions
Figure 3.4 some examples of defects classified with the digital taxonomy (courtesy of Alinari)
Chapter 3
52
Table 3.1 lists typical defects of old printed photos, classified according to the origin of
the damage. This classification is that used by professional manual restorers of the
Alinari Archives in Florence. Each type of defect is labelled by a standard numerical
code, which univocally identifies the damage.
Fig 3.3 shows a typical file which is used by manual restorer to annotate and highlight
defects of a damaged printed photo. Note the code that is reported beside the name of
the defect. The meaning of the fields in table 3.1 is the same of that listed in the
previous section. Only some fields are added, which meaning is self-explaining.
With respect of the digital taxonomy, table 3.2 lists the digital defects observed in the
testing dataset, of which a brief description is given below. Fig. 3.4 shows some
examples of images affected by the defects listed in table 3.2. According to their digital
features:
- Spots: local defects. Fig.3.4.a. Underlying information is lost and must be fully
substituted. No specific colors. More or less rounded shaped. Color and shape
descriptors could be useful to describe and detect these defects.
- Semi-Transparent Spots: local defects. Fig.3.4.b. Residual information can be
recovered with restoration techniques. Texture descriptors useful for the detection step.
- Scratches: local defects. Fig.3.4.f. Thin lines, with a preferential direction. Usually
lighter than the rest of the image. Can have a darker kernel. Dark scratches in movie
film and glass plates. Possible limited changing in width and slope.
- Foxing: local/diffuse defects. Fig.3.4.i. Covering o semi-transparent spots. Red-
brown color. See chapter 4 for further details.
- Folds: local defects. Fig.3.4.h. Located near the edges of the photo. It is composed
by a lighter central area and darker edges, due to the acquisition operation.
- Cracks: local defects. Fig.3.4.e. Undefined orientation. In some cases they can have
branches. Cracks are usually composed by a darker kernel surrounded by a lighter area.
- Deformations(lifting): local defects. Fig.3.4.g. Due to the digital acquisition of a
non planar support. Look like the negative of a branched crack. Too few examples in
the DB to identify its main features.
Chapter 3
53
- Blotches: diffuse defects. Fig.3.4.c. Semi-transparent spots which can be seen all
over the image. Usually lighter than the rest of the image. Silver Mirror is a particular
case of blotch. It is silver coloured and located mostly in the darkest areas of the image.
- Fading: global defect. Fig.3.4.j. Overall whitening of the image.
- Yellowing: global defect. Fig.3.4.k. Distortion in the chromaticity of the whole
image. Tends to yellow.
- Irregular Color: global defect. Not in the DB.
- Lacking Emulsion: local defect. Fig.3.4.l. Undefined shape (some similar to
scratches, some other to spots). Usually lighter the rest of the image, because of the
exposition of the color of the support. Information is totally lost. Color descriptors can
be used for these defects.
- Lacking Portions: local defect. Fig.3.4.m. Usually darker than the rest of the
image, but it depends on the acquisition condition. In most cases lacking portions have
jaggy edges.
- Handwritings: writings and scrawl. Fig.3.4.d. Complex curve lines, darker than the
rest of the image.
Table 3.2 shows also how digital and real defects are correlated in the testing dataset, a-
posteriori comparison of the same dataset that is annotated using the two taxonomies.
Manual annotation of the image dataset has been made by a professional manual
restorer of the Alinari Archives. I provided the annotation according to the digital
classification. Note that there is no 1-1 relationship between the defects in the two
taxonomies. For example, “digital” spots can be caused by the digital acquisition of
defects such as garbage, abrasions or chemical spots. On the other hand, real photos
which had been manually annotated as affected by abrasion, according to the digital
defect taxonomy are classified as spots, lacking emulsion, or scratches. It is clear that
digital and manual restorers often cannot be in agree about the classification of a defect
in an image. This work wants to be an useful tool to let digital and manual restorers to
draw nigh their different points of view.
Chapter 3
54
3.3.1 Description Ability
One of the key points in the description step of the restoration model presented in the
previous chapter is the “description ability”, the capability of a descriptor to describe a
specific defect. As discussed in section 2.6.1 the classification method should use both
damage oriented descriptors (see next chapter), which are designed to discriminate a
single damage, and generic descriptors, whose have been discussed in section 3.2.
For the classification step, at the moment, three color descriptors (Color Coherence
Vector, Dominant Color and Color Structure) and two texture (Edge Histogram and
Gabor Filter based) descriptors have been implemented.
Preliminary tests showed that there is a relationship between these descriptors and the
damages in our taxonomy. Namely, some generic descriptors are better suited to
describe some defects rather other ones. In particular it has been observed that, for
texture descriptors, Edge Histogram better describes line-like defects (scratches above
all), while Gabor-based is appropriate for more structured defects (like branched cracks)
or spot-like (spots, foxing, etc.). Global (color) defects cannot be analyzed using texture
descriptors.
With respect of the color descriptors, global defects are, obviously, correlated to
Dominant Color descriptor, which is also involved whenever a single color is the main
feature of the damage (e.g. red for foxing, black for lacking portions). Color Structure
can be used to describe covering and semi-transparent spots, and blotches, while for
other types of defects Color Coherence Vector should be chosen. However in most
cases CCV an DC give very similar results..
Table 3.3 summarizes the relationships between damages and the implemented
descriptors.
This table would suggest that some defects, e.g. covering and semitransparent spots,
could be classified as the same damage, because they can be described using the same
descriptor set. However, as discussed above, different are the techniques used to detect
and restore the proposed typologies of defects. Furthermore they are presented as
separate defects.
Chapter 3
55
Table 3.3 Description ability DEFECT DESCRIPTORS spot CSD, GABOR semi-transparent spot CSD, GABOR foxing CCV, DC, GABOR crack CCV, DC, GABOR handwritings CCV, DC, GABOR fold CCV, DC lacking emulsion DC, GABOR lacking portion CCV, DC, GABOR deformations CCV, GABOR blotches CSD, GABOR whitening CCV, DC yellowing CCV, DC scratch CSD, EHD
These empiric observations constitute only a preliminary step, and much work and
further tests are needed, but this information, together with the implementation of
damage specific descriptors, will be an essential factor in designing a knowledge-based
classifier.
3.4 Conclusions
Manual and digital restorers don’t speak the same language. The first ones watch for the
origin of the defect, in order to manually remove it from the support using the
appropriate technique. Digital restorers must concern with the digital features of the
image, because digital are the techniques which have to be used to describe, detect and
restore defects. That’s why a defect taxonomy which is based on the causes (physical,
chemical, etc.) of damages in printed photos, cannot be used to catalogue defects of
their digitized versions.
This chapter wants to be a first attempt to show the differences and the analogies of the
two approaches to the problem.
This chapter presented also a preliminary study on the relationships between MPEG-7
visual descriptors and defects in the proposed taxonomy. A more analytic study is
needed to enrich the basis of knowledge with consistent information, in order to
implement the defect classification application discussed in the previous chapter.
Chapter 4
Classification – A Case Study: Foxing
In this chapter a classification application for images affected by foxing is proposed. It
is based on a set of low level descriptors, used to extract information from foxed
images. An image retrieval tool, which uses the features extracted by the proposed
descriptors, is developed to classify foxed images. Results are compared to those
obtained using some state-of-the-art color descriptors.
4.1 Introduction and related works
The art of photography is more than 150 years old, but it absorbed quickly
technological innovations of the following years. Methods, cameras, techniques
changed and improved, and so supports changed, from physical (paper) to digital ones.
Even if the discussion about advantages and disadvantages of digital and film cameras
is still open, the need for digital preservation of old documents became more and more
pressing. Their economic worth and high cultural value induced the use of digital
techniques to protect and preserve these goods. Old photographic prints may present
several types of defects, due to several factors. In most cases, damages are originated by
an inaccurate handling and/or store of the original image, or by chemical factors, or by
decomposition of the support caused by age(see chapter 3). While knowledge of the
causes of degradation is important for defect analysis on the physical support, different
defects may look similar once the document has been digitized. Manual annotation of
the damage cannot be a solution. It is expensive and time consuming, because of the
typical huge amount of data to be analyzed. Automatic or semiautomatic methods are
needed to help in this task.
Chapter 4
57
Several works rely on the damage analysis of digitized/digital documents. The works of
Abras et al [25] deals with feature extraction for defects such cracks or craquelures in
paintings. The PrestoSpace project[10] focused on the analysis and removal of defects
for the preservation of audiovisual collections. For a complete overview of the existing
works in the field of content-based image retrieval see Smeulders et al [26].
This chapter focuses on a specific damage of printed photos, the “foxing” spots. The
purpose of this chapter is to present a set of features for the content analysis of digitized
photos affected by foxing. An image retrieval application, based on the extracted
features, is proposed to detect whether an image is affected by foxing or not.
4.2 Foxing spots
Foxing is a typical chemical damage which can be seen in old books, documents,
postage stamps, and so forth. The term “foxing” was used for the first time in the 18th
century, to indicate those red-brown (the color of the fox fur) spots in the surface of the
paper of old documents. Actually causes are not clean. Two are the most reliable
theories about the chemical origin of these spots[27]. One is that spots are caused by the
growth of some fungoid micro-organisms on the surface of the paper. Other one asserts
that foxing would be caused by the oxidation of iron, cop or other substances of which
the photographic support is made. Probably multiple factors are involved. Foxing spots
are composed by a dark red-brown kernel, surrounded by an area into which colors are
smoothed (see fig. 4.1 for some examples). Information in the center of the spot is
totally damaged and must be considered lost. Surrounding area can have some residual
information that could be enhanced with manual or digital techniques. However a
discussion about the restoration techniques for documents affected by foxing is out of
the scope of this chapter.
4.3 Foxing detection
The digital acquisition of an image implies the acquisition of the defects of which the
image is affected. In a digital image, defect detection includes the ability to detect the
Chapter 4
58
presence of a particular defect into the image and, eventually, to locate the position of
the damaged pixels.
This section briefly presents the foxing detection algorithm proposed by Stanco et
al.[28] which will be used in the feature extraction process.
Due to the particular nature of the foxing defect, this algorithm is based on color of the
spots, and its distribution. The algorithm works in two steps:
- image is decomposed in the YCbCr color space and only the Cr chrominance is
processed. It has been shown that Cr histogram of foxed images presents a tail on the
right, composed by a set of bins having almost the same small height, and with the peak
on the left. The bins on the right tail represent the points damaged by foxing.
- finding all the pixels where the original information is only partially affected by
foxing. They are characterized by a lighter color than those in the center of the foxing
spot, and their position is near the reddish-brown area.
4.4 Feature extraction
For an automated application of analysis and inspection of an image, some local and
global information must be extracted. That is, some meta-data must be extracted from
the image. These data will be used by some specific operators to analyze the image
content. “Descriptor” is the representation of one or more features of an image. The
MPEG-7 standard group proposed(see section 3.2) a set of descriptors (color, texture,
shape, motion, etc.) to formalize the content of multimedia data. The definition of new
(a)
(b) (c) Figure 4.1 Some examples of images affected by foxing. In Fig. 4.1.c it can be easily observed both the darker kernel of the foxing spot and the lighter surrounding area. (Courtesy of Alinari Archives)
Chapter 4
59
descriptors for a specific damage of an image can be useful for the development of
damage oriented applications (restoration, classification, etc.).
For the foxing description, the choice of the best set of descriptors must depend on two
aspects:
- the specific features of the foxing spots, color and its distribution;
- the definition of relevant distance metrics, designed for an image retrieval
application.
This work focuses on three different set of features, to describe a digital foxing spot: the
Cr chrominance histogram, the statistical values of the damaged pixels, and the spot
size. Each of these features is used to analyze a specific aspect of the spot.
(a)
(b)
(c)
Figure 4.2 a) Image affected by foxing spots. b) histogram of the Cr chrominance (zoom on the tail). c) detected damaged area.
The first proposed descriptor is the tail of the Cr chrominance histogram, detected as
described in section 4.3 (see fig.4.2.b). Bins and heights of the histogram, from the right
to the left of the tail, are stored,
( )( ){ } 256...1,1 =≥= iBBBhBd liii (4.1)
where Bi are the bins of the histogram, h(Bi), the corresponding heights and Bl the left
value of the tail.
This descriptor gives us information about the length, the minimum value, and the
distribution of the heights of the tail. That is information about the shade of the color of
the spot.
Chapter 4
60
The second proposed descriptor is a set of statistical values of the Cr chrominance
component: mean, variance, minimum and maximum value. Values are computed only
into the damaged area, detected with the algorithm described in section 4.3 (see
fig.4.2.c).
The third descriptor is based on the average size of the spots. This is computed by
considering the ratio of the number of the damaged pixels in the detected area to the
number of the distinguishable spots, located with a 8-connection labelling process(see
fig.4.2.c).
The next section presents an image retrieval application, based on the information
extracted by these descriptors.
4.5 Content-based foxing retrieval
A content-based image retrieval (CBIR) application deals with the problem of searching
for digital images in a large database. Content-based means that image retrieval is made
using information that can be extracted from the image itself, rather than using tags,
keywords or annotations by hand.
The goal of the proposed CBIR is to detect whether an image is affected by foxing or
not. That is, given a new image, its content is compared to that of all the images which
are in the dataset. If the most similar image, according to some distance metrics, is
affected by foxing, the new image can be reasonably supposed to be “foxed”. In this
section three different metrics, based on the proposed descriptors, are presented.
The first one is based on the histogram tail descriptor. It is obtained as the difference,
point to point, of the heights of the corresponding bins of the tails. Since tails can have
different supports, starting and ending point, that with the lower maximum is shifted
rightward to align the two maximum, and that with the shorter support is zero-padded to
have the same support of the other one. Given d11 and d1
2, the descriptors of the two
different tails, supposed d11 to have a maximum value n-units higher than that of d1
2,
the distance of the two tails is:
Chapter 4
61
( ) ∑=
− −=max
min
)()(, 2121
111
B
BBini
i
BhBhddD (4.2)
where Bmin is the lowest of the two minimum and Bmax is the highest of the two
maximum bins. This distance can be used to compare information about the different
“trends” of the color in the two images (only damaged pixels), regardless of their
absolute values.
The second proposed metric is based on the statistical values of the damaged pixels. It
is obtained as the sum of the absolute differences of the corresponding components of
the statistical descriptors of the two images:
( )max2
2max1
2min2
2min1
2
var22
var12
22
12
22
122 ,
dddd
ddddddD meanmean
−+−+
+−+−=. (4.3)
Note that standard deviation is considered rather than variance. This distance is used to
compare the absolute values of the color of the spot. That means the “general” color
aspect of the defect.
The third proposed distance is based on the average size of the spots in the image. It is
the absolute difference of the size descriptors of the two images:
( ) 23
13
23
133 , ddddD −=
. (4.4)
Many combinations of the three distances has been tested, e.g. a combined distance
with different weights. Experiments showed that the best solution is a multi-step
classifier. Given a new image, it is compared with all the images in the dataset using
only one of the proposed distances. The N most similar images in the dataset, according
to this distance, are selected. Matching is then made using one of the other two
distances only within this image subset, and the M (M<N) best images are extracted. A
final comparison is made, within this new subset, using the last metric, and the best
image is shown. If this image is affected by foxing, the new image is classified as
foxed.
Chapter 4
62
4.6 Experimental results
Tests are made on an image dataset composed by about 220 images coming from the
Alinari Photographic Archives in Florence. Images in the dataset are affected by several
typical defects of old photos (scratches, spots, cracks, foxing, etc.) and has been
manually classified by a restoration expert of the Archives. For the experiments, the
whole dataset is used for testing. Each image is used as a test image and it is matched to
the other images in the dataset. About the 30% of the image is affected by foxing. Table
4.1 shows the experimental results, for each distance discussed in section 3.5 and for the
3- step classifier. Tests suggested the best configuration as follows: step 1, histogram
distance and N=8; step 2, statistical distance and M=4; step 3, size distance. Table 4.1 Experimental results
results\distance HIST STAT SIZE 3-STEP CCV DC CS
Correct classification(%) 84,3 87,6 71,4 90,3 88,5 89,4 81,6
False positives(%) 8,3 6,4 13,4 5,1 7,4 5,1 9,2
False negatives(%) 7,4 6 15,2 4,6 4,1 5,5 9,2
feat extract 1,3 Avg exec time (s) matching 0,1 <0,1 <0,1 0,1
3,9 3,7 9,1
Results are compared to those obtained with distances based on three standard color
descriptors[22][24]: Color Coherence Vector (CCV), Dominant Color (DC) and Color
Structure (CS).
For each distance table 4.1 reports the percentage of correct classifications, of false
positives (images without foxing classified as foxed) and of false negatives (images
with foxing classified as not foxed). Average execution time is also shown to compare
the efficiency of the classification methods. For the proposed distances, time is shown
for the all-at-once feature extraction and for each matching process. It has been
observed that:
- the statistical descriptor, among the proposed, gives best results, which are
comparable to those obtained with standard CCV and DC, but it takes much less
execution time;
- the size descriptor gives, as expected, no good results, because it is not color-
based; its role is to refine the retrieve in the last step of the classifier;
Chapter 4
63
- CCV and DC give very similar results;
- CS descriptor gives worst results and is less efficient than the two other standard
ones.
The multi-step classifier improves results obtained with the statistical descriptor, with
no significant increment in the execution time. Results are similar to those obtained by
standard descriptors, with much less execution time. Further combinations with the
standard descriptors may improve the results.
A multi-step classifier was implemented using only standard descriptors, with no
significant improvements in the results.
4.7 Conclusions and future works
Detection algorithms may extract misleading information, if they are applied to images
which are not affected by the damage they are designed for. The classification step
gives an interpretation to this information, comparing it with a dataset of images
corrupted by several types of degradation. The goal is to discriminate if the image is
affected by foxing.
Experiments showed that the proposed application gives same results compared to those
obtained using standard descriptors, with improvement in execution time.
Actually I plan to analyze some other typical defects of which old documents are
affected (scratches, cracks, spots, fading) in order to find appropriate descriptors for
each type of defect. The goal is to implement a more complex classifier which will be
able to discriminate between a wide set of damages.
Acknowledgements
This work has been funded by the MIUR (Italian Ministry of Education, University and
Research) project FIRB 2003. I wish to thank the Alinari Photo Archives in Florence
for having permitted the use of their photos in this research. I also acknowledge
Giuseppe Miceli for his implementation work.
Chapter 5
Detection and Removal of Quasi-Horizontal Scratches
This chapter presents a fast and effective method to support the user in the detection and
the removal of quasi-horizontal scratches in still images. The first step is a
semiautomatic detection method, and then an unsupervised restoration algorithm is
applied. The test dataset is composed by a digitized archive of aerial photos concerning
the Sicilian territory.
The chapter is organized as follows. After the introductive section, it analyzes scratch
features of the defects in the test photo dataset. Sections 5.3-5 present the proposed
method, while experimental results are shown in section 5.6. Some concluding remarks
are then given in the final section.
5.1 Introduction and related works
Mechanical scratches are typical defects in old movie films. This specific damage is
caused by the lost of the emulsion of the film surface, due to contacts with mechanical
parts of film projector or other devices in the film development process. Bright or dark
scratches run all over the frames of a movie film. Reconstruction of damaged
information is a fundamental task with the purpose of digitalization and preservation of
old photos or movies archives. Manual restoration is the standard method to reconstruct
damaged information, but is expensive and time consuming, because of the typical huge
amount of data to be restored. Automatic or semiautomatic methods are needed to help
the user in this task.
The problem of scratch removal has been addressed in many papers in scientific
literature for old movies film restoration. Since some authors[29] used spatiotemporal
information to restore scratches, many of them proposed static removal approaches
Chapter 5
65
processing a single frame, so these methods can be used to process also still images.
Kokaram[30] proposed a 2-dimensional autoregressive model to interpolate missing
information consistently with the local neighborhood, without using information from
adjacent frames. Bretschneider et al[31] proposed a technique based on wavelet
decomposition. Bruni et al[32] generalized the Kokaram’s model for scratch detection
on the hypothesis that scratch is not purely additive on a given image. The same
authors[33] also presented a method based on the Weber’s law to detect and restore
damages. Tegolo et al[34] proposed a detection method based on statistical information
extracted from the whole image, and adopted a genetic algorithm for the restoration
phase.
5.2 Scratch Features
Within the test photographic archive (2500 color images of digitized aerial photos of
Sicilian territory, with average resolution of 15000x15000), scratches occur as long
bright lines that run more or less horizontally along the image. These specific scratch
features are caused by the manual inspection of the photos negatives with a mechanical
device. They can affect the whole image, with or without interruptions, or only a part of
it. Typical line width values are 3-7 pixels vertically and a maximum slope value of 10
degrees is measured, with possible smooth changes of slope along the photo. Intensity
value of a scratch is different in darkest and brightest areas of the image. Lines
brightness is lower in darkest regions, because of the natural interpolation process of a
digital scanner.
5.3 The Proposed Method
The scratch removal problem could be divided into two sub-problems: detection and
restoration. The detection phase consists in searching defects which are not natural lines
in the image. The output of this process is a binary mask in which pixels are labelled as
good or damaged. This mask is strictly tied with the precision of the detection
algorithm. In fact, this region has to be neither too small, since it can still not contain
degraded pixel, nor too large, since good information of the image could be destroyed.
The restoration step has the purpose to reconstruct lost information using pixels close to
Chapter 5
66
the scratch. The key point for this phase is the choice of the appropriate neighbourhood.
The algorithm is based on a semiautomatic detection method and an unsupervised
restoration process.
5.4 Scratch detection
Test image dataset is corrupted by scratches like straight lines with a quasi horizontal
displacement, so their main feature is the orientation. Kass and Witkin [35] affirm that a
line is π/2 shifted in the frequency domain and suggest the use of a pass band filter to
select it. Indeed, the band-pass avoids illumination problems, due to low harmonics,
and noise corruption, due to high ones.
( ) n
vh Dv
Du
vuH 2**
414.01
1,
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛+⎟⎟
⎠
⎞⎜⎜⎝
⎛⋅+
=
(5.1)
where Dh and Dv are the two cutoff frequencies while u* and v* are the translated and
rotated frequency coordinates as follows:
( )( )
( ) ( ) ( )( )( ) ( ) ( )( )⎩
⎨⎧
+++⋅−=+++⋅=
⎩⎨⎧
⋅=⋅=
tyutxuvtyutxuu
centertycentertx
θθθθ
θθ
cossinsincos
;sincos
*
*
(5.2)
so that the center of the sub-band has tx, ty coordinates and it is rotated with the same
angle. The sub-bands must be symmetric with respect to the origin, so the angle θ must
be shifted of π.
The filter order n can control the slope of the sub-band, so that n=4 is chosen to
concentrate the filtering in a precise zone of the spectrum. This explains the use of this
bandpass filter instead of a Gabor one. A homomorphic filter is applied to enhance the
scratch and to produce a dark uniform background. The result after the bandpass and the
homomorphic filter can be seen in fig 5.1.b. Now a threshold is applied to the image
Chapter 5
67
(fig 5.1.c) with the aim to obtain a binary image containing the scratch position. The
threshold value is computed as mean plus standard deviation of the image intensity.
Once the input image has been pre-processed the Hough transform is applied to the
filtered binary image, to extract any relevant lines. Hough transform [36] maps each
pixel (x,y) into the parameter space (a,b), where a and b are the slope and intercept of a
generic line y = ax + b. Usually, a normal representation of the line is used, and the
parameter space is subdivided into a number of accumulator cells in order to reduce the
computational complexity. Each pixel in the binary image is processed and a counter in
the accumulator cell is incremented. The algorithm outputs the cell with the highest
score. The detected scratch line and the resulted binary mask are shown in fig. 5.1.d and
1.e. The method has proven to be robust to noise and suitable for the purposes.
5.5 Restoration phase
The proposed restoration method is a pixel-by-pixel filling process. It takes as input the
previous detected binary mask and uses pixels close to the mask to reconstruct lost
information.
a. original image
b. after bandpass and homomorphic filtering
c. after thresholding
d. detected line with Hough
e. binary mask f. restored image
Figure 5.1 Processing steps for scratch detection/restoration
Chapter 5
68
It can be divided into two steps:
- Estimation of the direction of the information propagation vector
- Pixel filling
5.5.1 Direction Estimation
A block-matching method is used to estimate the direction toward which information is
propagated from above to below the scratch area. In this step the brightness component
of the image is used. Pixels are processed in scan order. For each pixel into the scratch
mask, two rectangular areas are considered, one above and one below the mask. Their
positions, related to the pixel to fill, depend on the scratch mask width, while their size
depend on the gradient vector computed at the pixel-to-fill position. Within this two
areas, the method tests all the possible candidate vectors that link a block in the area
above the scratch with a block in the area below, centred in the pixel position. The
vector that minimizes the Sum of Absolute Differences (SAD) of the pixels in the two
end-point blocks is chosen as the most probable direction vector toward which
information is propagated .
( ) ( )
( )
⎟⎠⎞
⎜⎝⎛ +++++−
⎟⎠⎞
⎜⎝⎛ +−+−−=
=
∑∑
∑ ∑
−= −=
= −=
jkyiwkxp
jkyiwkxpSAD
kkyxSADdd
yx
i jyx
D
k
D
Dkyx
kkyx
x
x
y
yyyx
,2
,2
,,,minarg,
1
1
1
1
0,
(5.3)
where w is the scratch width, kx ky are the candidate direction vector components, Dx
and Dy the vertical and the horizontal size of the area into which matching is searched.
Mask width w is updated after each horizontal scan. Dx and Dy depend on the mask
width and on the gradient vector computed at the pixel-to-fill position. If the horizontal
component of the gradient is greater than the vertical one, Dy is set to a higher value.
Similarly for the vertical component. Blocks to be matched must be symmetrical with
respect to the pixel position, in order to avoid annoying distortion artefacts in the
reconstructed area. Blocks inside the lower part of the scratch mask are allowed to be
Chapter 5
69
used for matching. It has been observed that the damaged area holds some residual
information, so partial overlapping between blocks and scratch mask is useful to
recover that information. No pixel from the real scratch line is used, because no
matching can be found between upper blocks and blocks with the high brightness value
of the scratch.
5.5.2 Pixel filling
The pixel filling phase works in the three RGB color channels. It uses the estimated
direction vector to find below and above the scratch two pixels, for each color channel,
along that direction, with the same distance from the pixel to fill. The value to be
assigned to the pixel, for each channel, is the median value between this two points and
a further value, computed as the average of the two vertically closest pixels outside the
scratch mask:
( ) ( )
( ) ( )( )2
,,1
,,,,2
,2
ywxpyxpv
vdownupmedianyxpdywdxpdown
dywdxpup
xx
yx
++−=
=′⇒⎟⎠⎞
⎜⎝⎛ +++=
⎟⎠⎞
⎜⎝⎛ −−−=
(5.4)
dx and dy are the components of the estimated direction vector. If up and down are
similar, the new pixel value is assigned as one of them. If they are much different, new
value is computed with a simple vertical interpolation, introducing no artefacts but only
few blurring.
Finally, a median filter is applied to the edge pixels of the scratch mask, to remove
some residual artefacts.
Chapter 5
70
5.6 Experimental Results
The algorithm has been tested on a subset of 105 cropped images from a photographic
archive composed of 2500 aerial photos of the Sicilian territory. It has been
implemented in Matlab for the detection step, and in ANSI-C for the restoration phase.
a.original image
b. detected scratch
c.restored image
d. original image
e. first detected scratch
f. partially restored image
g. second detected scratch
h. restored image
Figure 5.2 Some experimental results
Chapter 5
71
Tests measured a percentage of 85,7% of line scratch correct detection and an average
execution time, for the single detection step, of 1,2 sec per scratch (maximum crop size
500x500).
The method works well to reconstruct straight or curve lines, with no false edges,
chromatic aberration or distortion. Some blurring is introduced where scratch lines
cross areas which have much different information below and above the scratch. Figg.
5.1.c, 5.1.f and 5.2 show some of the results. An average restoration time of much less
than 1 sec per scratch is measured. No objective measure is possible to evaluate the
restoration quality of reconstructed images, because there is no reference undamaged
image to compare with.
Results are comparable with those obtained with commercial restoration tools, but
much less user intervention is required (only confirmation and mask width setting).
5.7 Conclusions
This chapter presented a fast and effective method to detect and remove scratches in
still images. It is compose by a semiautomatic detection method, based on band-pass
filtering and Hough transformation, to detect candidate scratches. This method needs
the user to confirm the candidate scratch and to set the scratch mask width. Restoration
phase reconstruct lost information in automatic way. The direction toward which
information is propagated into the scratch mask is estimated with a block-matching
method. Pixels in the mask are filled with information along the estimated direction.
This method has been tested on a photographic archive composed by aerial photos
concerning Sicilian territory. Results, comparable with those obtained manually with
some commercial tools, are obtained as discussed in the section on experimental results.
Actually I plan to extend the method to process scratches with any orientation, and to
develop a fully automatic batch procedure for scratch removal in huge photographic
archives.
Chapter 5
72
Acknowledgements
This work has been partially funded by the MIUR (Italian Ministry of Education,
University and Research) project FIRB 2003 D.D. 2186 - Ric December 12th 2003. The
authors wish to thanks the SAS s.r.l. in Palermo for having permitted the use of their
photos in this research.
Chapter 6
Texture Synthesis Restoration within the Bit-Plane
Representation
In this chapter a new methodology is proposed for handling the problem of restoration
of greyscale textured images. The purpose is to recovery missing data of a damaged
area. The key point is to decompose an image in its bit-planes, and to process bits rather
than pixels. Two texture synthesis methods for restoration are proposed. The first one is
a random generation process, based on the conditional probability of bits in the bit-
planes. It is designed for images with stochastic textures. The second one is a best-
matching method, running on each bit-plane, that is well suited to synthesize periodic
patterns. Results are compared with a state-of-the-art restoration algorithm.
6.1 Introduction and Related Works
Filling-in gaps in a digital image, often known as digital inpainting, is one of the most
active fields in image processing research. Restoration of damaged or unknown areas in
an image is an important topic for applications as: image coding (e.g. recovering lost
blocks); removal of unwanted objects (e.g. scratches, spots, superimposed text, logos);
video special effects; 3D texture mapping. There are two different main approaches for
a filling-in problem in literature: PDE (Partial Differential Equation) methods, and
constrained texture synthesis.
PDE methods give impressive results with natural images but introduce blurring, that is
more evident for large regions to inpaint. Bertalmio et al.[37] pioneered a restoration
algorithm based on a 3rd order PDE model. It was the first time the term “inpainting”
was used for a digital image processing application. An earlier 2nd order PDE
Chapter 6
74
inpainting model was proposed by Masnou and Morel[38] for a disocclusion problem in
computer vision. Olivera et al.[39] proposed a faster inpainting method. Missing region
is repeatedly filtered with a 3x3 convolution mask to diffuse known information to the
unknown pixels. Chan and Shen[40] proposed a Curvature-Driven Diffusion model
based on Euler-Lagrange equation. These methods are computationally expensive and
not suitable for textured images.
Texture synthesis methods reconstruct an image from a sample texture. For inpainting
purposes, region to fill-in is the area into which synthesize the texture, and information
to replicate is taken from the surrounding pixels. Most of these methods use Markov
Random Fields as theoretical model[41] to represent a texture. That is, for each pixel,
color (or brightness) probability distribution is determined by a limited set of its
surrounding pixels. Heeger and Bergen[42] proposed a method which synthesizes
textures by matching histograms of a set of multi-scale and orientation filters. Portilla
and Simoncelli[43] proposed a statistical model based on a wavelet decomposition.
Efros and Leung[44] synthesized one pixel at time, matching pixels from target image
with the input texture. Their “image quilting” technique[45] used constrained block-
patching for the synthesis process. Wei and Levoy[46] proposed a multi-resolution
texture synthesis algorithm, based on gaussian pyramids decomposition. Kokaram[47]
proposed a 2D autoregressive statistical model for filling-in and texture generation.
Criminisi et al.[48] proposed an hybrid “exemplar-based” method for removing large
objects from digital images. All these methods are extremely time consuming and many
of them failed to reconstruct highly-structured texture.
A new approach is proposed to recover damaged information in textured images.
Images are processed within a simple domain, the bit-plane representation. Two texture
synthesis methods are proposed for the restoration problem. The first one is based on a
conditional random generation process, and has been designed for images with
stochastic textures. The second one is designed for textures with a periodic pattern. The
purpose is to fill-in gaps of an image using surrounding information.
Chapter 6
75
6.2 The bit-plane representation
Bit-plane slicing is a well known technique used to represent the content of a greyscale
image. It is mostly used for application in the fields of digital watermarking [49][50]
and image compression [51][52]. Garcia et al.[53] proposed a method based on bit-
plane splitting to classify greyscale textured images.
The key point of the two proposed methods is to observe features of a damaged image
in a simple domain, the bit-plane representation. Image is split in its bit-planes, with a
bit-plane slicing decomposition, and Gray-coding is applied, to decorrelate information
between different planes. Both the proposed restoration methods work with bits in the
bit-plane space, rather than with pixels of the image. Planes are processed from the
most significant to the less, and at each step restoration depends on the previously
restored planes. They cannot be processed separately, since annoying artefacts would be
b) plane 7
c) plane 6
a) whole image
d) plane 1
e) plane 0
Figure 6.1 Image bit-plane decomposition. (a) original image, (b-c) most significant, (d-e) less significant bit-planes. Most significant bit-planes are more structured than less significant ones. Lower planes are quite similar to pure noise.
Chapter 6
76
visible into the reassembled image. Gray coding helps to decorrelate planes, but this
solution is not enough to avoid distortions after reassembling the image. Many are the
advantages in terms of efficiency:
- bit information can be stored in a first step (analysis) and used in the second step
(synthesis) (see subsection 6.4.1). To my knowledge, none of the related works
proposed a method to store pixel statistics, because it is an hard task, both for memory
b. bit-plane 7
c. bit-plane 6
d. bit-plane 5
e. bit-plane 4
f. bit-plane 3
g. bit-plane 2
a. original
h. bit-plane 1
i. bit-plane 0
Figure 6.2 Damaged image (spot) and its bit-planes decomposition (black means bit 0, white 1)
Chapter 6
77
usage and access time problems. Typically a search for the needed information is
recomputed at each step of the restoration process, with a waste in execution time.
- working with bits is faster and simpler than working with pixels. At each step the
output of the restoration process is simply a binary value (or mask) instead of a pixel
value (or mask);
- since most part of information is stored in the most significant planes, lower planes
can be processed roughly, (e.g. using smaller windows), speeding-up the process,
- without losing quality in the restored image. Furthermore, it has been noted that, for
natural damaged images, with no superimposed damage, defects are not visible in the
lower planes(see fig.6.2). Less significant planes therefore can be not processed,
speeding-up the execution time of the algorithm.
6.3 Restoration methods
Two texture synthesis based approaches are discussed in next sections. The adopted
texture model is based on the Markov Random Field theory[41], since it has proven to
be satisfactory in representing a wide set of texture types. Textures are seen as instances
of a stationary random model, in which each pixel is statistically determined by its
neighbourhood.
Both the two approaches don’t focus on automatic damage detection. The user must
select the area to restore, to create a binary matrix, in which all the pixels are labeled as
good or damaged, used as an input for the algorithms.
The next two sections provide a detailed description of the two proposed methods.
6.4 The conditional random generation method
The first proposed method is a generative process, based on bits statistics in the bit-
planes. Once the image is decomposed in the bit-plane representation, two are the steps
of the algorithm:
- Information analysis
- Reconstruction
Chapter 6
78
An evaluation of the algorithm computational cost is given in the subsection 6.4.3.
6.4.1 Information Analysis
The purpose of this step is to build a dictionary to store uncorrupted information, which
will be used in the reconstruction step.
A square window WN (where N is the window size set by the user) runs along each bit
plane. Bit-planes are processed from the most significant to the less. For each
undamaged bit in a bit-plane, an index is created with the scan-ordered bit sequence
inside the window WN. Similarly, another index is created with bits in the previous
significant bit plane, with an M-size square window set in the same position, and added
as a header to the first index:
( ) ( )( )
( )( )∑∑∈
⋅
∈
+ ⋅+⋅⎥⎥⎦
⎤
⎢⎢⎣
⎡⋅=
+ iNll
iMjj Wyx
lll
iNN
Wyx
jjj
i yxbyxbyxk,,
1 2,22,,1
(6.1)
where bi(x,y) ̀ are bits from the current bit-plane, bi+1(x,y) ̀ are bits from the previous
significant one. The frequencies of these sequences into the bit-planes are stored in a
histogram, which is the “dictionary”. Each value is an estimation of the a posteriori
probability of a bit sequence in a i-plane, conditioned by the corresponding sequence in
the previous (i+1)-plane. According to the Markov Random Field hypothesis, it is
supposed that this estimation is equal to the conditional probability value:
( ) ( )1|, += iM
iN WWPkiH . (6.2)
The most significant plane is processed as a special case, with no contribution from a
previous plane.
6.4.2 Reconstruction
According to the 2D-Wold decomposition model for homogeneous random
fields[54][55] textures can be decomposed into a deterministic and a purely
indeterministic components. The most important features for human texture perception
are: periodicity, directionality and randomness. Two competing processes work to
reproduce these features from the global image into the damaged area: a bit-by-bit
Chapter 6
79
constrained random generation process, which aims to reproduce texture directionality
and randomness of the global image, and a patching process, to replicate texture
periodicity.
As a preliminary remark, note that results are strongly affected by the order into which
pixels (or bits) are synthesized, because it sets the neighbourhood used to reconstruct
the damaged area. With a simple scan order the restoration process tends to reproduce
up-to-down left-to-right diagonal shapes. The algorithm processes bits along a direction
that depends on image average gradient vector. This solution helps us to reconstruct the
natural bias of the image.
The reconstruction phase is the dual process of the dictionary building process. A N
square window runs on the damaged area of each plane. As in the previous step, planes
are processed from the most significant to the less. For each damaged bit of a plane, the
corresponding window will contain uncorrupted, corrected and damaged bits. A M
square window is considered in the previous plane, at the same position. The whole
information is known for this window (bits are either undamaged or corrected).
The bit-by-bit generation process computes the probability that the central bit of the
window is 1 or 0, given the known neighbour bits in the plane and the bits in the
previous plane. The statistics of each sub-mask of a window can be computed building
up those of all the possible statistics of the windows which share that sub-mask:
(6.3) ( ) ( )( ) ( )
{ }iNi
Ni
ki
ki
j
ci
Mi
jci
Mi
N
ci
Mi
jci
Mi
N
WWWWW
bWWPbWWP
bWWPbWWP
ˆˆ|
1,|1,|ˆ0,|0,|ˆ
11
11
=∩∈
===
===
∑∑
++
++
.
(6.4)
The two needed statistics:
(6.5) ( ) ( )[ ] ( )
( ) ( )[ ] ( )∑
∑−
=
+
−
=
+
===
===
12
0
11
12
0
10
1,|ˆ,,,
0,|ˆ,,,
DN
DN
pc
iM
iNp
ii
pc
iM
iNp
ii
bWWPyxWiHyxS
bWWPyxBiHyxS
(6.6)
Chapter 6
80
where H[i,k] is the dictionary built in the analysis phase, ND is the number of the
damaged bits in the window, Bip is the index for the sequence with a “black” (zero)
central bit in the window, and Wip is the sequence with a “white” (one) central bit, bc is
the central bit of the mask in the i-plane. Both of these indexes contain bits from the ŴN
sub-mask.
The next step is a random generation, conditioned by the statistics computed in eq.6.5
and eq.6.6, in order to choice which information (0/1) to put in the central position of
the window. The two statistics are weighted by weights that depend on an user-defined
parameter α :
( )( )α,,max, 101,010
10
111
10
000
PPfwwSS
SwPSS
SwP
=+
⋅=+
⋅=
.
(6.7)
By setting α close to 1, this process is the same as a random process with the two
probabilities:
( )( ) ( )
( )( ) ( )yxSyxS
yxSPyxSyxS
yxSP ii
i
ii
i
,,,
,,,
01
11
01
00
+=
+=
(6.8)
which fits for synthesizing highly stochastic textures. As α increase, the bit value is
chosen as the central bit of the most frequent window with those surrounding
conditions. That is suitable for strongly oriented textures. In this way the method can
control the randomness and directionality of the generated texture.
To avoid the “growing garbage” problem, if no statistics match the current sequence in
the dictionary, a random generation process is used with the following probabilities:
( ) ( )( )( ) ( )( )yxbyxbPP
yxbyxbPPii
ii
,|1,,|0,
11
10
+
+
====
. (6.9)
At the same time, a second competing process works to propagate global texture
features into the area to restore. A patching process aims to reproduce texture
periodicity. For each damaged bit, the two most frequent sequences (one with 0 as its
Chapter 6
81
central bit, one with 1), which share the known-bits sub-mask, are extracted from the
dictionary:
(6.10)
(6.11)
( ) ( )( ) ( ){ }iN
iN
ik
ik
ij
ci
Mi
ji
ci
Mi
ji
WWWWW
bWWPyxW
bWWPyxW
ˆˆ|
1,|maxarg,
0,|maxarg,1
max1
1max0
=∩∈
==
==+
+
.
If one of the statistics is much greater than the other, the bit-by-bit generation process is
disabled, for the current step, and the whole window is filled in with the most frequent
sequence. The activation threshold of this process, that is the meaning of “much
greater”, is set by an user defined parameter. As discussed in this section, less
significant bit-planes have a more random global structure. Therefore patching is
useless or harmful to process these planes, and it is disabled. Filling-in the whole
window rather than bit-by-bit extremely speeds up the execution time, and it helps in
replicating texture periodicity, if it is at a scale either equal or smaller than the window
size.
After all the planes are restored, they are merged to reconstruct the whole image, and a
soft edge-preserving smooth filter is applied to remove the residual high-frequency
noise due to this reassembling phase.
6.4.3 Computational Cost
Computational cost depends on damaged area size and on the windows size:
( ) ( )MMNNS
dOdnO TS
×+×=⋅+− −2 (6.12)
where d is the damaged pixels number, n is image size, and T is the table index size.
The first term of eq. 6.12 results from the dictionary building phase. It also depends on
windows size. The second term is the computational cost of the reconstruction phase.
Exponential term is due to the structure that is used to store information in the analysis
step. The dictionary is stored in a hash table, with collision lists, which is the best
solution to speed-up the access time. T is the table size. If d<<n and windows are small,
first term is predominant and computational cost is O(n). Increasing M, N and d,
Chapter 6
82
computational cost becomes exponential in the worst case, that is much far from the real
execution time measured within the experiments.
6.5 The best matching method
The second proposed algorithm is a best matching method. Many are the contact points
between the two methods:
- they both work on bit-planes;
- they need an input binary matrix, with the position of the damaged bits;
- planes are processed from most significant to the less;
- for each plane, bits are processed along a direction that depends on the average
gradient vector.
The algorithm is much simpler than the first one. For each bit-plane, a window WN runs
along the damaged area, and a corresponding window WM, set in the same position,
runs along the previous significant plane. For each damaged bit in each bit-plane the
algorithm creates a word wi(x,y) of bits, as in eq.6.1, using both information from the
current and the previous significant bit-plane. The most similar word, according to the
Hamming distance, is searched in a neighborhood (neighborhood size is set by the
user). Only known bits are considered for matching. Finally, bits from the best
matching word are used to replace the unknown bits of the word wi(x,y). Once all the
planes are restored, they are merged to reconstruct the whole image. Computational cost
is linear with neighborhood and window sizes, and with number of damaged bits.
6.6 Experimental results
Algorithms have been implemented in ANSI-C, and executed on an Intel Core Duo PC
(1,83 GHz, 2 GB RAM).
Tests has been made on two different image dataset.
Chapter 6
83
The first testing dataset is composed by about 20 images from the photographic archive
of the Alinari Archives in Florence (see previous chapters) . Only the CRG method is
tested on this dataset, because the second one, which is designed for highly-textured
images, doesn’t work well with these, poor-textured, images.
a)
b)
spots lacuna STATS good pics damaged restored good pics damaged restored
mean 165,269 163,4415 165,183 184,7088 186,3909 183,9592 std dev 3,9103 7,9855 4,1024 13,4507 14,9383 13,5544 skew 0,1154 -3,8227 -0,066 -0,8785 -0,4826 -0,923
kurtosis 3,3652 34,2812 4,1986 8,4864 6,2318 8,0128 size 112x94 284x162
def % 16,8 16 dict build
time 0,03 sec 0,16 sec restoration
time 0,06 sec 0,29 sec
c)
foxing STATS good pics damaged restored restored(5 p)
mean 189,9209 184,9979 1.899.977 189,9877std dev 7,1406 11,338 7,2012 7,3315skew 0,27544 -0,8026 0,235 0,2357kurtosis 3,6515 5,404 3,8162 3,7046size 1663x482
def % 32,3 dict build time 2,2 sec 1.9 sec restoration time 7,6 sec 4.8 sec
Figure 6.3 – Experimental results for typical defects in old photos a) spots, b) lacuna c) foxing. Statistical parameters are listed for: uncorrupted pixels in the original image, the whole original image, the restored image. Two restored foxing images: 8 planes and most significant 5 planes processed. No sensible differences in statistical parameters, less execution time
Chapter 6
84
Figure 6.3 shows some results of the algorithm on a set of damaged images. Significant
statistical parameters are provided in order to compare images before and after
restoration. Statistical features for restored images are very similar to those from
undamaged pixels. In addition, the proposed method does not introduce blurring, and
(d)
(a)
(e)
stats original damaged CRG method Criminisi
m 76,9864 77,6598 76,9071 76,9819
σ 64,4647 65,2693 64,4709 64,4647
s 1,0717 1,0745 1,0704 1,0716
k 2,7908 2,8105 2,7889 2,7906 S/N 22,7384 24,7963 29,5066
d 1552 (0,38%)
an. 4,5 t(s) syn. 3,4
29.6
(f)
(b) (g)
stats original damaged CRG
method Criminisi
m 130,8099 122,0571 130,9902 130,7417
σ 61,2621 67,3885 60,8456 61,6086
s 0,0749 0,0396 0,0772 0,0673
k 1,7097 1,8936 1,7085 1,7180 S/N 4,4710 8,8456 8,5823
d 28626 (7%)
an. 1,5 t (s) syn. 1,1 618,5
(h)
(c) (i)
stats original damaged CRG
method Criminisi
m 35,0554 36,0589 34,8039 35,0759
σ 35,8527 39,9498 35,5104 35,8855
s 3,46605 3,4968 3,4508 3,4582
k 15,3146 15.5208 15,1343 15,2490 S/N 6,2192 17,0377 22,9851
d 3091 (0,75%)
an. 1,8 t(s) syn. 9,2 213.9
Figure 6.4 (a-c) corrupted images (D21, D9, D25 from the Brodatz set, with superimposed damages ); (d-i) restored images (detail from the reconstructed zone) with CRG method (d,f,h) and with Criminisi inpainting algorithm (e,g,i). A set of significant statistical parameters is provided to compare the two methods: m= mean, σ= standard deviation, s= skewness, k= kurtosis. d= number of damaged pixels. S/N (dB)= Signal to noise ratio between original-damaged and original-restored images. Execution time is shown for both analysis and synthesis phase (CRG method) and for the whole Criminisi process.
Chapter 6
85
even the finest granularity is reconstructed. With respect to efficiency, an execution
time much lower than 1s is measured for small sized images,. Experimental results
show that to reconstruct large defects the algorithm takes no more than 10 s.
The second test dataset is made of on over 40 640x640 grayscale images from the
Brodatz texture set. Both stochastic and periodic texture are used for tests. Each image
is arbitrarily damaged to create an area to fill-in.
Figure 6.4 shows some results obtained with the CRG algorithm, compared with those
obtained with the Criminisi inpainting algorithm[48]. Both visual and numerical
comparison are provided. Visual comparison shows that results are very similar to those
obtained with Criminisi algorithm. For a quantitative evaluation of the results, a set of
significant statistical parameters has been measured. No remarkable differences in
statistical features measured for the two methods (with respect to the parameters of the
original image). Only some difference in the S\N parameter for small area to fill. This
can be explained by considering that the Criminisi algorithm is based on a patching
method. CRG method, on the other hand, is one or two order of faster than the
Criminisi method, depending on the damaged area size. Execution time is about 1 sec,
for stochastic texture and small holes, and rises up to 5-6 minutes, for highly-structured
textures and large-sized holes, processed with larger-sized masks.
Figure 6.5 shows some results for the second proposed method, compared to those
obtained with the Criminisi algorithm. Visual comparison shows that CRG method
gives impressive results, comparable to those obtained with Criminisi. For the
quantitative evaluation, very little differences are measured in statistical parameters, but
a higher signal-to-noise ratio gain. Execution time is about a half of the time measured
to execute the Criminisi algorithm.
6.7 Remarks and limitations
Experimental results showed that the two methods are complementary, in the sense of
the typology of textures they are suited for.
Chapter 6
86
(a)
stats original damaged BM method Criminisi
m 38,0926 37,5912 37,9622 38,1038
σ 32,5217 32,5940 32,5307 32,5152
s 3,6218 3,5787 3,6185 3,6231
k 18,5613 18,4199 18,5372 18,5789 S/N 15,0393 20,0748 18,0450
d 5315 (1,3%)
t (s) 24,39 39,62
(b)
stats original damaged BM method Criminisi
m 165,2731 165,7008 165,1780 165,2943
σ 42,9374 43,2738 42,9035 42,9339
s -1,3630 -1,3163 -1,3665 -1,3634
k 5,1276 5,0862 5,1336 5,1276 S/N 15,9187 28,3516 28,9309
d 2293 (0,99%)
t(s) 16,51 32,61
(c)
stats original damaged BM method Criminisi
m 147,7488 146,8935 147,6906 147,7991
σ 64,0724 64,8224 64,0326 64,0907
s -0,3168 -0,3307 -0,3179 -0,3171
k 1,6361 1,6833 1,6392 1,6361 S/N 14,2774 22,0735 20,2075
d 3190 (0,78%)
t(s) 23,42 42,02
Figure 6.5 (a-c) corrupted images (D34, D1, D35 from the Brodatz set, with superimposed damages ); (d-i) restored images (detail from the reconstructed zone) with BM method (d,f,h) and with Criminisi inpainting algorithm (e,g,i). Statistical parameters and signal to noise ratio are provided to evaluate the quality of the results of the two methods. Execution time is measured to compare efficiency.
The conditional random generation process works well with stochastic textures. The
most evident limitation of this approach is the window sizes. There are two problems
with large-sized windows: the larger the window, the higher the execution time is; the
larger the window, the less consistent the statistics stored in the dictionary is. Note that
to create consistent statistics hole size must be much lower than image size, which is
Chapter 6
87
usually the case in real-world images. Therefore tests have been made with a maximum
window size of 7x7. This is not a problem for processing stochastic texture (a 3x3
window performs well). Textures that have periodicity in larger scale are harder to
reconstruct. Note that only the most significant bit-planes need larger windows. Lower
planes have more random structure, and if higher planes are well-reconstructed, they
can be restored using smaller windows.
The best matching method gives impressive results with textures with periodic patterns,
no matter what the scale of the periodicity is. It takes the most similar information
outside the damaged area, and replicates it into the gap. That’s the reason why it works
well with periodic patterns. For stochastic textures, note that using bits rather than
pixels, the probability to have more than one best-matching word is higher, above all
for small-sized windows. Therefore, the way to decide which best word to choice is a
critical point. The risk is to choice always the same sequence, and to have excessive
repetition of the patches, that is an evident artifact in stochastic textures. A method to
select the best word in case of more than one candidate is an open issue.
6.8 Conclusions and future works
Working with bits is faster and simpler than processing pixels. This is the key point of
the presented approach. Image is decomposed in its bit-plane representation. Two
methods, working on bit-planes, are proposed to process a wide set of texture types: a
conditional random generation process and a best matching method. The first method
gives better results with stochastic textures. The second one works well with periodic
patterns. Experimental results showed that efficiency is improved, in respect of related
works, with no loss in visual quality.
In the future I plan to study a method to automatically select the best method for an
image, and to eliminate dependence from the user-defined parameters. Texture features
could be estimated during a pre-analysis phase, and parameters suggested for the
restoration process.
Conclusions
Digitization is the definitive solution to preserve historical images and their contents
against time and careless conservation. Digital copies last almost forever, since they can
be used and duplicated without losing quality. Furthermore, digital restoration
techniques are used to recover lost information and to take images back to their original
state.
Professional digital restorers often use commercial software, like Adobe Photoshop, but
this kind of restoration is heavily user-guided because the defects are subjectively
detected and the type of correction is user-selected too.
This dissertation presented the problems related to automatic digital restoration of old
photos. The purpose is to create a useful tool to support non-expert users in the
restoration process of damaged images.
Within the aims of the Italian scientific project, in the context of which I developed my
research work, obtained results are:
- A digital restoration model for defects in historical images, inspired by the process
of manual professional restorers.
- A knowledge base to represent elements of the restoration process model. Images,
degradations, descriptions and restoration paths are objects of a database. The best
restoration typology, for each degradation, is derived from the relationships between
objects in the database. Preliminary results are presented in:
ARDIZZONE E, DINDO H, MANISCALCO U, MAZZOLA G. (2006). Damages of Digitized Historical
Images as Objects for Content Based Applications. European Signal Processing Conference (Special
Session on Cultural Heritage) 2006. Firenze, Italia. September 4 - 8 2006. (pp. CD-ROM).
- A prototypal software tool to support non-expert users in the restoration process,
and to retrieve useful contents from the database.
- A taxonomy for defects by which old photos are affected. A dual taxonomy is
proposed for both real defects and their digitized version.
89
- A set of low level descriptors for images affected by foxing. A classification
application, based on these descriptors, has been implemented.
- A fast and effective application to detect and remove quasi-horizontal scratches
from still images, made of a supervised detection method and an automatic restoration
algorithm. Method and results presented in:
E. ARDIZZONE, H. DINDO, O. GAMBINO, MAZZOLA G. (2007). Scratches Removal in Digitised
Aerial Photos Concerning Sicilian Territory. 14th International Conference on systems, Signals and
Image Processing IWSSIP 2007. Maribor, Slovenia .27-30 June 2007. (pp. 411-414). ISBN/ISSN: 978-
961-248-029-5/07.
- A methodology to restore gaps in textured images within the bit-plane
representation. Two texture synthesis methods are proposed: a conditional random
generation method for images with stochastic texture, and a best matching method for
periodic textures. Approach and results discussed in:
ARDIZZONE E, DINDO H, MAZZOLA G. (2008). Filling-In Gaps In Textured Images Using Bit-Plane
Statistics. Third International Conference on Computer Vision Theory and Applications 2008. Funchal,
Madeira, Portugal. January 22-25 2008. (in press)
ARDIZZONE E, DINDO H, MAZZOLA G. (2007). Texture Synthesis for Digital Restoration in the Bit-
Plane Representation. The Third IEEE International Conference On Signal-Image Technology &
Internet–Based Systems 2007. Shangai, China. December 16-19 2007. (pp. CD-ROM)
ARDIZZONE E, DINDO H, MAZZOLA G. (2007). Restoration of Digitized Damaged Photos using Bit-
Plane Slicing. IEEE International Conference on Multimedia and Expo 2007. Beijing, China. July 2-5
2007. (pp. 1643-1646). ISBN/ISSN: 1-4244-1017-7. doi:10.1109/ICME.2007.4284982.
References
[1] P. Paolini, N. Di Blas, F. Alonzo, “ICT PER I BENI CULTURALI: ESEMPI DI APPLICAZIONE”, MONDODIGITALE n.3 - September 2005
[2] Decreto Legislativo 22 Gennaio 2004, n. 42. "Codice dei beni culturali e del paesaggio, ai sensi dell'articolo 10 della legge 6 luglio 2002, n. 137" Gazzetta Ufficiale n. 45 del 24 febbraio 2004 - Supplemento Ordinario n. 28
[3] A. Piva, R. Caldelli, “Tecniche di marchiatura elettronica per la protezione dei diritti di proprietà intellettuale”, Elettronica e Telecomunicazioni, Agosto 2000, pp. 3-21.
[4] “European Content in Global Networks Coordination Mechanisms For Digitisation Programmes - The Lund Principles: Conclusions Of Experts Meeting”, Lund, Sweden, 4 April 2001.
[5] “Quality Principles for cultural Web sites: a handbook” http://www.minervaeurope.org/
[6] “Handbook for quality in cultural Web sites: improving quality for citizens” http://www.minervaeurope.org/
[7] “Good practice handbook Identification of good practices and competence centres” http://www.minervaeurope.org/
[8] “Technical Guidelines for Digital Cultural Content Creation Programmes” http://www.minervaeurope.org/
[9] http://www.michael-culture.org/it/home
[10] http://prestospace.org/
[11] Y. Rui, T.S. Huang, "Image Retrieval: Current Techniques, Promising Directions and Open Issues", Journ. of Visual Communication and Image Representation, 10, 4, Apr. 1999.
[12] T.-S. Chua, L.-Q. Ruan, "A Video Retrieval and Sequencing System", ACM Transactions. On Information Systems, 13,4, 373-407, Oct. 1995.
[13] A. Del Bimbo, "Visual Information Retrieval", Morgan Kaufmann Pu., 1999.
[14] J.M. Martinez, MPEG-7 Overview, ISO/IEC JTC1/SC29/WG11N5525, March 2003.
[15] J.R. Smith, "Video Indexing and Retrieval Using MPEG-7", in B. Furht, O. Marques, Handbook of Video Databases, CRC Press, 2003, 625-650.
[16] C. Zhang, T. Chen, "An Active Learning Framework for Content Based Information Retrieval", IEEE Trans. on Multimedia , 4, 2, Jun. 2002, 260-268.
[17] Y. Ishikawa, R. Subramanya, C. Faloutsos, "Mindreader: Query Database through Multiple Examples", in Proceedings. of the 24th VLDB Conference., New York, 1998.
[18] J. Hunter, "An Overview of the MPEG-7 Description Definition Language (DDL)", IEEE Transactions on Circuits & Systems for Video Technology , 6, July 2001, 765-772.
[19] N. Sebe, M.S. Lew, D.P. Huijsmans, "Toward Improved Ranking Metrics", IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 10, Oct. 2000, 1132-1143.
[20] R. C. Gonzales, R. E. Woods, “Digital Image Processing”, Prentice Hall, second edition 2002.
91
[21] A De Polo, F. Stanco, G Ramponi, “Towards the Automated Restoration of Old Photographic Prints: A Survey”, In Proceedings of IEEE EUROCON 2003, pp. 370-374, Ljubljana, Slovenia, September, 2003.
[22] T. Sikora, “The MPEG-7 Visual standard for content description - an overview”. IEEE Trans. on Circuits and Systems for Video Technology, special issue on MPEG-7, vol.11 pp:696--702, June 2001.
[23] A.Yamada, M. Pickering, S. Jeannin, L. Cieplinski, J.- R. Ohm, and M. Editors, “MPEG-7 Visual Part of Experimentation Model Version 8.0”, Eds., Oct. 2000, ISO/IEC JTC1/SC29/WG11/N3673.
[24] G. Pass, R. Zabih and J. Miller, “Comparing Images Using Color Coherence Vectors”', ACM Conference on Multimedia, Boston, Massachusetts, Nov. 1996, pp. 65-74.
[25] F. S. Abas, and K. Martinez, “Craquelure Analysis for Content-Based Retrieval”, In 14th International Conf. on Digital Signal Processing, July 1-3, 2002, Santorini, Greece.
[26] A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain, “Content-based image retrieval at the end of the early years,” IEEE Trans. Pattern Anal. Machine Intell., vol. 22, pp. 1349–1380, Dec. 2000.
[27] ] T. D. Beckwith, W. H. Swanson, and T. M. Iiams, “Deterioration of Paper: the Cause and Effect of Foxing”, University of California Publications in Biological Science 1(13):299-356, 1940.
[28] F. Stanco, L. Tenze, and G. Ramponi, “Foxing in Vintage Photographs: Detection and Removal”, in European Conf. on Signal Processing, pp. 493-496, Sept. 6-10, 2004 Vienna, Austria.
[29] L, Joyeux, O. Buisson, B. Besserer, S. Boukir, “ Detection and removal of line scratches in motion picture films”, CVPR'99, International conference on computer vision and pattern recognition, Fort Collins, Colorado, June 1999.
[30] A. Kokaram. “Detection and removal of line scratches in degraded motion picture sequences”, in Signal Processing VIII, volume I, pages 5-8, September 1996.
[31] T. Bretschneider, O. Kao, P. Bones "Removal of vertical scratches in digitised historical film sequences using wavelet decomposition", in Proceedings of Image and Vision Computing, New Zealand, pp. 38-43, 2000
[32] V. Bruni, D. Vitulano, "A Generalized Model for Scratch Detection", IEEE Transactions on Image Processing, Vol. 13, No. 1, January 2004.
[33] V. Bruni, D. Vitulano, A. C. Kokaram “Fast Removal of Line Scratches in Old Movies”, ICPR (4) 2004: 827-830
[34] D. Tegolo, F. Isgrò, "A Genetic Algorithm for Scratch Removal in Static Images", in Proceedings of ICIAP 2001, pp. 507- 511, 2001.
[35] M. Kass and A. Witkin “Analysing oriented patterns Computer Vision", Graphics and Image Processing, 37 (1987) 362-397
[36] R. O. Duda , P. E. Hart, “Use of the Hough transformation to detect lines and curves in pictures”, Communications of the ACM, v.15 n.1, p.11-15, Jan. 1972
[37] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” in Proceedings SIGGRAPH, 2000.
[38] S. Masnou, J.M. Morel, “Level Lines based Disocclusion,” Proceedings of the 1998 IEEE International Conference on Image Processing, pp 259 - 263, 1998.
92
[39] M. Oliveira, B. Bowen, R. McKenna, and Y. -S. Chang, “Fast Digital Image Inpainting,”, in Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), pp. 261-266, Marbella, Spain. September 3-5, 2001.
[40] T. F. Chan and J. Shen. “Mathematical models for local non-texture inpaintings,” SIAM Journal of Applied Math., 62(3):1019–1043, 2002
[41] A. C. Cross, and A. K. Jain, “Markov Random Field Texture Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence 5: 25–39, 1983.
[42] D. J. Heeger and J. R. Bergen, “Pyramid-based texture analysis/synthesis,” in IEEE International Conference on Image Processing, October 1995, pp. 648–651.
[43] J. Portilla and E. P. Simoncelli, “A Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients,” International Journal of Computer Vision, 40(1):49–71, December 2000.
[44] A. Efros and T. K. Leung, “Texture synthesis by non-parametric sampling,” in Proceedings of the IEEE International Conference on Computer Vision, September 1999, vol. 2, pp. 1033–1038.
[45] A. A. Efros and W- T. Freeman, “Image quilting for texture synthesis and transfer,” in Proceedings SIGGRAPH, 2001, pp.341–346.
[46] L. Wei, M. Levoy, “Fast Texture Synthesis using Tree-structured Vector Quantization,” in Proceedings of SIGGRAPH, 2000 pp 479-488.
[47] A. Kokaram , “Parametric texture synthesis for filling holes in pictures,” in Proceedings of International Conference on Image Processing, 22-25 Sept. 2002 vol. 1 pp. 325-328.
[48] A. Criminisi, P. Perez, K. Toyama. “Region Filling and Object Removal by Exemplar-Based Inpainting” IEEE Transactions on Image Processing 13(9): 1200-1212, 2004.
[49] S. Katzenbeisser and F. A. P. Petitcolas, Information Hiding Techniques for Steganography and Digital Watermarking. Norwood, MA: Artech House, 2000.
[50] R. Z.Wang, C. F. Lin, and J. C. Lin, “Image hiding by optimal LSB substitution and genetic algorithm,” Pattern Recognition., vol. 34, pp. 671–683, 2001.
[51] N. P. Galatsanos, S. S. Yu, “Binary Decompositions For High-Order Entropy Coding Of Grayscale Images”. IEEE Transactions On Circuits And Systems For Video Technology, vol. 6, pp. 21-31, 1996
[52] S. Kamata, R. O. Eason, E. Kawaguchi, “Depth-First Coding For Multivalued Pictures Using Bit-Plane Decomposition” IEEE Transactions on Communications, 43(5): 1961 – 1969, May 1995 .
[53] P. García-Sevilla , M. Petrou , S. Kamata, “The use of Boolean model for texture analysis of grey images” Computer Vision and Image Understanding, 74(3): 227-235, 1999.
[54] F. Liu, R. W. Picard, “Periodicity, Directionality, and Randomness: Wold Features for Image Modeling and Retrieval,” IEEE Transaction on Pattern Analysis and Machine Intelligence 18(7): 722-733 (1996).
[55] R. Sriram, J. M. Francos, W. A. Pearlman, “Texture coding using a Wold decomposition model” IEEE Transactions on Image Processing 5(9): 1382-1386, 1996.