Enhanced Reflectance Transformation Imaging for Arts and Humanities

PeterFornaropeter.fornaro@unibas.chDigitalHumanitiesLab,UniversityofBaselSwitzerlandAndreaBiancoandrea.bianco@unibas.chDigitalHumanitiesLab,UniversityofBaselSwitzerlandAeneasKaiseraeneas.kaiser@unibas.chDigitalHumanitiesLab,UniversityofBaselSwitzerlandLukasRosenthalerlukas.rosenthaler@unibas.chDigitalHumanitiesLab,UniversityofBaselSwitzerlandLotharSchmittlschmitt.basel@gmail.comDigitalHumanitiesLab,UniversityofBaselSwitzerlandHeidrunFeldmann heidrun.feldmann@unibas.chDigitalHumanitiesLab,UniversityofBaselSwitzerland

Introduction TheDigitalHumanitiesLab(DHLab)isaresearch

group within the faculty of Humanities of theUniversityofBasel.TheresearchprofileoftheDHLabintegrates computer science, digital imaging,computational photography and the accessibility ofdigital objects in humanities research. The project“Digital Materiality”, an interdisciplinary project incollaboration with the Seminar of Art History,examines how new digital methods and techniques

can be used to describe the reflection of light onsurfaces of artworks. Of main interest are mosaicsand early prints; both categories have a stronginteraction with light and standard photographicapproaches are not able to capture the dynamiccomponentof the light-surface interdependence thatisspecificforthiskindofobjects.

For art historians who study and work withmosaics or any other object of complex surfacecomposition,itisdifficulttocaptureandvisualizetheimportant surface features in such a way thatresearch can be done with the reproduction. Themajor problem is the static nature of photographicreproductions, which does not allow interaction.Static,two-dimensionalphotographscannotvisualizeappropriately the sparkling effect caused by thesurface properties of the countless light reflectingtesserae of a mosaic, for example. Similarconsiderations apply for early prints, books andparchments.Thevisualimpressionthattheseobjectsconvey is hardly delivered by a photograph.Metallicinclusions give the artwork a dynamic appearancecausedbythechangeofreflectancebehaviorofglossycomparedtomattematerial.Furthermore,thescholarmaywanttointegrateinformationcomingfromothertypes of scientific photographs, such as infrared orultraviolet illuminated or induced fluorescenceimages, to increase the visual impression ofrenderingsofartwork.Inparticular,thecombinationof such scientific photographs with other imagingtechniques like Reflectance Transformation Imagingisadvantageousbecausemultiplevisual impressionscanbecombinedinawaythatwouldnotbepossibleinreality.

Reflection Transformation Imaging RTI(ReflectanceTransformationImaging)(Mudge

et al, 2007; Malzbender et al, 2001) is a promisingapproachtogobeyondthelimitationsofconventionalphotographic methods. However, RTI, as it is usedtoday, has some drawbacks that are critical in thecontext of the reproduction of mosaics, metallicinclusionsormostotherartwork.

RTImakes it possible to interactively display thelightreflectionasafunctionofthelightincidentangleandthestructureofthesurfacecaptured.RTIneedsaseriesofdigital imagesfromafixedcameraposition,inwhichthelightsourceisilluminatingtheobjectineach capture from a different position. In order toproduce RTI renderings of objects of different size,several lighting systems have been developed thatconsistofanumberoflightsourcesthataremounted

on the inside of a hemispherical dome. Those lightsources can be switched on and synchronizedsequentially with a digital camera. The fixed designensures a fixed and equal distance between thenumerous light sources and the object. Thus, withsuchanautomatedsetuptheimagecaptureprocessisdrastically accelerated and, in addition, ensures ahigh reproducibility. In the Digital Humanities Labsuch a dome has been developed as well; In thisspecific case it is an alloy design,which is equippedwith50LEDsandanelectroniccontrolsystem,whichenables the automatic sequential triggering of thelight sources and the synchronization with thecamera. The image sequence, recorded in this way,servesinasecondstepasdatabasisforasubsequentpixel-based modeling. For this purpose, amathematicalmodel - normally a functionof secondorder - is fitted for each pixel position, whichrepresents the set of all image points from thedifferent directions of illumination, i.e., it isparameterized so that the square mean error of alldata points relative to the function curve becomesminimal.

ThereflectionmodelsofarusedintheoriginalRTImethod described byHewlett Packard (Mudge et al,2006) corresponds to thementionedsimple second-order function. Thus it has only a relatively lowmathematical complexity and therefore a limitedprecision to represent the actual surface reflection.This specific function, however, describes mattsurfaces, a Lambertian radiator, very precisely.However, this method is not suitable for glossymaterials, since this simple model does notcorrespondtothephysical lawsofglossreflection.Afurther disadvantage is that the method, as it isusuallyusedtoday,needsaspeciallydesignedviewerapplication.

In the Digital Humanities Lab, the method hasbeen enhanced in order to be able to present morecomplex surfaces that are made from differentmaterials. In our approach the complexity of themathematical model is increased so that we canhandle diffuse and gloss surface components at thesame RTI image. In other words, this improvementmakesitpossibletomodelanddisplaymaterialswithdifferent gloss levels using the same mathematicalmodel.

A suite for humanities scholars ‘needs Forhumanitiesresearchanotherimportantaspect

is the compatibility to web-based Virtual ResearchEnvironments (VRE). In principle, a functional

graphical"front-end"withaconnectiontoadatabasecan be called a VRE. The aim of such a digitalinfrastructure for research is to allow scholars toworkwithmethodsandtoolsinthedigitaldomainastheywoulddoitinaconventional“analogue”process.Thisshouldbedoneinsuchaformthatthescientistscan intuitively recognize and use the well-knownconcepts and working methods that are offered bytheVRE.Thisrequiressomebasicfunctions:

• In any case, a browser-based client-serversolution is preferable to a stand-aloneapplication. In such a way collaborativeworkcanbemoreeasilyachieved.

• In many humanities disciplines intensiveworkisbeingdonewithimagematerialandobjects, in which specific areas (region ofinterest, RoI) are often to be emphasized.Therefore corresponding graphic elements(lines, polygons) are necessary,withwhichsuch areas can bemarked. These graphicalelementscan,forexample,bepolygonallinesections or rectangles, which allow themarkingofobjectparts.

• The method of evaluating image material,which iscommon in thehumansciences, isof a more qualitative nature. This kind ofworkrequiresextensiveandpowerfultoolsto capture descriptive and contextualmetadata (annotations, transcriptions,comments). Thesemeta-objects have to belinked with the actual primary object andneedalsotobestoredinthisway.

The visualization in a VRE must be multi-media.Besides text, image, sound and video, it is also ofadvantagetobeabletodisplayobjectsinavirtual3Dspace.

TobeabletointegrateanRTIsolutionsintoaweb-environmenttosupportreal-timecollaborativeworkneeds specific web-technologies (Palma et al, 2010;MacDonald and Robson, 2010). The presented RTIviewer is fully web-compatible and it can beintegrated inmost browsers to support high-qualityclient-side visualization; the key technology thatallowsthis integrationisWebGL.ThisOpenGL-basedprogramminginterface,whichhasbeenoptimizedfor"embeddedsystems",isnowadaysintegratedintoanymodern Web browsers. WebGL is a license-freestandarddeveloped toworkseamless inconjunctionwith the programming language JavaScript. For theapplication in a browser this means that 3Dfunctionalitiesareprovidedwithouttheneedtoload

any additional plug-ins. The performance and rangeof functionsofWebGLare impressive.WebGL isalsosupported on most mobile devices, such assmartphonesandtablets,whichfurtherincreasestherange of applications. The improved performance ofWebGL, in contrast tomanyother graphics libraries,alsoallowsforfluidinteractivework.Longrenderingtimes are left out and the visual latency isminimal.WebGL offers a variety of functionalities, rangingfrom simple grid models to complex animated,textured and illuminated surfaces. The fact that theuse of graphic elements for marking RoIs is easilypossible with this large range of functions is, ofcourse,self-evident.

A Reflectance Transformation Imaging recording that is

represented in a browser using WebGL. Around the left eye of the sacred head is a branding to be recognized, as well as a corresponding commentary, left in the picture window. The parameters of the viewing situation are shown on the

right side, which are also stored in the data model. (Source: DHLab, University of Basel)

The integrationofall those technologiesandnewdevelopments allows us to present an improvedsolutiontoreproduceandrendersurfacesofdifferentmaterials (matt and glossy) in a fully web-basedenvironment implemented in JavaScript andWebGL,running on standard computers andmobile devices.In addition to RTI image processing, photographs intheUVandIRdomaincanbecaptured,displayedandsuperimposedwiththesamesystemtoallowtheuserto compare the same region under different lightcondition. For flexibility, performance and datapermanenceaspectsourRTIimageserverwillfollowthe International Image Interoperability Framework(IIIF).IIIFdefinesastandardizedURLsyntaxtoservedigital images online in the field of cultural heritageand research. The region of interest, resolution,rotationandthefileformatofarequestedimagecanbe indicated on the URL. SIPI, the Simple ImagePresentationInterface,developedbyDHLab,provides

anIIIFcompliantimageserverwhichisideallysuitedfor our scope. Due to the fact, that the front-end iscompatiblewithanystandardweb-browser,itcanbeintegrated in a virtual research environment usingKnora.Knora isasoftware framework,developedbyDHLab, for storing, sharing, and working withprimary sources and data in the humanities. Knorabuilds the fundament for the Swiss National DataCenter for Research Data in Humanities that isoperated by the DHLab. The source code is publiclyavailableonGithubattheaddressinreference.

Thisintegratedenvironmentallowsresearcherstointeractivelycontroltheviewingandlightconditionsofeg.adigitalmosaicrendering.Regionsof interestcan be chosen, annotated, saved, and shared withotherscholars.Thefullsetofcontextualandtechnicalmetadata is stored and time stamped to be fullyreloadableandcitablebacktoanypointofitshistoryofchanges.

Conclusions The combination of the Enhanced-RTI and the

ScientificImageViewer(SIV)enablesustoconveytheimpressions of these highly dynamic light-surfaceinteractionsand the informationprovidedby IRandUV imaging e.g. to researchers who cannot visit theactual artwork in situ. The visual impression can beenriched by meta information that can shared withother scholars. The presented RTI based solution isalso helpful to document the current condition ofobjects more accurately, e.g. before and after arestoration.SustainabilityandsimplicityofRTIimagedataisdrasticallyimprovedbytheuseofaIIIFserver.The presented infrastructure allows the strictseparationof imagedataandmeta information.Asaresult, anyRTI image rendering is fully reproducibleand therefore perfectly suited for digital archiving,following the requirements of performance,permanenceandreliability.

Bibliography Mudge,M., Ashley,M., and Schroer, C. (2007) A digital

future for cultural heritage. Available at:http://culturalheritageimaging.org/What\_We\_Do/Publications/cipa2007/CIPA\_2007.Pdf

Malzbender, T., Gelb, D., and Wolters, H. (2001)

Polynomial texture maps. Proceedings of the 28thannual conference on Computer graphics andinteractivetechniques(pp.519–528).

Mudge, M., Malzbender, T., Schroer, C., and Lum, M.

(2006). New Reflection Transformation Imaging

Methods for Rock Art andMultiple-Viewpoint Display.InThe7thInternationalSymposiumonVirtualReality,Archaeology and Cultural Heritage (VAST2006) (pp.195–200).

Palma, G., Corsini, M., Cignoni, P., Scopigno, R., and

Mudge,M. (2010) Dynamic Shading Enhancement forReflection Trasformation Imaging. ACM Journal onComputing and Cultural Heritage, (pp. 6:1-6:20).(September2010)

MacDonald,L.,andRobson,S.(2010)PolynomialTexture

Mapping and 3D Representations. InternationalArchives of Photogrammetry, XXXVIII(8), (pp. 422–427).