![Page 1: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/1.jpg)
On sweet spots and trustworthiness:
a tale of mass digitisation and digital preservation
Diastor Conference 5th of June 2014
Tom De Smet [email protected]
![Page 2: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/2.jpg)
![Page 3: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/3.jpg)
ONZE MISSIE ONZE
MISSIE
![Page 4: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/4.jpg)
Collections including: • Film from 1898 onwards • Advertising 1920 • Cinema journals ‘22–’80 • Radio from 1934 • Television from 1951 • News, Current affairs, Sport • Dutch royal house collection • Dutch Premier League archive • National Music Archive • Former broadcast museum • Amateur film • Documentary film • Photographs • Visual art collections • And much more…
![Page 5: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/5.jpg)
23 June 2014
Nederlands Instituut voor Beeld en Geluid
5
CENTRAL POSITION OF THE COLLECTION Content is key
• 80% audio visual heritage • > 850.000 hrs of audiovisual material
• 2 million photos • 20.000 objects • and counting…
à 1 catalogue à 1 annotation process
![Page 6: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/6.jpg)
![Page 7: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/7.jpg)
Demanding users: Broadcast Professional
![Page 8: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/8.jpg)
![Page 9: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/9.jpg)
The Digital Archive
137.200 hours video 17.510 hours film
123.900 hours audio 1.200.000 photo’s
yearly 8.000 hours TV (video)
54.000 hours radio 1500 hours of
miscellaneous material
Approx. 350TB per year
born digital: Dutch television & radio and other sources
Europe’s largest digitisation project
![Page 10: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/10.jpg)
![Page 11: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/11.jpg)
SOURCE
TIME COST OUTPUT
Triangle of Challenges
Accessibility Search
Retrieval Re-use
Permanence Storage costs
Coding Resolution
![Page 12: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/12.jpg)
Research Questions
� � t � n �r � B� t � � � t � c� s�� � re� t � B� � tBos� � �� � � s� �� f � �� � t � � Bs� � � B� t Bs� � � � � t � � � � � � � � s� �� � <� � u�eBtt � � � ru� n � � � � � c� � � B� � � B� es� tt � B � � � � � � � � � ��e e� � � � f -‐�à� � � � � �� � es� t � sc�w B �� � t � n �r � B� � B c� s � � � t � � � � s� �� � B� �� � sB�� � � t ,� B� e�w � � � � c� � � B� � Bs� � f � �� � � � �s � � � � � es�� w� �� � � �e �� � � � w� t � B� � � v� tw � � BB� t � � � � � Bs� � tf � � � tB� � Bs� n Bs� � Bn t?�
26/09/2011
![Page 13: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/13.jpg)
Challenge 1: The Material
" Formats & types - Colour / BW " Cinematographical quality " Physical state and problems:
" Acidification (shrinkage) " Glue remains " Splices " Perforations " A/B winds (emulsion side)
" Sound & synchronisation " Sizes of film and cans " Number of items per carrier and physical marks " Status of metadata in catalog
Source material
Budget and time Quality of result
![Page 14: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/14.jpg)
Challenge 2: Digital (& Video) Domain
" Film market in development"" Hardware development"" Digital format standardisation
vs market acceptance"" End-user requirements
![Page 15: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/15.jpg)
Rela
tive im
po
rtan
ce in
co
llect
ion
Cinematographical & technical quality
Sports (old)
Sports (more recent)
Sports (selected)
Tele-recordings
News
News (selected)
Current Affairs
Current Affairs
(selected)
Drama
Drama(selected)
Documentaries
Documentaries(selected) Opdrachtfilms
Polygoon (35mm)
Advertorials
Standard resolution
HD resolution
2K resolution
Resolutions
![Page 16: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/16.jpg)
Format selection & scanning
Other tape format(s) ? "
Data file format (JPEG2000 feasible) ? "
DPX is the best option"
Scan to Digi Transcode Lo-res proxy
Scan to Digi Encode to MXF D10-50 Transcode LoRes proxy
Scan to DPX Encode to XDCAM HD422 Transcode LoRes proxy
![Page 17: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/17.jpg)
Formats since 2010
DPX - SMPTE 268M-2003 v2 - 10 bit log - RGB or single channel BW - Scanner colour space (specified
and with LUT for Rec.709 - XDCAM HD)
BWAV - EBU tech 3285-2001 v1 - 24 bit linear PCM, 48 kHz
sample rate
XDCAM HD 422 - SMPTE 381, 382,
XDCAM_MXF_HD422_v080 - MXF OP1a: SMPTE 377, 378,
379 - Timecode Track in material
package MXF (EBU Rec 122)
![Page 18: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/18.jpg)
IMMIX CATALOGUE
SD Workflow
IMMIX CATALOGUE
![Page 19: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/19.jpg)
HD / 2K Workflow
WRITE LTO4
![Page 20: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/20.jpg)
Scanning Room
![Page 21: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/21.jpg)
• 2010 switch from SD to HD = ‘video only’ to data + video • Digi<sing both internally (1/3d) and externally (2/3ds) • Film prepara<on covered by digi<sing party • Digital Master in DPX format • B/W materials saved on one channel only (Y) • Deliverance/Mezzanine format XDCAM HD422 • Resolu<ons 2K / HD – depend on source material (economical
constraints) • Scanner colour space (na<ve; RGB) • Color correc<on needed in case of colour fading • Automated workflows with emphasis on QC in all stages • Managed storage (DPX from medio 2011)
Sound and Vision’s approach - summary
26/09/2011
![Page 22: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/22.jpg)
Trustworthy? OAIS
ISO 14721: functional model ISO 16363: guidelines for certification
“An OAIS is an Archive […] that has accepted the responsibility to preserve information and make it available for a Designated Community.”
![Page 23: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/23.jpg)
Digital preservation at Sound and Vision
- Impact of data loss is big (and risky!) - IT-infrastructure needs to be updated constantly - We feel it needs to be stated as a clear objective in your long-term
goals and mission - We set up a project for Digital Object Management requirements:
- Preservation metadata dictionary - Preservation workflow model
![Page 24: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/24.jpg)
Topic Lesson
1 - Collection How much is actually known/correct/complete about the source material (with regards to digitisation)? - Don’t overestimate this part.
2 - Technology
• Archival workflow has specific demands • Archival material poses specific challenges that
suppliers have limited access to / knowledge about. • Take more time to fix start-up problems if you are an
early adopter.
3 - Outsourcing Even when material properties are specified this doesn’t prevent wrong assumptions about the work load (e.g. film prep).
Lessons learned (1)
![Page 25: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/25.jpg)
Topic Lesson
4 - Formats
Due to the film grain, high detail level in the DPX and the coding properties of the XDCAM HD422, post-processing is needed to prevent visual artefacts (blocking) in the latter.
5 - Workflow & QC
Devising a digitisation workflow is not difficult, but it takes time to understand: • how all the components (have to) work together • what the properties of the formats are • how to perform the verification of formats • the extraction and mapping of metadata to internal
systems.
Lessons Learned (2)
![Page 26: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/26.jpg)
Lessons learned en route to TDR
- Not all at once - Choose pilots/examples of workflows that are closest to
your archive/institution - Define your SIP and DIP in collaboration with your
designated communities to avoid disappointment/frustration and a lot of manual work before and after ingest!
![Page 27: On sweet spots and trustworthiness: a tale of mass ... · On sweet spots and trustworthiness: a tale of mass digitisation and digital preservation Diastor Conference 5th of June 2014](https://reader031.vdocuments.us/reader031/viewer/2022011908/5f5c552373a50077f124d462/html5/thumbnails/27.jpg)
Further reading
� y eti((n n n ?es� t B� � s� ?Bs� (tr t � � ( � � � t( � � � s�s r (s� tBos� � ( � � � t � � � � ) cx?Wfl?e� � �
�