Apache TikaEnd-to-End

An introduction to Apache Tika,

and integrating it to your application

Nick Burch

Software EngineerAlfresco

Apache Tikahttp://tika.apache.org/

• Project which started in 2006• Grew out of the Lucene community, now widely

used• Provides detection of files – eg this binary blob is

really a word file, that one is UTF-8 plain text• Plain text, HTML and XHTML versions of a wide

range of different file formats• Consistent Metadata from different files• Tika hides the complexity of the different formats

and their libraries, instead presents a simple, powerful API

• Easy to use and extend

What's new?

• Lots of new parsers – text, office formats, publishing formats, images, audio, CAD, fonts etc

• Long standing parsers improved – better HTML from word for example

• Embedded resources and containers• Use expanding – used by many SOLR

users, Alfresco, lots of people crunching masses of data on Hadoop

Supported Formats Page 1• Audio – WAV, RIFF, MIDI• DWG (CAD)• Epub• RSS and ATOM Feeds• True Type Fonts• HTML• Images – JPEG, GIF, PNG, TIFF, Bitmap

(including EXIF where found)• iWork (Keynote, Pages etc)• RFC822 mbox Mail

Supported Formats Page 2• Microsoft Outlook .msg Email• Microsoft Office (Binary) – Word,

PowerPoint, Excel, Visio, Publisher, Works• Microsoft Office (OOXML) – Word,

PowerPoint, Excel• MP3 (id3 v1 and v2)• CDF (Scientific Data)• Open Document Format (Open Office)• Old-style Open Office (.sxw etc)• PDF

Supported Formats Page 3

• Zip and Tar archives• RDF• Plain Text• FLV Video• XML• Java class files

And I probably forgot one...!

Metadata

• Tika provides consistent metadata across the range of parsers

• No need to know if it's “Last Author”, “Last Editor” or “Previous Author” in a file format, they all come back with the same metadata key

• Keys and values are strings, but strongly typed metadata entries provide converters to dates, ints etc

Text Content• Tika generates HTML-like SAX events

as it parses• Uses Java SAX API• Events can be captured or transformed• Body Content Handler used for plain text• HTML and XHTML available• Can customise with your own handler,

with XSLT or with E4X from JavaScript• eg HTML Table → CSV

Calling Tika

// Get a content detector, and an auto-selecting ParserTikaConfig config = TikaConfig.getDefaultConfig();ContainerAwareDetector detector = new ContainerAwareDetector(config.getMimeRepository() );Parser parser = new AutoDetectParser(detector);

// We’ll only want the plain text contentsContentHandler handler = new BodyContentHandler();

// Tell the parser what we haveMetadata metadata = new Metadata();metadata.set(Metadata.RESOURCE_NAME_KEY, filename);

// Have it processedparser.parse(input, handler, metadata, new ParseContext());

// Plain text only content handlerContentHandler handler = new BodyContentHandler();String text = handler.toString();

// XHTML content handlerSAXTransformerFactory factory = SAXTransformerFactory.newInstance();TransformerHandler handler = factory.newTransformerHandler();handler.getTransformer().setOutputProperty(OutputKeys.METHOD, "xml");handler.getTransformer().setOutputProperty(OutputKeys.INDENT, "yes");StringWriter sw = new StringWriter();handler.setResult(new StreamResult(sw));String text = sw.toString();

Tika Parsers

Parser Interface• Two key methods – what mime types are

supported, and do the parsing

public interface Parser {Set<MediaType>

getSupportedTypes(ParseContext context);void parse(InputStream stream, ContentHandler

handler, Metadata metadata, ParseContext context) throws IOException, SAXException, TikaException;

}

public class HelloWorldParser implements Parser { public Set<MediaType> getSupportedTypes(ParseContext context) { Set<MediaType> types = new HashSet<MediaType>(); types.add(MediaType.parse("hello/world")); return types; } public void parse(InputStream stream, ContentHandler handler, Metadata metadata, ParseContext context) throws SAXException { XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata); xhtml.startDocument(); xhtml.startElement("h1"); xhtml.characters("Hello, World!"); xhtml.endElement("h1"); xhtml.endDocument(); metadata.set("hello","world"); metadata.set("title","Hello World!"); }}

Demo: Tika-App

Demo: Geo-Tagged Imagesin Alfresco Share via Tika

Any Questions?