speech balloon contour classification in comics

Lehigh University, Bethlehem, PA, USA - August 21 2013

Speech balloon contour classification in comics

Christophe RigaudDimosthenis KaratzasJean-Christophe Burie

Jean-Marc Ogier

http://l3i.univ-larochelle.fr/

http://www.cvc.uab.es/

GREC'13 - christophe-rigaud.com 2

Summary

● Project

● Speech balloons

● Detection

● Classification

● Dataset

● Evaluation

● Conclusion

http://www.tumblr.com

http://www.tumblr.com/tagged/animated-comic




Project

L3i project: eBDtheque● June 2011 – September 2014

● Participants

– 2 doctoral researchers

– 5 assistant professors

– 3 professors

● Comic books

– Cultural heritage

– Need to be valorized by the new technologies

● Objective: comics content understanding

– Augmented reading experience

– Information retrieval (e.g. semantic query, full text search)

– New dataset

● Progress

– Panels, text lines, balloons, people




Speech balloonsShapes and contours

Smooth contour: dialogue, conversation...

Zigzag contour: exclamation, event, action...

Wavy contour:Thought, dream, insinuation...

Image credits: eBDtheque dataset




Detection

Initialization

Detection

Results

min(E)=min(Einternal+Eexternal+Etext)

Active contour model[1]

[1] C. Rigaud, D. Karatzas, J. Van de Weijer, J-C Burie, J-M Ogier. An active contour model for speech balloon detection in comics. In 12th International Conference on Document Analysis and Recognition (ICDAR), 2013

http://refbase.cvc.uab.es/search.php?sqlQuery=SELECT%20author,%20title,%20type,%20year,%20publication,%20abbrev_journal,%20volume,%20issue,%20pages,%20keywords,%20abstract,%20address,%20corporate_author,%20thesis,%20publisher,%20place,%20editor,%20language,%20summary_language,%20orig_title,%20series_editor,%20series_title,%20abbrev_series_title,%20series_volume,%20series_issue,%20edition,%20issn,%20isbn,%20medium,%20area,%20expedition,%20conference,%20notes,%20approved,%20call_number,%20serial%20FROM%20refs%20WHERE%20serial%20=%202260%20ORDER%20BY%20author,%20year%20DESC,%20publication&client=&formType=sqlSearch&submit=Display&viewType=&showQuery=0&showLinks=1&showRows=15&rowOffset=&wrapResults=1&citeOrder=&citeStyle=APA&exportFormat=RIS&exportType=html&exportStylesheet=&citeType=html&headerMsg=




ClassificationShape/contour separation

Var = 3.97

Var = 0.35

Barycentre (red), start (green), anticlockwise.

Barycentre (red), start (green), anticlockwise.




Dataset

● eBDtheque subset● 22 speech balloons● Pixel level ground truth● Type {oval, rectangle, peak, cloud}● Tail direction

http://ebdtheque.univ-lr.fr

http://ebdtheque.univ-lr.fr/




Predicted class

Smooth Wavy Zigzag

Actualclass

Smooth 13 1 0

Wavy 1 2 0

Zigzag 1 0 4

Evaluation

● Label correspondences

● Confusion matrix

● Accuracy: 86.3%

Ground truth Classification Variance threshold

Oval, rectangle Smooth < 1.5

Cloud Wavy 1.5 < var <= 2

Peak Zigzag > 2




Conclusion

● One step further in the comics content understanding

● High dependence to balloon detection

● Shape/contour separation

● Contours are more discriminant than shapes

● Next:● Normalize the metric according to the size

● Frequency domain information

● More data

● Tail detection and speakers localization



speech balloon contour classification in comics

Technology