![Page 1: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/1.jpg)
Appearances Matter:Enhancing the Presentation of Text using
Cross-Modal Representation Learningand Emotion Analysis
Gerard de Melohttp://gerard.demelo.org
Appearances Matter:Enhancing the Presentation of Text using
Cross-Modal Representation Learningand Emotion Analysis
Gerard de Melohttp://gerard.demelo.org
![Page 2: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/2.jpg)
TextTextTextText
Theological Hall, Strahov Monastery Library, Prague
![Page 3: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/3.jpg)
“Wall of Text”“Wall of Text”
Image: Minnesota Orchestra via https://songofthelarkblog.com/2014/11/16/bad-news-good-news/
![Page 4: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/4.jpg)
Automatically ImproveAutomatically Improvethe Presentation?the Presentation?
Automatically ImproveAutomatically Improvethe Presentation?the Presentation?
https://commons.wikimedia.org/wiki/File:Muliple_colored_pencils_07.jpg
![Page 5: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/5.jpg)
Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
OutlineOutline
Font Selection
Affective Text Presentation
Cross-Modal Analysis and Enrichment
![Page 6: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/6.jpg)
Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
OutlineOutline
Font Selection
Affective Text Presentation
Cross-Modal Analysis and Enrichment
![Page 7: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/7.jpg)
Font SelectionFont Selection
Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
![Page 8: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/8.jpg)
Design vs. PerceptionDesign vs. PerceptionDesign vs. PerceptionDesign vs. Perception
Corporate fonts such asIntel Clear, IBM Plex,
Google Product Sans, etc.
Corporate fonts such asIntel Clear, IBM Plex,
Google Product Sans, etc.
![Page 9: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/9.jpg)
Design vs. PerceptionDesign vs. PerceptionDesign vs. PerceptionDesign vs. Perception
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
Van Rompay and Pruyn (2011)
Choice of fontaffects assumedbrand credibility
price expectations
Choice of fontaffects assumedbrand credibility
price expectations
![Page 10: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/10.jpg)
Design vs. PerceptionDesign vs. PerceptionDesign vs. PerceptionDesign vs. Perception
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
de Sousa et al. (2020). Journal of Sensory Studies
Choice of fontaffects
intent to purchaseand assumed
acidity, sweetness
Choice of fontaffects
intent to purchaseand assumed
acidity, sweetness
![Page 11: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/11.jpg)
Design vs. PerceptionDesign vs. PerceptionDesign vs. PerceptionDesign vs. Perception
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
Velasco et al. i-Perception 6(4), 2015
Choice of fontaffects assumed
taste
Choice of fontaffects assumed
taste
round font→ sweet
angular font→ bitter, salty, sour
![Page 12: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/12.jpg)
Design vs. PerceptionDesign vs. PerceptionDesign vs. PerceptionDesign vs. Perception
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
Shaikh (2007)O’Brien Louch & Stork (2014)
Choice of fontaffects assumed
attributes ofpeople using them
Choice of fontaffects assumed
attributes ofpeople using them
![Page 13: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/13.jpg)
Font SelectionFont SelectionFont SelectionFont Selection
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
![Page 14: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/14.jpg)
Font Selection:Font Selection:Crowdsourced DataCrowdsourced Data
Font Selection:Font Selection:Crowdsourced DataCrowdsourced Data
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
200 fontsrated with respect to37 attributes
![Page 15: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/15.jpg)
Learning Representations of FontsLearning Representations of FontsLearning Representations of FontsLearning Representations of Fonts
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
![Page 16: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/16.jpg)
Learning Representations of FontsLearning Representations of FontsLearning Representations of FontsLearning Representations of Fonts
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
![Page 17: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/17.jpg)
Learning Representations of FontsLearning Representations of FontsLearning Representations of FontsLearning Representations of Fonts
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
fontjoy.com
![Page 18: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/18.jpg)
Learning Representations of FontsLearning Representations of FontsLearning Representations of FontsLearning Representations of Fonts
http://fontjoy.com/projector/
VisualRepresentation Space
VisualRepresentation Space
![Page 19: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/19.jpg)
Learning Representations of FontsLearning Representations of FontsLearning Representations of FontsLearning Representations of Fonts
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
fontjoy.com
Infer Attribute-BasedRepresentations
Infer Attribute-BasedRepresentations
![Page 20: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/20.jpg)
Learning Representations of FontsLearning Representations of FontsLearning Representations of FontsLearning Representations of Fonts
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
High Potential Moderate Potential Low Potential
angular, artistic, attention, attractive, boring, complex, dramatic, happy, modern, playful, sloppy, strong, thin
calm, charming, formal, fresh, friendly, gentle, graceful, legible, sharp, soft, warm
bad, clumsy, delicate, disorderly, pretentious, technical
![Page 21: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/21.jpg)
Learning Representations of FontsLearning Representations of FontsLearning Representations of FontsLearning Representations of Fonts
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
![Page 22: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/22.jpg)
Broad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage Semantics
Source: http://www.downwithdesign.com/logo-design/hidden-razor-sharp-brilliance-gillette-logo/
![Page 23: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/23.jpg)
Broad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage Semantics
Haszlett et al. (2013)
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
![Page 24: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/24.jpg)
Broad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage Semantics
Haszlett et al. (2013)
Incongrencybetween word semantics
and font
Incongrencybetween word semantics
and font
70ms slower responseon averagefor incongruent fontsin Stroop style study
Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
![Page 25: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/25.jpg)
Broad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage Semantics
Image: https://www.somagnews.com/download-thousands-of-fonts-free-and-safely/
![Page 26: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/26.jpg)
Broad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage Semantics
Image: https://www.somagnews.com/download-thousands-of-fonts-free-and-safely/
Crawled Large-Scale Datafrom 1001fonts.com
– 10.4k fonts– 2.6k tags– 54k font–tag assignments
Crawled Large-Scale Datafrom 1001fonts.com
– 10.4k fonts– 2.6k tags– 54k font–tag assignments
Tags
christmas, bouncy, staggered,curly, cute, playful, casual,warm, fun, handwritten
![Page 27: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/27.jpg)
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector AlignmentCross-Modal Vector Alignment
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector AlignmentCross-Modal Vector Alignment
Cross-Modal
Alignment
ResNet18+PCA
Counter-fitting
Tugba Kulahcioglu, Gerard de Melo. Fonts Like This but Happier: A New Way to Discover Fonts. ACM MM
![Page 28: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/28.jpg)
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector AlignmentCross-Modal Vector Alignment
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector AlignmentCross-Modal Vector Alignment
Cross-Modal
Alignment
ResNet18+PCA
Counter-fitting
Tugba Kulahcioglu, Gerard de Melo. Fonts Like This but Happier: A New Way to Discover Fonts. ACM MM
![Page 29: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/29.jpg)
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector AlignmentCross-Modal Vector Alignment
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector AlignmentCross-Modal Vector Alignment
Tugba Kulahcioglu, Gerard de Melo. Fonts Like This but Happier: A New Way to Discover Fonts. ACM MM
![Page 30: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/30.jpg)
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector ArithmeticCross-Modal Vector Arithmetic
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector ArithmeticCross-Modal Vector Arithmetic
Tugba Kulahcioglu, Gerard de Melo. Fonts Like This but Happier: A New Way to Discover Fonts. ACM MM
![Page 31: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/31.jpg)
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector ArithmeticCross-Modal Vector Arithmetic
Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector ArithmeticCross-Modal Vector Arithmetic
Tugba Kulahcioglu, Gerard de Melo. Fonts Like This but Happier: A New Way to Discover Fonts. ACM MM
![Page 32: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/32.jpg)
Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
OutlineOutline
Font Selection
Affective Text Presentation
Cross-Modal Analysis and Enrichment
![Page 33: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/33.jpg)
Emotion vs. PerceptionEmotion vs. PerceptionEmotion vs. PerceptionEmotion vs. Perception
Image: Paul Ekman
Ekman's6 Basic
Emotions
Ekman's6 Basic
Emotions
![Page 34: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/34.jpg)
FontLexFontLexFontLexFontLex
Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations
Infer font–emotion mappingusing seed data + word2vecInfer font–emotion mappingusing seed data + word2vec
![Page 35: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/35.jpg)
FontLexFontLexFontLexFontLex
via Sentiment/Emotions associations of words
Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations
![Page 36: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/36.jpg)
FontLexFontLexFontLexFontLex
Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations
User Study: 30 participants, 25x2 tasks
5 fonts:2 congruent
>0.752 incongruent
<0.251 neutral
![Page 37: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/37.jpg)
FontLexFontLexFontLexFontLex
via Sentiment/Emotions associations of words
Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations
![Page 38: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/38.jpg)
FontLex:FontLex:Complex EmotionsComplex Emotions
FontLex:FontLex:Complex EmotionsComplex Emotions
Tugba Kulahcioglu, Gerard de Melo. Semantics-Aware Typographical Choices via Affective Associations
Plutchik’s Wheel of Emotions
![Page 39: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/39.jpg)
FontLex:FontLex:Complex EmotionsComplex Emotions
FontLex:FontLex:Complex EmotionsComplex Emotions
Tugba Kulahcioglu, Gerard de Melo. Semantics-Aware Typographical Choices via Affective Associations
Extension by using more fontsfor FontLex data
![Page 40: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/40.jpg)
Even More EmotionsEven More EmotionsEven More EmotionsEven More Emotions
Cowen & Keltner(2017)
Are8 basic
emotionsreally
sufficient?
Are8 basic
emotionsreally
sufficient?
![Page 41: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/41.jpg)
AffectVecAffectVecAffectVecAffectVec
https://www.flickr.com/photos/ill-padrino/6437837857/
joysadnessfear
guiltsuspense
0.5
0.1
0.4
...
0.3
0.1
prank
Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database
Matej Kren: Idiom. Prague Municipal Library
![Page 42: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/42.jpg)
AffectVecAffectVecAffectVecAffectVec
Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database
![Page 43: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/43.jpg)
AffectVecAffectVecAffectVecAffectVec
Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database
![Page 44: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/44.jpg)
AffectVecAffectVecAffectVecAffectVec
Download:
http://emotionlexicon.org/
Download:
http://emotionlexicon.org/
Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database
Also: Experiments on UnsupervisedDocument-Level Emotion Prediction
![Page 45: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/45.jpg)
Cross-LingualCross-LingualEmotion Lexicon InductionEmotion Lexicon Induction
Cross-LingualCross-LingualEmotion Lexicon InductionEmotion Lexicon Induction
Over 300 languages
http://emotionlexicon.org/
Over 300 languages
http://emotionlexicon.org/
Arun Ramachandran, Gerard de Melo. COLING 2020
![Page 46: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/46.jpg)
ColorColorColorColor
https://commons.wikimedia.org/wiki/File:Muscarelle_lights_2.jpg
![Page 47: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/47.jpg)
ColorColorColorColor
Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI
Bartram, Patra, Stone. CHI 2017
![Page 48: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/48.jpg)
Perception of Text:Perception of Text:Font and ColorFont and Color
Perception of Text:Perception of Text:Font and ColorFont and Color
Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI
Fonts: best for serious, trustworthy, disturbingColor palettes: best for calm, negative, playful
![Page 49: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/49.jpg)
Perception of Text:Perception of Text:Content → Font and ColorContent → Font and Color
Perception of Text:Perception of Text:Content → Font and ColorContent → Font and Color
Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI
United Nations
![Page 50: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/50.jpg)
Perception of Text:Perception of Text:Content → Font and ColorContent → Font and Color
Perception of Text:Perception of Text:Content → Font and ColorContent → Font and Color
Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations /Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI
The Smurfs Scream
![Page 51: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/51.jpg)
Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
OutlineOutline
Font Selection
Affective Text Presentation
Cross-Modal Analysis and Enrichment
![Page 52: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/52.jpg)
Emojis are ubiquitousEmojis are ubiquitousEmojis are ubiquitousEmojis are ubiquitous
![Page 53: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/53.jpg)
Emojis are ubiquitousEmojis are ubiquitousEmojis are ubiquitousEmojis are ubiquitous
Image: https://www.theverge.com/2016/2/24/11105250/facebook-reactions-emoji-how-to
Very prominent on social media,instant messaging
email subject lines etc.
Very prominent on social media,instant messaging
email subject lines etc.
Face with Tears of Joy:Oxford DictionariesWord of the Year 2015
Face with Tears of Joy:Oxford DictionariesWord of the Year 2015
![Page 54: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/54.jpg)
Emojis vs. EmotionsEmojis vs. EmotionsEmojis vs. EmotionsEmojis vs. Emotions
Abu Awal Md Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP
Same parts of brain activatedas when seeing a real face
Same parts of brain activatedas when seeing a real face
“Emoji” isJapanese for
“picture character”but clearly
closely linkedto emotions
Image: from Petra Kralj Novak. Sentiment of Emojis
![Page 55: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/55.jpg)
Dataset and Study:Dataset and Study:Emoji EmotionsEmoji Emotions
Dataset and Study:Dataset and Study:Emoji EmotionsEmoji Emotions
Abu Shoeb, Gerard de Melo. EmoTag1200 : Understanding the Association between Emojis and Emotions . EMNLP👍 😄 😻Abu Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP
Online:emoji.nlproc.org
Online:emoji.nlproc.org
![Page 56: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/56.jpg)
Emoji-BasedEmoji-BasedEmotion EmbeddingsEmotion Embeddings
Emoji-BasedEmoji-BasedEmotion EmbeddingsEmotion Embeddings
Image: https://www.theverge.com/2016/2/24/11105250/facebook-reactions-emoji-how-to
0.8
0.4
0.1
...
0.0
0.1
prank
Abu Awal Md Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP
![Page 57: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/57.jpg)
Use Case: Deep Neural Network for Use Case: Deep Neural Network for Tweet Emotion ClassificationTweet Emotion Classification
Use Case: Deep Neural Network for Use Case: Deep Neural Network for Tweet Emotion ClassificationTweet Emotion Classification
Abu Awal Md Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP
Methods Anger Fear Joy Sadness Average Dim
InterpretableAffective Tweets 0.65 0.66 0.60 0.69 0.65 n/a
EmoTag 0.70 0.73 0.69 0.75 0.72 620
Non-Interpretable
Random Int. 0.68 0.72 0.66 0.73 0.70 300
word2vec 0.70 0.72 0.67 0.75 0.71 300
GloVe 0.70 0.73 0.68 0.76 0.72 300
GloVe Twitter 0.72 0.74 0.68 0.76 0.73 200
![Page 58: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/58.jpg)
ImagesImagesImagesImages
Image: https://500px.com/photo/96738645/street-bookstore-by-pablo-tarrero
![Page 59: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/59.jpg)
Image–Text RelationshipsImage–Text RelationshipsImage–Text RelationshipsImage–Text Relationships
Image: https://gizmodo.com/this-hurricane-proof-house-made-from-612-000-recycled-p-1836106774
Many documents,especially on the Web,
are multimodal
Many documents,especially on the Web,
are multimodal
![Page 60: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/60.jpg)
Image–Text RelationshipsImage–Text RelationshipsImage–Text RelationshipsImage–Text Relationships
Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone.CITE: A Corpus of Image–Text Relationships. Proc. NAACL-HLT 2019.https://github.com/malihealikhani/CITE
?
![Page 61: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/61.jpg)
Image–Text RelationshipsImage–Text RelationshipsImage–Text RelationshipsImage–Text Relationships
Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone.CITE: A Corpus of Image–Text Relationships. Proc. NAACL-HLT 2019.https://github.com/malihealikhani/CITE
Traditionally, discourse studies in NLP mostly considerrelationships between sentences.
Traditionally, discourse studies in NLP mostly considerrelationships between sentences.
![Page 62: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/62.jpg)
Image–Text RelationshipsImage–Text RelationshipsImage–Text RelationshipsImage–Text Relationships
Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone.CITE: A Corpus of Image–Text Relationships. Proc. NAACL-HLT 2019.https://github.com/malihealikhani/CITE
?
![Page 63: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/63.jpg)
Image–Text RelationshipsImage–Text RelationshipsImage–Text RelationshipsImage–Text Relationships
Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone.CITE: A Corpus of Image–Text Relationships. Proc. NAACL-HLT 2019.https://github.com/malihealikhani/CITE
Elaboration
![Page 64: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/64.jpg)
Image–Text RelationshipsImage–Text RelationshipsImage–Text RelationshipsImage–Text Relationships
Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone.CITE: A Corpus of Image–Text Relationships. Proc. NAACL-HLT 2019.https://github.com/malihealikhani/CITE
![Page 65: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/65.jpg)
Image PlacementImage PlacementImage PlacementImage Placement
Illustrate Your Story: Enriching Text with Images. Sreyasi Nag Chowdhury, William Cheng, Gerard de Melo, Simon Razniewski, Gerhard Weikum. Proceedings of WSDM
![Page 66: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/66.jpg)
Image PlacementImage PlacementImage PlacementImage Placement
Illustrate Your Story: Enriching Text with Images. Sreyasi Nag Chowdhury, William Cheng, Gerard de Melo, Simon Razniewski, Gerhard Weikum. Proceedings of WSDM
![Page 67: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/67.jpg)
Image PlacementImage PlacementImage PlacementImage Placement
Illustrate Your Story: Enriching Text with Images. Sreyasi Nag Chowdhury, William Cheng, Gerard de Melo, Simon Razniewski, Gerhard Weikum. Proceedings of WSDM
![Page 68: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/68.jpg)
Image–Text RelationshipsImage–Text RelationshipsImage–Text RelationshipsImage–Text Relationships
Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone.CITE: A Corpus of Image–Text Relationships. Proc. NAACL-HLT 2019.https://github.com/malihealikhani/CITE
![Page 69: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/69.jpg)
Application to SummarizationApplication to SummarizationApplication to SummarizationApplication to Summarization
This is related work by Vempala & Preoţiuc-Pietro.Categorizing and Inferring the Relationship between the Text and Image of Twitter Posts. ACL 2019
![Page 70: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/70.jpg)
Application to SummarizationApplication to SummarizationApplication to SummarizationApplication to Summarization
This is related work by Vempala & Preoţiuc-Pietro.Categorizing and Inferring the Relationship between the Text and Image of Twitter Posts. ACL 2019
![Page 71: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/71.jpg)
User InterfacesUser Interfaces
![Page 72: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/72.jpg)
User InterfacesUser Interfaces
![Page 73: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/73.jpg)
“Wall of Text”“Wall of Text”
Image: Minnesota Orchestra via https://songofthelarkblog.com/2014/11/16/bad-news-good-news/
![Page 74: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/74.jpg)
User Interfaces:Summaries
User Interfaces:Summaries
Automatic summarizers may make mistakes.
Loss of text structure, formatting, images.
Automatic summarizers may make mistakes.
Loss of text structure, formatting, images.
Example from See et al. (2017) Get To The Point: Summarization with Pointer-Generator Networks.
lagos, nigeria (cnn) a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation. ….
muhammadu buhari says he plans to aggressively fight corruption that has longplagued nigeria. he says his administration is confident it will be able to thwartcriminals. the win comes after a long history of military rule, coups and botchedattempts at democracy in africa’s most populous nation.
![Page 75: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/75.jpg)
User Interfaces:Summaries
User Interfaces:Summaries
Different users have different needs.Context and details wanted by user may be lost.
Hard to drill down into specific aspects.
Different users have different needs.Context and details wanted by user may be lost.
Hard to drill down into specific aspects.
Example from See et al. (2017) Get To The Point: Summarization with Pointer-Generator Networks.
lagos, nigeria (cnn) a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation. ….
muhammadu buhari says he plans to aggressively fight corruption that has longplagued nigeria. he says his administration is confident it will be able to thwartcriminals. the win comes after a long history of military rule, coups and botchedattempts at democracy in africa’s most populous nation.
![Page 76: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/76.jpg)
HiTextHiText
Yang, Cheng, Wang, de Melo. HiText: Text Reading with Dynamic Salience Marking. Proceedings of WWW
![Page 77: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/77.jpg)
HiTextHiText
Yang, Cheng, Wang, de Melo. HiText: Text Reading with Dynamic Salience Marking. Proceedings of WWW
![Page 78: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/78.jpg)
HiTextHiText
Yang, Cheng, Wang, de Melo. HiText: Text Reading with Dynamic Salience Marking. Proceedings of WWW
![Page 79: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/79.jpg)
HiTextHiText
Yang, Cheng, Wang, de Melo. HiText: Text Reading with Dynamic Salience Marking. Proceedings of WWW
User studies confirmed that studentsgot better results in a reading comprehension
test using HiText
User studies confirmed that studentsgot better results in a reading comprehension
test using HiText
![Page 80: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/80.jpg)
VMSE: VisualizingMulti-Document Semantics
VMSE: VisualizingMulti-Document Semantics
Fact extraction and salience rankingFact extraction and salience ranking
Sheng et al. Visualizing Multi-Document Semantics via Open Domain Information Extraction. Proc. ECML-PKDD 2018
![Page 81: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/81.jpg)
Structured DataStructured Data
Bhowmik & de Melo. Be Precise and Concise. Proc. Web Conference
Huge difference betweeninput and output
Need to choose relevant facts andconstruct a suitable noun phrase.
Swiss Tennis Player
![Page 82: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/82.jpg)
User Interfaces:Visual Presentation
User Interfaces:Visual Presentation
“Which companies were created during the last century in Silicon Valley ?”
YAGO2 UI:WWW
Best Demo Award
YAGO2 UI:WWW
Best Demo Award
Gerard de Melo
![Page 83: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/83.jpg)
Graph User Interfacesfor Explainable AI
Graph User Interfacesfor Explainable AI
Xian et al. Reinforcement Knowledge Graph Reasoning for Explainable Recommendation. SIGIR 2019
![Page 84: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/84.jpg)
Automated Content Compositionand Design
Automated Content Compositionand Design
Bike Image: Adapted from https://www.flickr.com/photos/swambo/14119129185
AutomatedContent
Composition& Design
MultimodalData
StructuredData
TextData
![Page 85: Appearances Matter: Enhancing the Presentation of Text](https://reader033.vdocuments.us/reader033/viewer/2022042410/625ead188194ee364a6c0099/html5/thumbnails/85.jpg)
SummarySummarySummarySummary
Font Selection►Seed-based Approach►Broad Coverage Approach
Affective Text Presentation►Affect to choose Fonts, Colour►AffectVec Emotion Lexicon
Cross-Modal Enrichment►Emojis►Images►User Interfaces
Get in Touch!
http://[email protected]
Get in Touch!
http://[email protected]
Thank You!