appearances matter: enhancing the presentation of text

Appearances Matter:Enhancing the Presentation of Text using

Cross-Modal Representation Learningand Emotion Analysis

Gerard de Melohttp://gerard.demelo.org

Appearances Matter:Enhancing the Presentation of Text using

Cross-Modal Representation Learningand Emotion Analysis

Gerard de Melohttp://gerard.demelo.org

TextTextTextText

Theological Hall, Strahov Monastery Library, Prague

“Wall of Text”“Wall of Text”

Image: Minnesota Orchestra via https://songofthelarkblog.com/2014/11/16/bad-news-good-news/

Automatically ImproveAutomatically Improvethe Presentation?the Presentation?

Automatically ImproveAutomatically Improvethe Presentation?the Presentation?

https://commons.wikimedia.org/wiki/File:Muliple_colored_pencils_07.jpg

Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/

OutlineOutline

Font Selection

Affective Text Presentation

Cross-Modal Analysis and Enrichment

Font SelectionFont Selection


Design vs. PerceptionDesign vs. PerceptionDesign vs. PerceptionDesign vs. Perception

Corporate fonts such asIntel Clear, IBM Plex,

Google Product Sans, etc.

Corporate fonts such asIntel Clear, IBM Plex,

Google Product Sans, etc.


Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts

Van Rompay and Pruyn (2011)

Choice of fontaffects assumedbrand credibility

price expectations

Choice of fontaffects assumedbrand credibility

price expectations



de Sousa et al. (2020). Journal of Sensory Studies

Choice of fontaffects

intent to purchaseand assumed

acidity, sweetness

Choice of fontaffects

intent to purchaseand assumed

acidity, sweetness



Velasco et al. i-Perception 6(4), 2015

Choice of fontaffects assumed

taste


taste

round font→ sweet

angular font→ bitter, salty, sour



Shaikh (2007)O’Brien Louch & Stork (2014)


attributes ofpeople using them


attributes ofpeople using them

Font SelectionFont SelectionFont SelectionFont Selection


Font Selection:Font Selection:Crowdsourced DataCrowdsourced Data

Font Selection:Font Selection:Crowdsourced DataCrowdsourced Data


200 fontsrated with respect to37 attributes

Learning Representations of FontsLearning Representations of FontsLearning Representations of FontsLearning Representations of Fonts




fontjoy.com


http://fontjoy.com/projector/

VisualRepresentation Space

VisualRepresentation Space



fontjoy.com

Infer Attribute-BasedRepresentations

Infer Attribute-BasedRepresentations



High Potential Moderate Potential Low Potential

angular, artistic, attention, attractive, boring, complex, dramatic, happy, modern, playful, sloppy, strong, thin

calm, charming, formal, fresh, friendly, gentle, graceful, legible, sharp, soft, warm

bad, clumsy, delicate, disorderly, pretentious, technical

Broad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage SemanticsBroad-Coverage Semantics

Source: http://www.downwithdesign.com/logo-design/hidden-razor-sharp-brilliance-gillette-logo/


Haszlett et al. (2013)



Haszlett et al. (2013)

Incongrencybetween word semantics

and font

Incongrencybetween word semantics

and font

70ms slower responseon averagefor incongruent fontsin Stroop style study



Image: https://www.somagnews.com/download-thousands-of-fonts-free-and-safely/


Image: https://www.somagnews.com/download-thousands-of-fonts-free-and-safely/

Crawled Large-Scale Datafrom 1001fonts.com

– 10.4k fonts– 2.6k tags– 54k font–tag assignments

Crawled Large-Scale Datafrom 1001fonts.com

– 10.4k fonts– 2.6k tags– 54k font–tag assignments

Tags

christmas, bouncy, staggered,curly, cute, playful, casual,warm, fun, handwritten

Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector AlignmentCross-Modal Vector Alignment


Cross-Modal

Alignment

ResNet18+PCA

Counter-fitting

Tugba Kulahcioglu, Gerard de Melo. Fonts Like This but Happier: A New Way to Discover Fonts. ACM MM

Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector ArithmeticCross-Modal Vector Arithmetic

Broad-Coverage Semantics:Broad-Coverage Semantics:Cross-Modal Vector ArithmeticCross-Modal Vector Arithmetic



OutlineOutline

Font Selection



Emotion vs. PerceptionEmotion vs. PerceptionEmotion vs. PerceptionEmotion vs. Perception

Image: Paul Ekman

Ekman's6 Basic

Emotions

Ekman's6 Basic

Emotions

FontLexFontLexFontLexFontLex

Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations

Infer font–emotion mappingusing seed data + word2vecInfer font–emotion mappingusing seed data + word2vec


via Sentiment/Emotions associations of words




User Study: 30 participants, 25x2 tasks

5 fonts:2 congruent

>0.752 incongruent

<0.251 neutral


via Sentiment/Emotions associations of words


FontLex:FontLex:Complex EmotionsComplex Emotions


Tugba Kulahcioglu, Gerard de Melo. Semantics-Aware Typographical Choices via Affective Associations

Plutchik’s Wheel of Emotions



Tugba Kulahcioglu, Gerard de Melo. Semantics-Aware Typographical Choices via Affective Associations

Extension by using more fontsfor FontLex data

Even More EmotionsEven More EmotionsEven More EmotionsEven More Emotions

Cowen & Keltner(2017)

Are8 basic

emotionsreally

sufficient?

Are8 basic

emotionsreally

sufficient?

AffectVecAffectVecAffectVecAffectVec

https://www.flickr.com/photos/ill-padrino/6437837857/

joysadnessfear

guiltsuspense

0.5

0.1

0.4

...

0.3

0.1

prank

Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database

Matej Kren: Idiom. Prague Municipal Library


Download:

http://emotionlexicon.org/

Download:



Also: Experiments on UnsupervisedDocument-Level Emotion Prediction

Cross-LingualCross-LingualEmotion Lexicon InductionEmotion Lexicon Induction

Cross-LingualCross-LingualEmotion Lexicon InductionEmotion Lexicon Induction

Over 300 languages


Over 300 languages


Arun Ramachandran, Gerard de Melo. COLING 2020

ColorColorColorColor

https://commons.wikimedia.org/wiki/File:Muscarelle_lights_2.jpg

ColorColorColorColor

Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI

Bartram, Patra, Stone. CHI 2017

Perception of Text:Perception of Text:Font and ColorFont and Color

Perception of Text:Perception of Text:Font and ColorFont and Color


Fonts: best for serious, trustworthy, disturbingColor palettes: best for calm, negative, playful

Perception of Text:Perception of Text:Content → Font and ColorContent → Font and Color



United Nations



Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations /Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI

The Smurfs Scream


OutlineOutline

Font Selection



Emojis are ubiquitousEmojis are ubiquitousEmojis are ubiquitousEmojis are ubiquitous

Emojis are ubiquitousEmojis are ubiquitousEmojis are ubiquitousEmojis are ubiquitous

Image: https://www.theverge.com/2016/2/24/11105250/facebook-reactions-emoji-how-to

Very prominent on social media,instant messaging

email subject lines etc.

Very prominent on social media,instant messaging

email subject lines etc.

Face with Tears of Joy:Oxford DictionariesWord of the Year 2015

Face with Tears of Joy:Oxford DictionariesWord of the Year 2015

Emojis vs. EmotionsEmojis vs. EmotionsEmojis vs. EmotionsEmojis vs. Emotions

Abu Awal Md Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP

Same parts of brain activatedas when seeing a real face

Same parts of brain activatedas when seeing a real face

“Emoji” isJapanese for

“picture character”but clearly

closely linkedto emotions

Image: from Petra Kralj Novak. Sentiment of Emojis

Dataset and Study:Dataset and Study:Emoji EmotionsEmoji Emotions

Dataset and Study:Dataset and Study:Emoji EmotionsEmoji Emotions

Abu Shoeb, Gerard de Melo. EmoTag1200 : Understanding the Association between Emojis and Emotions . EMNLP👍 😄 😻Abu Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP

Online:emoji.nlproc.org

Online:emoji.nlproc.org

Emoji-BasedEmoji-BasedEmotion EmbeddingsEmotion Embeddings

Emoji-BasedEmoji-BasedEmotion EmbeddingsEmotion Embeddings

Image: https://www.theverge.com/2016/2/24/11105250/facebook-reactions-emoji-how-to

0.8

0.4

0.1

...

0.0

0.1

prank


Use Case: Deep Neural Network for Use Case: Deep Neural Network for Tweet Emotion ClassificationTweet Emotion Classification

Use Case: Deep Neural Network for Use Case: Deep Neural Network for Tweet Emotion ClassificationTweet Emotion Classification


Methods Anger Fear Joy Sadness Average Dim

InterpretableAffective Tweets 0.65 0.66 0.60 0.69 0.65 n/a

EmoTag 0.70 0.73 0.69 0.75 0.72 620

Non-Interpretable

Random Int. 0.68 0.72 0.66 0.73 0.70 300

word2vec 0.70 0.72 0.67 0.75 0.71 300

GloVe 0.70 0.73 0.68 0.76 0.72 300

GloVe Twitter 0.72 0.74 0.68 0.76 0.73 200

ImagesImagesImagesImages

Image: https://500px.com/photo/96738645/street-bookstore-by-pablo-tarrero

Image–Text RelationshipsImage–Text RelationshipsImage–Text RelationshipsImage–Text Relationships

Image: https://gizmodo.com/this-hurricane-proof-house-made-from-612-000-recycled-p-1836106774

Many documents,especially on the Web,

are multimodal

Many documents,especially on the Web,

are multimodal


Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone.CITE: A Corpus of Image–Text Relationships. Proc. NAACL-HLT 2019.https://github.com/malihealikhani/CITE

?



Traditionally, discourse studies in NLP mostly considerrelationships between sentences.

Traditionally, discourse studies in NLP mostly considerrelationships between sentences.



?



Elaboration

Image PlacementImage PlacementImage PlacementImage Placement

Illustrate Your Story: Enriching Text with Images. Sreyasi Nag Chowdhury, William Cheng, Gerard de Melo, Simon Razniewski, Gerhard Weikum. Proceedings of WSDM

Application to SummarizationApplication to SummarizationApplication to SummarizationApplication to Summarization

This is related work by Vempala & Preoţiuc-Pietro.Categorizing and Inferring the Relationship between the Text and Image of Twitter Posts. ACL 2019

User InterfacesUser Interfaces

“Wall of Text”“Wall of Text”

Image: Minnesota Orchestra via https://songofthelarkblog.com/2014/11/16/bad-news-good-news/

User Interfaces:Summaries


Automatic summarizers may make mistakes.

Loss of text structure, formatting, images.

Automatic summarizers may make mistakes.

Loss of text structure, formatting, images.

Example from See et al. (2017) Get To The Point: Summarization with Pointer-Generator Networks.

lagos, nigeria (cnn) a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation. ….

muhammadu buhari says he plans to aggressively fight corruption that has longplagued nigeria. he says his administration is confident it will be able to thwartcriminals. the win comes after a long history of military rule, coups and botchedattempts at democracy in africa’s most populous nation.



Different users have different needs.Context and details wanted by user may be lost.

Hard to drill down into specific aspects.

Different users have different needs.Context and details wanted by user may be lost.

Hard to drill down into specific aspects.

Example from See et al. (2017) Get To The Point: Summarization with Pointer-Generator Networks.

lagos, nigeria (cnn) a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation. ….

muhammadu buhari says he plans to aggressively fight corruption that has longplagued nigeria. he says his administration is confident it will be able to thwartcriminals. the win comes after a long history of military rule, coups and botchedattempts at democracy in africa’s most populous nation.

HiTextHiText

Yang, Cheng, Wang, de Melo. HiText: Text Reading with Dynamic Salience Marking. Proceedings of WWW

HiTextHiText

Yang, Cheng, Wang, de Melo. HiText: Text Reading with Dynamic Salience Marking. Proceedings of WWW

User studies confirmed that studentsgot better results in a reading comprehension

test using HiText

User studies confirmed that studentsgot better results in a reading comprehension

test using HiText

VMSE: VisualizingMulti-Document Semantics

VMSE: VisualizingMulti-Document Semantics

Fact extraction and salience rankingFact extraction and salience ranking

Sheng et al. Visualizing Multi-Document Semantics via Open Domain Information Extraction. Proc. ECML-PKDD 2018

Structured DataStructured Data

Bhowmik & de Melo. Be Precise and Concise. Proc. Web Conference

Huge difference betweeninput and output

Need to choose relevant facts andconstruct a suitable noun phrase.

Swiss Tennis Player

User Interfaces:Visual Presentation

User Interfaces:Visual Presentation

“Which companies were created during the last century in Silicon Valley ?”

YAGO2 UI:WWW

Best Demo Award

YAGO2 UI:WWW

Best Demo Award

Gerard de Melo

Graph User Interfacesfor Explainable AI

Graph User Interfacesfor Explainable AI

Xian et al. Reinforcement Knowledge Graph Reasoning for Explainable Recommendation. SIGIR 2019

Automated Content Compositionand Design

Automated Content Compositionand Design

Bike Image: Adapted from https://www.flickr.com/photos/swambo/14119129185

AutomatedContent

Composition& Design

MultimodalData

StructuredData

TextData

SummarySummarySummarySummary

Font Selection►Seed-based Approach►Broad Coverage Approach

Affective Text Presentation►Affect to choose Fonts, Colour►AffectVec Emotion Lexicon

Cross-Modal Enrichment►Emojis►Images►User Interfaces

Get in Touch!

http://[email protected]

Get in Touch!

http://[email protected]

Thank You!

appearances matter: enhancing the presentation of text

Documents