crowd work cv: recognition for micro work

Institute for Web Science and Technologies · Univ ersity of Koblenz-Landau, Germany

Crowd Work CV:Recognition for Micro Work

Cristina Sarasua and Matthias Thimm

SoHuman 2014, Barcelona

Crowd Work CV: Recognition for Micro Work 2Cristina Sarasua

The Problem


“Better theoretical models, markets or automatic computational matching processes could drastically reduce development costs and

address search friction, an important issue in labor economics“

The Future of Crowd Work (Kittur et al., 2013)

Task assignment Career ladders Transparency


Motivational Scenario

Is Alice a good candidate for this micotask?

Spanish qualification

� Same job

� Tweet sentiment analysis

Alice

Alice´s previous work


Proposed Approach


Curriculum Vitae in Microtask Crowdsourcing

• RDF data model to describe a microtaskcrowdsourcing agent's• interests

• obtained qualifications

• work history

• It enables the representation of crowdsourcingagents’ identities and promotes their workexperience across different microtask marketplaces


Data Model


Requirements

1. Domain independence

2. Marketplace independence

3. Semantic and syntactic interoperability

4. Extensibility

5. Compatibility with traditional CV information


Design


Design (II)


Design (III)


Why Semantic Web technologies?

� agreed upon vocabulary� integration of Crowd Work

CV data (e.g. with marketplaces,other applications, Web data sets)

� graph of requesters,microtasks and contributors

� extensible data model thatenables knowledgeinference

Picture: Linking Open Data cloud diagram 2014, by Max Schmachtenberg, Christian Bizer, AnjaJentzsch and Richard Cyganiak. http://lod-cloud.net/"

Crowd Work CV ontology (OWL):https://github.com/criscod/CrowdWorkCV


Exampleex:rex a cwcv:RequesterExperience;

cwcv:byUserAccount ex:acc1;

cwcv:basedOn ex:mm1;

ex:cv1 cwcv:hasRequesterExperience ex:rex;

e:mm1 a cwcv:MasterMicrotask;

cwcv:publishedAt ex:ClixSense;

cwcv:publishedAt ex:GetPaid;

cwcv:purpose microtaskpurposes:SentimentAnalysis;

ex:mm2 a cwcv:MasterMicrotask;

cwcv:hasQualification ex:q1;

cwcv:publishedAt ex:Neobux;

Data described using theCrowd Work CV ontology


Example (cont.)ex:mex1 a cwcv:WorkerExperience;

cwcv:basedOn ex:m1;

cwcv:basedOn ex:mi;

cwcv:byUserAccount ex:acc2;

cwcv:accuracy "0.9";

ex:m1 a cwcv:Microtask;

cwcv:hasMasterMicrotask ex:mm1;

ex:q1 a cwcv:Qualification;

cwcv:name "Spanish A1";

cwcv:isRelatedToComptenece disco:Capability1;

ex:cv2 cwcv:hasWorkerExperience ex:mex1;

cwcv:obtainedQualification ex:q1;


Verification

� Verified correctness with OOPS! Pitfall Scanner http://oeg-lia3.dia.fi.upm.es/oops/index-content.jsp

� Requirements fulfilment1. Domain attached separately (e.g. SKOS vocabularies) � Domain

independence

2. Abstract concepts (e.g. evaluation) � Marketplace independence

3. Ontology defined separately with OWL � Semantic and syntactic interoperability

4. Subclasses, subproperties � Extensibility

5. Work experience, educational achievements, related skills and competences � Compatibility with traditional CV information


Differences from Related Work

# matching tasks and workers No shareable and reusable description of worker expertise thatcan be used across-platforms

# vocabularies in crowdsourcingDescribes only tasks, users, rewards and capabilities primarilyfor routing. We describe work experience, microtasks,marketplaces, qualifications, requesters’ info; CV summaries

# CV / HCOMP-PROV vocabulariesLacking crowd work-specific nuances

Khazankin and colleagues, Goeland colleagues, Difallah and colleagues

Ul Hassan and colleagues

Bojars, Celino


Crowd Work CV: FAQs

• Is a Crowd Work CV a userprofile? It is more than that! Workhistory plays a key role

• Who should own and share theCrowd Work CV data? The agentrepresented in the CV (e.g.crowd worker)

• How to avoid spam? Withcertified information


Crowd Work CV Data Management


Overview


Closing


Conclusions

� Crowd Work CV is an approach for modelling andsharing knowledge about crowd work experience ,which enriches the way reputation and credentials aremanaged in the current crowd workplace.

� Crowd Work CV could facilitate a fruitful requester-crowdworker interaction in microtask marketplaces and weaverelations of trust .


Future research directions

To what extent do crowd workers (and requesters) work (and publish work) across different marketplaces?

How to extract summarized information out of many Crowd Work CV RDF triples?

To what extent does considering cross-platform CV information while selecting candidates, improve the

accuracy of results?


Possible topics for the discussion

� What kind of information would you expect from a Crowd Work CV if youwere a …

� … requester?

� … crowd worker?

� How would you like Crowd Work CVs to be used if you were a …

� … requester?

� … crowd worker?

� Which arguments would you need for adopting the Crowd Work CV approach in your marketplace, if you were a marketplace owner (e.g. MTurk)?

� What pros and cons do you identify?

� Any other comment or question is welcome!

Institute for Web Science and Technologies · Univ ersity of Koblenz-Landau, Germany

Thank you for your attention !

Cristina SarasuaInstitute for Web Science and TechnologiesUniversität [email protected]


References

Kittur, A., Nickerson, J.V., Bernstein, M.S., Gerber, E.M., Aaron, S.,Zimmerman, J., Lease, M., Horton, J.J.: The future of crowd work. In: 16thACM Conference on Computer Supported Cooperative Work (CSCW 2013)(2013)

Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Pick-a-crowd: tell me whatyou like, and I’ll tell you what to do. In: Proceedings of the 22nd internationalconference on World Wide Web (WWW2013) (2013)

Gagan Goel, A.K., Singla, A.: Matching workers expertise with tasks:Incentives in heterogeneous crowdsourcing markets. In: NIPS13 Workshopon Crowdsourcing: Theory, Algorithms and Applications (2013)

Ul Hassan, U., O’Riain, S., Curry, E.: Slua: Towards semantic linking of userswith actions in crowdsourcing. In: CrowdSem (2013)

Khazankin, R., Psaier, H., Schall, D., Dustdar, S.: QoS-based taskscheduling in crowdsourcing environments. In: Proceedings of the 9thinternational conference on Service-Oriented Computing (2011)


References (II)

Bojars, U., Breslin, J. ResumeRDF: Expressing Skill Information on theSemantic Web . In: Proceedings of the 1st International ExpertFinderWorkshop,Knowledge Web General Assembly 2007 (2007)

Celino, I. Human Computation VGI Provenance: Semantic Web-BasedRepresentation and Publishing. In: IEEE Transactions on Geoscience andRemote Sensing. (2013)

Sarasua, C., Thimm, M.: Microtask available, send us your cv! In:International Workshop on Crowd Work and Human Computation(CrowdWork2013) (2013)

crowd work cv: recognition for micro work

Technology

germanycrowd work cv

barcelonacrowd work

netcrowd work cv ontology

basedon ex

publishedat ex

cristina sarasuaexampleex

byuseraccount ex

cristina sarasuarequirements1