Why people „Turk‟?

Neha Gupta Ethnographer & PhD Researcher

University of Nottingham & Xerox Research Centre Europe (XRCE)

David Martin

Jacki O‟Neill (Now MSR India)

Ben Hanrahan

Image: www.sfep.org.uk

Neha Gupta

Andy Crabtree

Tom Rodden

The University of Nottingham

What is Mechanical Turk?

Co-ordinate a crowd (a large group of people), to do micro-work (mostly small

contributions) that solves problems (that software and one user cannot do) Image: mturk.com

Amazon Mechanical Turk (AMT):

• mturk launched November 2, 2005

• Artificial artificial intelligence

• Work : Requesters Providers / Workforce: Turkers

• Human Intelligence Tasks (HITs): payment often in cents/seconds

• Other crowdsourcing platforms, other types of work e.g. freelancer, odesk, elance

Types of task on AMT

• Micro-tasking: image tagging, duplicate recognition, SEO, data entry, translation, transcription, content generation, obj. classification

Image: mturk.com

Why research on AMT ?

• Little in-depth qualitative studies of Turkers

• Survey of demographics – Ross et. al.

• Turkers Bill of Rights – Irani, Silberman

• Law in crowdsourcing industry – Felstiner

• AMT design – Khanna et. al.

• Concerns: Are workers perfectly happy with conditions and wages?

• Information deficit/opacity

• Lack of choice and opportunities

• Imbalance / Lack of power of redress

How did we conduct research ?

• Virtual ethnography of online forums like mturkforum and Turker Nation by Dr. Dave Martin, XRCE

• Study involved deep reading of threads, collection of key threads/ posts, recurrent themes, exemplars

• Ethnographic Studies of Turkers in their respective homes and places of work in India

• Study involved Interviews (skype, phone and in-person), walkthroughs, observations, data capture through audio / video / screen recordings

Why Do Turkers Turk?

• Motivated by earning money

• Learn new things, “earn money while having fun”, but the aim still is to earn money

• Compare themselves to others

• Set themselves goals and targets

• For some AMT is a compromise: Problems accessing the regular job market, Need to supplement their income, Some housebound.

Process of Turking for a Turker

Look for HIT Accept HIT Work on HIT

Submit HIT and wait for

approval

If approved, get paid, if not, get rejected.

Wait for feedback

Wait for money to arrive

Go back for more work

Worry about blocks, ratings,

qualifications, suspensions,

future work.

Turking = flexible working?

Example 1: Is turking flexible?

Mansoor

You cannot find much work on AMT during the day… in the morning I go to institute for BBA for taught and practical classes.. Class starts at 11.30 am, I leave home at 10.30am, we have 1 hour theory, 1 hour practical and then we practice for 1 – 2 hrs.. and then I come back at 4 pm and rest. Then do work on AMT. I do the most work at night time after 7.30 sometimes till 1, 2 or 3 am at night… I also have to do house work in the morning […] but if I got more work in the mornings I am willing to sit and work all day.

Turkers Turking for Fun?

Example 2, 3: Turkers Turking for Fun?

Danturker

This attitude would be requesters dream come true. The workers come here to have fun and play and the lousy pay for work is not an issue. This attitude helps create low pay for the AMT work force that does care about fair pay.

Larak56

I agree with most everyone here. While I do find some of the HITS fun and actually learn an incredible amount by doing HITS, I do it for the cash.

Varying Expectations

Example 4: Varying Expectations

Defectturk

I would rather pull weeds, do laundry, wash the car, clean the toilet than work for this shit paying requester.

Clunky interface, too many tags/ratings/whatever per hit, works out to about a buck an hour. They should be banned from mturk, or even better forced to work on their own shit paying hits.

Varying Expectations

Example 5: Varying Expectations

Nagen

I only do tasks that I have done before so that I am familiar with the instructions. […] But I think one of past few times, something I did went wrong as I got some warning emails from Amazon and finally got blocked. I was pretty sure I had done the task correctly and had seen the instructions before but perhaps I misunderstood something and now I might get suspended. We keep trying to contact them (AMT) via emails. I still have around $50 in my account. They are an American company, they are pukka (genuine/honest/honourable) in their procedures, I trust they‟ll be fair to me and give me my money back..

Reputation of Turkers

Example 6: Reputation based on qualifications, ratings

Mansoor

Reputation of Turkers

Example 6: Reputation based on qualifications, ratings

Mansoor

Now these HITs are available 24X7 for these guys (who have score of a 100) and I see them everyday but can‟t do them, which hurts a little. I was careless when the Requester was testing Turkers with qualifications, so my score is less. I should have worked hard [...] but I was in a rush.”

How Much Do Turkers Earn?

• Considerable variation but Turking is low wage work, wages off-set against time taken to find work, amount of concentration required etc.

• Highest earners on Turker Nation ~$15-16k per year High earners in India ~$8k-$10k per year Low earnings can be as low as $260 - $520 per year ($5-10 per week)

• U.S Turkers generally aspire to earning $7-10 per hour. Indian Turkers aspire to earn according to benchmarks set e.g. Call centre work ~ Rs 10,000 ~ $160 per month

• Lower paid easy work to increase reputation and ranking e.g. Oscar Smith

How Much Do Turkers Earn?

• Lower paid easy work to increase reputation and ranking e.g. Oscar Smith

Invisible Turkers & their hidden work

• AMT as a labour market:

• Easy for requesters to devalue/ignore workers

• Invisibility accentuated by anonymity, distance, digital interaction

The work to make the Turking work:

• Work of searching, learning, optimising, organising. hidden, no monetary compensation

• Discussing best practices with fellow-turkers, supporters & requesters

Hidden work for Turkers

Good and Bad Requesters

Example 7: A Good Requester

Explodeman

All hits I have done for Project Tatooine in the past week have approved and are fair pay. Good Requester.

Good and Bad Requesters

Example 8: A Bad Requester

Iambob

I have done a number of his hits. Then one day I get this email:

“You received a bonus from Top Page Solutions. The value of your bonus is $0.01 USD yes ONE CENT. The requester included this note:

If you spent half the amount of time looking for a job as you spent scamming your way through Mechanical Turk HITs maybe you’d be able to find a real job”

The next day I get a block message from him. WTF?! … My hit total is under 2000 so I do not see how I am a scammer in his mind.

Turker Ethics?

Both U.S. and Indian Turkers take a strong ethical position

• Duty to be honest about requesters and admit mistakes.

• Good behaviour is practically and ethically the best way to help fellow Turkers and influence the market

• Ripping requesters off is not endorsed

• Turkers would like their voice heard and help from Amazon

Turker-Requester Relationships

• U.S. Turkers like anonymity and flexibility, Indian Turkers don‟t mind anonymity but prize relationship with American requesters

• Turkers want productive relationships with requesters

• They want fair pay for fair work: decent wages, fairness in judging work, timely payment

• Value courteous and fair communication

• Want to communicate with requesters about work, design of tasks etc.

• Respect works both ways: regular work from good requesters is highly prized

Key Takeaways for Requesters

• Think about Turkers as people, colleagues

• Communicate your thoughts

• Pay Turkers fairly

• Provide feedback (positive / negative) when possible

• Try to be involved / engaged

Thank You !

If you use / would like to use AMT to get some work done, I would love to hear your experiences, questions or anything you would like to

discuss. Please feel free to talk to me.

neha.gupta@nottingham.ac.uk

@Neha_Gi_Ji