Registration Number: 090125502 Name: Mr Mahesh Kumar Gudala

Word-length:

2314

Introduction and context: Amazon Mechanical Turk www.mturk.com is one of the well known crowdsourcing marketplace website. MTurk website is the marketplace where requesters post the work by dividing the complex tasks into small and simple tasks which is known as Human Intelligent Tasks (HITs)in MTurk where anyone who is registered in the websites can participate to do the tasks posted and get paid for the work done that produces accurate result thereby satisfied by the Requester, which is advantageous for the requesters rather than hiring a temporary employee and unsure whether the produced result is right or wrong, where as in Mechanical Turk the requester can select the best workforce that may require qualification, location and success rate of the previous tasks that they carried out and pay them only when they get the right solution that they are looking for from the tasks assigned. The Amazon will take care about the payments. According to the website the range of tasks varies from cleaning the data like data entry, algorithm testing, and data verification to moderating photos and content to the feedback of the products like test research, product usability testing. Also categorising items like classification, sentiment analysis and tagging of different items. Here the website is classified into two different segments: 1. Requesters: Requesters are the ones who post the work which is known as Human Intelligent Tasks (HITs) where they design, publish and manage the hits 2. Workers: Where they have the choice to choose the tasks and get paid for the work done that produces the correct result which is satisfied by the requester.

Aim: The Aim of the research is to study and categorise the variety of projects (HITs) that are being carried out on Amazons Mechanical Turk and to monitor the activities over a period of time.

Objectives: Carry out the detailed analysis about Crowdsourcing and Amazon Mechanical Turk. Analyze and classify the tasks happening in the Turk based on the categories. Monitor the Activities and changes on the Amazon Mechanical Turk over a period of time. To collect the data by accomplishing the in depth literature review by pulling together the available statistics.

Research Question What is Crowd sourcing? What are the popular systems in Crowdsourcing? What are the advantages and disadvantages of the Crowdsourcing?

What is Amazons Mechanical Turk? What kinds of activities are carried out in Mechanical Turk? What are the most popular activities in MTurk?

Literature Review: This is the main area in the project as it provides thorough knowledge about the research stated by Gill, John (2002) about literature review as A statement of the state of art and the major questions in the field under consideration. As the Project is about investigating the tasks carried out in the Amazon Mechanical Turk website, internet will be the main resource which involves extensive research of the activities carried out over a set of time. Apart from that an in-depth knowledge of crowdsourcing will be included by taking references of the various electronic journals to get clear idea of what crowdsourcing is as Amazon Mechanical Turk is one of the most popular systems of the crowdsourcing systems it would be better to know about crowdsourcing before moving into the main research project which is the main aim so that it will get clear understanding to the readers. Crowdsourcing is the act of taking a job traditionally performed by a designated employee and outsourcing it to an undefined, generally large group of people in the form of an open call. Jeff Howe (2006). For the first time the term crowd sourcing appeared in the wired magazine article i.e. The Rise of Crowdsourcing and since then the demand for crowd sourcing is rising and turned out to become very popular where people i.e. crowd who have access to internet from different fields come together and participate in numerous ways like posting reviews about the product, showcasing the designs created by one users and let others users vote the best design either to get fame or rewards. Below are the examples of well known crowdsourcing systems. Jeff Howe (2006) in his book Crowdsourcing: How the power of the crowd is driving the future of business gave many scenarios of crowdsourcing systems. One of them is about threadless.com, a website which was started by two young people where anyone can showcase their own designs and the best design will get free t-shirt as well as reward and also the buyers will get reward by referring others likewise the company is now doing multimillion dollar business. Wikipedia is one of the most famous crowdsourcing websites where anyone can edit the content posted by anyone thats why its one of the well known websites all over web.While individual users of Flickr are simply using the site to store and share photos, their collective activity reveals a striking amount of geographic and visual information about the world. D. Crandall et al. (2009). As media sharing sites likes flickr, YouTube, Picasa which uses geo-tagging to help people to find information about a particular place of what they are searching for where the content was posted by other users. Its like sharing knowledge among others by posting the content for free. Crowdsourcing is exemplified by websites such as Digg, Flickr, YouTube, and Wikipedia. Huberman, B.A (2008). With the ability to post the content in various forms in the above websites millions of people create and share the content there by generating more traffic to this websites rather than the sites from where the original content was posted.

There are few disadvantages too apart from the above stated advantageous scenarios. One of the disadvantages with crowdsourcing is towards social behaviour. According to

Mark (2010) Crowds disturb, escalate and then threaten the social order. There might be differences in the solutions as the opinion from different backgrounds varies even though the work is carried out which is unlike in the office environment where the proof of work done cannot be seen. As it is anonymous network many people feel free to participate in this system. Only people who connect to the internet can participate in the crowdsourcing. There might not be the accurate solution as the crowd from diverse backgrounds participate and there might be differences in the solution as well. There are many systems alike and Amazon Mechanical Turk with tagline Artificial Artificial Intelligence is one of the most popular crowd sourcing website where it has its own terms like Requester, HIT and workers. The requesters post the tasks where the workers can select the tasks that they are interested in but few tasks may have certain criteria like their approval rate, number of tasks completed before with success rate and metrics which are maintained by MTurk and also there might be qualification test in order to do that task and they are rewarded with money depending on the task from a minimum of $0.01 to $0.10 into their account which can be credited to their bank account or Amazon gift certificates that can be used for shopping in Amazon. This site is well known for the success of the projects where the project is divided into small tasks and each task is carried out by many workers set by the requester in order to compare and review the work done and select the best result. The requester can choose the number of runs of the task and also set the time limit, if the project expires within a time and there are no runs then again it is posted at the later date. The requester may accept or reject the result of the worker but it might affect the workers rating in the website. Amazon gets 10% profit of each task from the requesters if the task is successful and workers can work from anywhere in the world that are connected to the internet. MTurk workers could produce annotation judgements that are comparable to experts. Callison Burch (2009). It says why Amazon MTurk is famous as many people work on the same task set by the requester so that the requester can select the best result among the tasks submitted.

Why Amazon Mechanical Turk? The main advantage of MTurk is the availability of the capable work force as thousands of people from different parts of the world participate and choose the tasks and work on it where they get paid for the satisfied result from the worker who posts the work and thereby it increases the worker rating so that there will be more chances for him to participate in other tasks that may require the basic qualification. I noticed number of the Human Intelligence Tasks fluctuates every minute as thousands of HITs are posted as well as solved for every hour. MTurk provides results at a cheaper and faster rate compared to traditional lab work. Sean Liu (2010). Its clear from the above statement that MTurk produces quality results within an allotted time as it is cost effective as well as time saving for workers as well as requesters. As the interaction is over internet there wont be any delay in communication between the worker and requester. Many large complex projects are divided into smaller simple tasks where they are solved independently by the workers as it produces the accurate result in very short time.

The cost of implementation of the project is very cheap compared to the onsite implementation as hiring a temporary contract employee for very small tasks of which they are unsure of the results from the produced work and again searching for the right employee where as in MTurk many people who are qualified for that task as they take part in the qualification test, work on it until it reaches the number of runs requested by the worker so that they get the accurate result. In Quadrant of Euphoria assessment by the National Taiwan University they conducted few experiments on three different platforms i.e. Community, laboratory and MTurk. The results proved that the investment for the MTurk is very less comparative to the Laboratory. And also the number of runs is less in MTurk compared to the others.

Methodology: This is the theoretical perspective of the research that is the overall nature of the research activity, although the term is applied to many aspects of the research process in various disciplines. Alison Jane P (2007). Research objectives and questions will be well explained in an ordered format so that it will be understood by everyone as it starts from a basic concept to the most advanced stages as the project continues till end. The methodology to be followed for this research project is quantitative methodology as it involves the analysis of the MTurk activities and interpreting it according to the criteria. To provide in-depth analysis of the data collected in the literature review. As data is collected from various resources such as internet, journals, papers and databases it produces better outcome with consistent results. Studying the Amazon Mechanical Turk System especially by monitoring the activities in the website, by reviewing the background information and analyse the outcome of the data in the literature review. In order to collect the Information and study the detailed analysis of the MTurk there is a need to have an account in the website as it is not possible to see the tasks without MTurk account so one account is created in the website which is free of cost to monitor the activities over a period of time. The monitoring of the activities includes scanning the detailed tasks and depicting them in the graphs that identifies the main tasks that are popular in MTurk sorted by the reward, availability and creation date of the Human Intelligent Tasks. Based on the graphs and the study of activities over a particular period of time it distinguishes the tasks carried out in MTurk. The project is divided into few phases where the first phase will be about the Crowdsourcing which is divided into sub-topics such as the introducing the concept of Crowdsourcing and the popular systems of Crowdsourcing. The next topic will leads to the advantages and disadvantages of crowdsourcing. The second phase is about Amazon Mechanical Turk which is the core part of this proposed research which is divided into sub-topics such as giving the detailed information about the Mechanical Turk ranging from introduction to the features of MTurk and then the next topic is about focussing on the activities carried out in

MTurk by investigating and categorising the tasks which is the central theme of the project as it is the main aim of the project. The last topic will be about the popular activities in MTurk which is based on the previous topic as it classifies the range of tasks carried out in the Amazon Mechanical Turk. A detailed abstract will be given before the introduction of the project to give overall demonstration of the research to the readers. Data Collection: As it involves analyzing the MTurk website, Internet is the main source of medium to accumulate the background information for the literature review. It includes electronic journals, articles and databases related to crowd sourcing and MTurk as well as MTurk website. A list of the collected materials will be maintained and analyzed carefully in order to avoid ambiguity

Practicalities: Only information about crowdsourcing and MTurk will be based on data collection where as the investigation and categorisation of tasks in MTurk and

monitoring the activities over a period of time will involve real facts based on the experiments conducted in the website which will provided in the graphical depiction of each and every details researched.
Ethical Aspects: There wont be any involvement of any humans in this project like questionnaires or taking surveys as the project is to analyze the activities in the website.

References: Amazon Mechanical Turk http://www.mturk.com Callison Burch, C. (2009) Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk. In Proc. EMNLP 2009, ACL and AFNLP, 286 295 Corney, J. et. al. (2010). Towards crowdsourcing translation tasks in library cataloguing, a pilot study." Digital Ecosystems and Technologies (DEST), IEEE International Conference, 572 577. Crowdsourcing: Consumers as Creators (2006), http://www.businessweek.com/innovate/content/jul2006/id20060713_755844.htm, [accessed, 02-Mar-2011]

Crowdsourcing and crowdfunding News and Headlines http://www.crowdsourcing.org/, [accessed, 02-Mar-2011] Ganjisaffar, Y.; Javanmardi, S.; Lopes, C. (2009). "Review-Based Ranking of Wikipedia Articles," Computational Aspects of Social Networks, 98-104 Gill, John (2002). Research methods for managers. London: Sage. Alison Jane P (2007). Research Methods in Information. London: Facet Publishing. Huberman, B.A. (2008). "Crowdsourcing and Attention," Multimedia, IEEE 41(11), 103-105. Huberman, B.A. (2009). Crowdsourcing attention and productivity. Journal of Information Science 35, 758-65. Jeff Howe (2008). Crowdsourcing: How the power of crowd is Driving the future of Business. London: Random House Business. Kuan-Ta Chen. et al. (2010). "Quadrant of euphoria: a crowdsourcing platform for QoE assessment," Network, IEEE, 24(2), 28-35. Liu, S. et al. (2010). "A collective data generation method for speech language models," Spoken Language Technology Workshop (SLT), EEE, 223-228. Mark N.W. (2011). Reconfiguring the sociology of the crowd: exploring crowdsourcing. Emerald Insight 31, 6-20. Sorokin, A. et al. (2010). "People helping robots helping people: Crowdsourcing for grasping novel objects." Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference, 2117-2122. The Rise of Crowdsourcing (2006), http://www.wired.com/wired/archive/14.06/crowds.html, [accessed, 02-Mar-2011] Newsam.S. (2010). "Crowdsourcing What Is Where: Community-Contributed Photos as Volunteered Geographic Information," Multimedia, IEEE 17(4), 36-45. What is Crowdsourcing? http://what-is-crowdsourcing.com/, [accessed, 02-Mar-2011] Zhaojun Yang et al. (2010). "Collection of user judgments on spoken dialog system with crowdsourcing," Spoken Language Technology Workshop (SLT), IEEE, 277282.