These days it seems there's barely a project that gets conceived that doesn't have some element of crowdsourcing involved: from bringing a smartphone mapping service up to speed, to developing the next generation of military amphibious vehicles. Now the secrets to ensuring project leaders get the best possible work out of their army of volunteers have been revealed.
Dana Chandler of MIT and Adam Kapelner of the University of Pennsylvania have published the results of a groundbreaking study examining what makes crowdsourced workers tick and how to get the best results.
They recruited a pool of 2,400 volunteers from Amazon's Mechanical Turk labour spot market to help them out find signs of tumours in a bundle of medical images. They wanted to see whether it was possible to motivate a group of volunteers to produce better results.
“We chose a task that would appear meaningful for many people if given the right context: helping cancer researchers mark tumour cells in medical images,” the team reported in their research paper.
The researchers split their volunteers into three groups, each given slightly different instructions. One group were told in detail about the task, with phrases such as “tumour cells” frequently repeated, driving the point of the task home. Their work had, as the researchers put it been given “meaning”.
Both other groups were merely told how to identify “areas of interest”, with no hint of the task's purpose. One of these groups – those in the “shredded condition” - was also told that their work would not be recorded, with the aim of demotivating them.
That there's a link between motivation and performance is hardly a revelation – and the team did indeed find that the more motivated volunteers produced more, better quality results for lower pay. But their results also threw up some surprises.
Having established the groups, all were told that they would be paid a decreasing fee for each successive image the worker evaluated. This allowed the researchers to explore the financial implications of motivating a crowdsourced workforce.
Finally, the researchers were curious to know whether the different groups would produce better quality work, so they compared the number of tumours correctly identified by each group.
“The 'meaningful condition' seems to increase quantity without a corresponding increase in quality and the 'shredded' treatment decreases quality without a corresponding decrease in quantity,” they wrote.
“As the world begins to outsource more of its work to anonymous pools of labour, it is vital to understand the dynamics of this labour market and the degree to which non-pecuniary incentives matter. This study demonstrates that they do matter, and they matter to a significant degree,” they concluded.
J1043+2408 was observed for more than 10 years, and its radio light curve exhibited a periodic signal repeating in about 563 days
Success of Unity's test flight means Virgin Galactic is now close to taking its first paying tourist into space
V3 puts the pro-level football GPS tracker through its paces, and asks if it's more than a gimmick
Finding refutes many earlier studies that suggest that galaxies don't have much dark matter at the time of their birth