Skip to main content
</tr> </tr> </tr> </tr> </tr> </tr> </tr> </tr> </tr> </tbody> </table>
Topic Readings
  • Overview of uses of crowdsourcing
  • Non-language uses of crowdsourcing
  • types of problems (labels, text, speech, people...)
  • </td>
Taxonomy of crowdsourcing and human computation
  • Categorization system: motivation, quality control, aggregation, human skill, process flow.
  • </td>
Crowdsourcing platforms: Mechanical Turk and CrowdFlower
  • Terminology and mechanics: Turkers, Requesters, HITs, micropayments
  • Demographics and motivation of Mechanical Turk workers
  • </td>
How to set up and run experiment
  • Designing and running experiments (step-by-step overview)
  • Example of quickly using MTurk via the web interface
  • Accessing MTurk via the boto API and the CrowdFlower API that People Pattern created
  • </td>
Ethics of Crowdsourcing
  • Fair wages?
  • Privacy/IRB considerations
  • </td>
Quality Control
  • MTUrk's Reputation system and qualifications
  • Aggregation and redundancy
  • Embedded gold standard data
  • Second-pass reviewing
  • Economic incentives
  • Statistical models
  • Should we do quality control if we are training a statistical model?
  • </td>
Statistical analysis of MTurk data
  • Quality versus quantity
  • Active data collection
  • Accounting for worker variation
  • </td>
Case studies in NLP: Chris Callison-Burch
  • Crowdsourcing Translation
  • Using crowdsourcing to collect data to train SMT systems
  • </td>
Case studies in NLP: Lyle Ungar
  • Word Sense Disambiguation
  • Computational Social Science at the World Well-Being Project (WWBP)
  • </td>
Wrap up