Skip to main content
You can watch the lectures live via Zoom on Tuesdays and Thursdays 3-4:30pm Eastern or you can view recordings of the lecture a few hours after the live lectures finish.

The lecture schedule will be updated as the term progresses.

Date Topic Recommended Readings
Thu, Jan 21, 2021 Intro Lecture [slides] [video] James Surowiecki, The Wisdom of Crowds: Introduction
UW iSchool's "Calling Bullshit" course, Example of fake news
International Labour Oranization, Digital Labour Platforms and the Future of Work (Chapter 1 and 2)
Tue, Jan 26, 2021 Crowd Workers [slides] [video] Jeff Bigham, My MTurk (half) Workday
Lilly Irani and Six Silberman, Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk
Chris Callison-Burch, Crowd-Workers: Aggregating Information Across Turkers To Help Them Find Higher Paying Work
Ben Bederson and Alex Quinn, Web Workers, Unite! Addressing Challenges of Online Laborers
David Martin, Benjamin Hanrahan, Jacki O’Neill and Neha Gupta, Being A Turker
Saiph Savage, Chun Chiang, Susumu Saito, Carlos Toxtli, and Jeffrey Bigham, Becoming the Super Turker: Increasing Wages via a Strategy from High Earning Workers
Neha Gupta, David Martin, Benjamin Hanrahan, and Jacki O’Neill, Turk-Life in India
Karen Fort, Gilles Adda and Bretonnel Cohen, Amazon Mechanical Turk: Gold Mine or Coal Mine?
Thu, Jan 28, 2021 The Amazon Mechanical Turk crowdsourcing platform [slides] [video] Jeff Barr and Luis Felipe Cabrera, AI Gets a Brain
Gabriele Paolacci, Jesse Chandler and Panos Ipeirotis, Running experiments on Amazon Mechanical Turk
Amazon Mechanical Turk
Mechanical Turk blog
Chris Callison-Burch, Fast, Cheap and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk
Tue, Feb 2, 2021 Course Selection Period ends
Tue, Feb 2, 2021 Ethnographic Studies of Turkers [slides] [video] by Benjamin V. Hanrahan David Martin, Benjamin Hanrahan, Jacki O’Neill and Neha Gupta, Being A Turker
Neha Gupta, David Martin, Benjamin Hanrahan, and Jacki O’Neill, Turk-Life in India
Thu, Feb 4, 2021 A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk [slides] [video] Kotaro Hara, Abi Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch and Jeff Bigham, A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Benjamin V. Hanrahan, Jeffrey P. Bigham and Chris Callison-Burch, Worker Demographics and Earnings on Amazon Mechanical Turk: An Exploratory Analysis.
Abi Adams-Prassl, Chris Callison-Burch, Kotaro Hara, Kristy Milland, Saiph Savage and Jeffrey Bigham, (In Submission) The Gender Wage Gap in an Online Labour Market: The Cost of Interruptions
Tue, Feb 9, 2021 - Data Labeling Startup [slides] [video] by Ivan Lee
Tue, Feb 9, 2021 An Analytic Framework for Human Computation [slides] [video] Alex Quinn and Ben Bederson, Human Computation: A Survey and Taxonomy of a Growing Field
Panos Ipeirotis, Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers?
Thu, Feb 11, 2021 Citizen Science [slides] [video] by Christopher Cieri James Fiumara, Christopher Cieri, Jonathan Wright, Mark Liberman, LanguageARC: Developing Language Resources Through Citizen Linguistics
Christopher Cieri, James Fiumara, Mark Liberman, Chris Callison-Burch, Jonathan Wright, Introducing NIEUW: Novel Incentives and Workflows for Eliciting Linguistic Data
Christopher Cieri, James Fiumara, LanguageARC – a tutorial
Thu, Feb 11, 2021 Games with a Purpose [slides] [video] by Karën Fort John Chamberlain, Karën Fort, Ugo Kruschwitz, Mathieu Lafourcade, Massimo Poesio, Using Games to Create Language Resources: Successes and Limitations of the Approach
Bruno Guillaume, Karën Fort, Nicolas Lefèbvre, Crowdsourcing Complex Language Resources: Playing to Annotate Dependency Syntax
Karën Fort, Bruno Guillaume, Yann-Alan Pilatte, Mathieu Constant, Nicolas Lefèbvre, Rigor Mortis: Annotating MWEs with a Gamified Platform
Alice Millour, Marianne Grace Araneta, Ivana Lazić Konjik, Annalisa Raffone, Yann-Alan Pilatte, Karën Fort, pKatana and Grand Guru: a Game of the Lost Words (DEMO)
Tue, Feb 16, 2021 Python Review [slides] [video]
Thu, Feb 18, 2021 Python Review - part 2 [slides] [video]
Fri, Feb 19, 2021 Grade Type Change Deadline
Tue, Feb 23, 2021 Intro to Machine Learning [slides] [video] Jeremy Howard /, Practical Deep Learning for Coders
Daniel Jurafsky and James Martin, Logistic Regression
Thu, Feb 25, 2021 How Crowdsourcing Enabled Computer Vision [slides] [video] Dave Gershgorn - Quartz, The data that transformed AI research – and possibly the world
Fei-Fei Li, slides about ImageNet: Crowdsourcing, benchmarking & other cool things
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei, ImageNet: A Large-Scale Hierarchical Image Database
Stanford University instructors, notes for Stanford's CS231n course "Convolutional Neural Networks for Visual Recognition"
Tue, Mar 2, 2021 Crowdsourcing Translation [slides] [video] Omar Zaidan and Chris Callison-Burch, Crowdsourcing Translation: Professional Quality from Non-Professionals
Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch, The Language Demographics of Amazon Mechanical Turk
Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch, Machine Translation of Arabic Dialects
Chris Callison-Burch, Fast, Cheap and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk
Thu, Mar 4, 2021 Linguistic Wisdom From the Crowd [video] by Michael Tseng Nancy Chang, Russell Lee-Goldman, and Michael Tseng, Linguistic Wisdom from the Crowd
Christina Funk, Michael Tseng, Ravindran Rajakumar, and Linne Ha, Community-Driven Crowdsourcing: Data Collection with Local Developers
Jennimaria Palomaki, Olivia Rhinehart, and Michael Tseng, A Case for a Range of Acceptable Annotations
Clark et al., TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
Kwiatkowski et al., Natural Questions: A Benchmark for Question Answering Research
Min et al., AmbigQA: Answering Ambiguous Open-domain Questions
Parikh et al., ToTTo: A Controlled Table-To-Text Generation Dataset
Zhang et al., PAWS: Paraphrase Adversaries from Word Scrambling
Tue, Mar 9, 2021 Quality Control part 1 [slides] [video] Luis von Ahn and Laura Dabbish, Labeling Images with a Computer Game
Rion Snow, Brendan O’Connor, Daniel Jurafsky and Andrew Ng, Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
Winter Mason and Duncan Watts, Financial Incentives and the Performance of Crowds
Thu, Mar 11, 2021 No class (Spring Break)
Tue, Mar 16, 2021 Quality Control part 2 [slides] [video] Anindya Ghose, Panos Ipeirotis, and Arun Sundararajan, Opinion Mining Using Econometrics: A Case Study on Reputation Systems
Kazushi Ikeda and Michael Bernstein, Pay It Backward: Per-Task Payments on Crowdsourcing Platforms Reduce Productivity
Thu, Mar 18, 2021 Quality Control part 3 [slides] [video] A.P. Dawid and A.M. Skene, Maximum Likelihood Estimation of Observer Error-rates using the EM Algorithm
Panos Ipeirotis, Foster Provost, Jing Wang, Quality Management on Amazon Mechanical Turk
Tue, Mar 23, 2021 Crowdsourcing and human computer interaction (HCI) design [slides] [video] Michael Bernstein, Greg Little, Rob Miller, Björn Hartmann, Mark Ackerman, David Karger, David Crowell, and Katrina Panovich, Soylent: A Word Processor with a Crowd Inside
Jeff Bigham, Real-Time Crowd Support for People with Disabilities (video)
Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Vizwiz: Nearly Real-time Answers to Visual Questions
Thu, Mar 25, 2021 Crowdsourcing and HCI part 2 [slides] [video] Michael Bernstein, Joel Brandt, Rob Miller, and David Karger, Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces
Nicolas Kokkalis, Thomas Köhn, Carl Pfeiffer, Dima Chornyi, Michael Bernstein, and Scott Klemmer, EmailValet: Managing Email Overload through Private, Accountable Crowdsourcing
Mon, Mar 29, 2021 Last day to withdraw from a course
Tue, Mar 30, 2021 No class (Engagement Day)
Thu, Apr 1, 2021 Programming concepts for human computation [slides] [video] Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller, TurKit: Human Computation Algorithms on Mechanical Turk
Tue, Apr 6, 2021 Toward Future Crowd-AI Systems [video] by Ting-Hao (Kenneth) Huang Ting-Hao K. Huang, Joseph Chee Chang, Jeffrey P. Bigham, Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
Gimlet Media, Sandra (a sci-fi thriller podcast about crowd-powered conversational agents)
Chieh-Yang Huang, Shih-Hong Huang, Ting-Hao 'Kenneth' Huang, Heteroglossia: In-Situ Story Ideation with the Crowd
Ting-Hao 'Kenneth' Huang, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Yen-Chia Hsu, C. Lee Giles, CODA-19: Using a Non-Expert Crowd to Annotate Research Aspects on 10,000+ Abstracts in the COVID-19 Open Research Dataset
Thu, Apr 8, 2021 Iterative and Parallel Processing [slides] [video] Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller, Exploring Iterative and Parallel Human Computation Processes
Tue, Apr 13, 2021 Minimum Wage Pledge for Requesters by Six Silberman and Hannah Johnston WeAreDynamo, Guidelines for Academic Requesters
Crowdsourcing Code of Conduct
Oxford Internet Institute, Fairwork Principles
Amy Glasmeier, MIT Living Wage Calculator
Thu, Apr 15, 2021 Expert Crowds [slides] [video] Panos Ipeirotis and Evgeniy Gabrilovich, Quizz: Targeted Crowdsourcing with a Billion (Potential) Users
Maria Christoforaki and Panos Ipeirotis, STEP: A Scalable Testing and Evaluation Platform
Wikipedia, Item response theory
Tue, Apr 20, 2021 What data can tell us about sex and relationships [slides] Rajesh Ranganath, Dan Jurafsky, and Dan McFarland, Detecting Flirting and its Misperception in Speed-Dates
Christian Rudder, Dataclysm: Who We Are (When We Think No One's Looking)
Christian Rudder - OKCupid, OKCupid's blog
Thu, Apr 22, 2021 Prediction Markets [slides] Prediction Markets, Justin Wolfers and Eric Zitzewitz
Using Prediction Markets to Track Information Flows: Evidence from Google, Bo Cowgill, Justin Wolfers and Eric Zitzewitz
Tue, Apr 27, 2021 Extracting More Wisdom from the Crowd by John McCoy Dražen Prelec, H. Sebastian Seung, and John McCoy, A Solution to the Single-Question Crowd Wisdom Problem
Thu, Apr 29, 2021 Understanding Human Happiness by Matt Killingsworth Matt Killingsworth,