Skip to main content
You can watch recordings of the lecture videos online.

The lecture schedule will be updated as the term progresses.

Date Topic Required Readings
Thu, Jan 17, 2019 Intro Lecture [slides] [video] James Surowiecki, The Wisdom of Crowds: Introduction
International Labour Oranization, Digital Labour Platforms and the Future of Work (Chapter 1 and 2)
Tue, Jan 22, 2019 Crowd Workers [slides] [video] Jeff Bigham, My MTurk (half) Workday
Lilly Irani and Six Silberman, Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk
Chris Callison-Burch, Crowd-Workers: Aggregating Information Across Turkers To Help Them Find Higher Paying Work
Ben Bederson and Alex Quinn, Web Workers, Unite! Addressing Challenges of Online Laborers
David Martin, Benjamin Hanrahan, Jacki O’Neill and Neha Gupta, Being A Turker
Karen Fort, Gilles Adda and Bretonnel Cohen, Amazon Mechanical Turk: Gold Mine or Coal Mine?
Thu, Jan 24, 2019 Gig Work and the Platform Economy [slides] [video] Kotaro Hara, Abi Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch and Jeff Bigham, A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Pew Research Center, Gig Work, Online Selling and Home Sharing
JPMorgan Chase Institute, Paychecks, Paydays, and the Online Platform Economy
The Supreme Court of California, Decision on workers classified as employees or as independent contractors
Tue, Jan 29, 2019 Governing Crowd Work [slides] [video] by Six Silberman Lilly Irani and Six Silberman, Turkopticon: Interrupting worker invisibility in Amazon Mechanical Turk
Thu, Jan 31, 2019 The Amazon Mechanical Turk crowdsourcing platform [slides] [video] Jeff Barr and Luis Felipe Cabrera, AI Gets a Brain
Gabriele Paolacci, Jesse Chandler and Panos Ipeirotis, Running experiments on Amazon Mechanical Turk
Amazon Mechanical Turk
Mechanical Turk blog
Chris Callison-Burch, Fast, Cheap and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk
Tue, Feb 5, 2019 Dataturks: Machine learning data annotations made super easy for teams [slides] [video] by Mohan Gupta Dataturks
Dataturks installation guide
Thu, Feb 7, 2019 What is the size of the crowdsourcing marketplace? [slides] [video] Panos Ipeirotis, Analyzing the Amazon Mechanical Turk Marketplace
Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panos Ipeirotis, Philippe Cudré-Mauroux, The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk
Adam Marcus and Aditya Parameswaran, Crowdsourced Data Management: Industry and Academic Perspectives
World Bank, The Global Opportunity in Online Outsourcing
Tue, Feb 12, 2019 Intro to Machine Learning [slides] [video] Jeremy Howard / fast.ai, Practical Deep Learning for Coders
Daniel Jurafsky and James Martin, Logistic Regression
Thu, Feb 14, 2019 How Crowdsourcing Enabled Computer Vision [slides] [video] Dave Gershgorn - Quartz, The data that transformed AI research – and possibly the world
Fei-Fei Li, slides about ImageNet: Crowdsourcing, benchmarking & other cool things
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei, ImageNet: A Large-Scale Hierarchical Image Database
Stanford University instructors, notes for Stanford's CS231n course "Convolutional Neural Networks for Visual Recognition"
Tue, Feb 19, 2019 Taxonomy of crowdsourcing and human computation [slides] [video] Alex Quinn and Ben Bederson, Human Computation: A Survey and Taxonomy of a Growing Field
Panos Ipeirotis, Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers?
Thu, Feb 21, 2019 Programming concepts for human computation [slides] [video] Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller, TurKit: Human Computation Algorithms on Mechanical Turk
Fri, Feb 22, 2019 Drop Deadline
Tue, Feb 26, 2019 Iterative and Parallel Processing [slides] [video] Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller, Exploring Iterative and Parallel Human Computation Processes
Thu, Feb 28, 2019 Quality Control part 1 [slides] [video] Luis von Ahn and Laura Dabbish, Labeling Images with a Computer Game
Rion Snow, Brendan O’Connor, Daniel Jurafsky and Andrew Ng, Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
Winter Mason and Duncan Watts, Financial Incentives and the Performance of Crowds
Tue, Mar 5, 2019 No Lecture - Spring Break
Thu, Mar 7, 2019 No Lecture - Spring Break
Tue, Mar 12, 2019 Quality Control part 2 [slides] [video] Anindya Ghose, Panos Ipeirotis, and Arun Sundararajan, Opinion Mining Using Econometrics: A Case Study on Reputation Systems
Kazushi Ikeda and Michael Bernstein, Pay It Backward: Per-Task Payments on Crowdsourcing Platforms Reduce Productivity
Thu, Mar 14, 2019 Extracting More Wisdom from the Crowd [video] by John McCoy Dražen Prelec, H. Sebastian Seung, and John McCoy, A Solution to the Single-Question Crowd Wisdom Problem
Tue, Mar 19, 2019 Crowdsourcing and human computer interaction (HCI) design [slides] [video] Michael Bernstein, Greg Little, Rob Miller, Björn Hartmann, Mark Ackerman, David Karger, David Crowell, and Katrina Panovich, Soylent: A Word Processor with a Crowd Inside
Jeff Bigham, Real-Time Crowd Support for People with Disabilities (video)
Thu, Mar 21, 2019 Crowdsourcing and HCI part 2 [slides] [video] Michael Bernstein, Joel Brandt, Rob Miller, and David Karger, Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces
Nicolas Kokkalis, Thomas Köhn, Carl Pfeiffer, Dima Chornyi, Michael Bernstein, and Scott Klemmer, EmailValet: Managing Email Overload through Private, Accountable Crowdsourcing
Tue, Mar 26, 2019 guest lecture on Crowd-Powered Conversational Assistants [slides] [video] by Ting-Hao (Kenneth) Huang Ting-Hao K. Huang, Joseph Chee Chang, Jeffrey P. Bigham, Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
Ting-Hao K. Huang, Walter S. Lasecki, Amos Azaria, Jeffrey P. Bigham, Is there anything else I can help you with? Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent
Gimlet Media, Sandra (a sci-fi thriller podcast about crowd-powered conversational agents)
Thu, Mar 28, 2019 in-class feedback on term project milestone 1
Tue, Apr 2, 2019 Augmenting Human Work and Learning [video] by Jonathan Bragg Jonathan Bragg, Mausam, Daniel S. Weld, Sprout: Crowd-Powered Task Design for Crowdsourcing
Meng-Han Wu, Alexander J. Quinn, Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk
Jonathan Bragg, Mausam, Daniel S. Weld, Crowdsourcing Multi-Label Classification for Taxonomy Creation
Daniel S. Weld, Mausam, Christopher H. Lin, Jonathan Bragg, Artificial Intelligence and Collective Intelligence
Thu, Apr 4, 2019 Crowdsourcing Translation [slides] [video] Omar Zaidan and Chris Callison-Burch, Crowdsourcing Translation: Professional Quality from Non-Professionals
Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch, The Language Demographics of Amazon Mechanical Turk
Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch, Machine Translation of Arabic Dialects
Fri, Apr 5, 2019 Withdraw Deadline
Tue, Apr 9, 2019 Crowdsourcing for low resource langauges by Stephen Mayhew Stephen Mayhew and Dan Roth, TALEN: Tool for Annotating Low-resource ENtities
Tue, Apr 9, 2019 Crowdsourcing for Automatic Speech Recognition [slides] [video] Scott Novotney and Chris Callison-Burch, Cheap, Fast and Good Enough: Automatic Speech Recognition with Non-Expert Transcription
Maxine Eskénazi and others, Crowdsourcing for Speech Processing: Applications to Data Collection, Transcription and Assessment
Eric Fosler-Lussier, (Video) Introduction to Automatic Speech Recognition
Preethi Jyothi and Mark Hasegawa-Johnson, Acquiring speech transcriptions using mismatched crowdsourcing
Preethi Jyothi and Mark Hasegawa-Johnson, Dataset: SBS Audio with Matched and Mismatched Transcripts
Mozilla, Common Voice (crowdsourced )
Thu, Apr 11, 2019 Expert Crowds [slides] [video] Panos Ipeirotis and Evgeniy Gabrilovich, Quizz: Targeted Crowdsourcing with a Billion (Potential) Users
Maria Christoforaki and Panos Ipeirotis, STEP: A Scalable Testing and Evaluation Platform
Optional: Wikipedia, Item response theory
Tue, Apr 16, 2019 Understanding Human Happiness by Matt Killingsworth Matt Killingsworth, trackyourhappiness.org
Thu, Apr 18, 2019 What data can tell us about sex and relationships [slides] Rajesh Ranganath, Dan Jurafsky, and Dan McFarland, Detecting Flirting and its Misperception in Speed-Dates
Christian Rudder, Dataclysm: Who We Are (When We Think No One's Looking)
Christian Rudder - OKCupid, OKCupid's blog
Tue, Apr 23, 2019 tenative - Prediction Markets
Thu, Apr 25, 2019 Ghost Work by Mary L. Gray and Siddarth Suri Optional: Mary L. Gray and Siddarth Suri, Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass
Tue, Apr 30, 2019 tenative - Goodbye
Wed, May 8, 2019 Final presentations during fial exam slot - Wednesday, May 8, 2019 from 12:00 - 02:00 in LEVH 101