The lecture schedule will be updated as the term progresses.
| Date | Topic | Recommended Readings |
|---|---|---|
| Thu, Jan 21, 2021 | Intro Lecture [slides] [video] |
James Surowiecki,
The Wisdom of Crowds: Introduction
UW iSchool's "Calling Bullshit" course, Example of fake news International Labour Oranization, Digital Labour Platforms and the Future of Work (Chapter 1 and 2) |
| Tue, Jan 26, 2021 | Crowd Workers [slides] [video] |
Jeff Bigham,
My MTurk (half) Workday
Six Silberman and Hannah Johnston, Crowdsourcing Wage Pledge Lilly Irani and Six Silberman, Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk Chris Callison-Burch, Crowd-Workers: Aggregating Information Across Turkers To Help Them Find Higher Paying Work Ben Bederson and Alex Quinn, Web Workers, Unite! Addressing Challenges of Online Laborers David Martin, Benjamin Hanrahan, Jacki O’Neill and Neha Gupta, Being A Turker Saiph Savage, Chun Chiang, Susumu Saito, Carlos Toxtli, and Jeffrey Bigham, Becoming the Super Turker: Increasing Wages via a Strategy from High Earning Workers Neha Gupta, David Martin, Benjamin Hanrahan, and Jacki O’Neill, Turk-Life in India Karen Fort, Gilles Adda and Bretonnel Cohen, Amazon Mechanical Turk: Gold Mine or Coal Mine? |
| Thu, Jan 28, 2021 | The Amazon Mechanical Turk crowdsourcing platform [slides] [video] |
Jeff Barr and Luis Felipe Cabrera,
AI Gets a Brain
Gabriele Paolacci, Jesse Chandler and Panos Ipeirotis, Running experiments on Amazon Mechanical Turk Amazon Mechanical Turk Mechanical Turk blog Chris Callison-Burch, Fast, Cheap and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk |
| Tue, Feb 2, 2021 | Course Selection Period ends | |
| Tue, Feb 2, 2021 | Ethnographic Studies of Turkers [slides] [video] by Benjamin V. Hanrahan |
David Martin, Benjamin Hanrahan, Jacki O’Neill and Neha Gupta,
Being A Turker
Neha Gupta, David Martin, Benjamin Hanrahan, and Jacki O’Neill, Turk-Life in India |
| Thu, Feb 4, 2021 | A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk [slides] [video] |
Kotaro Hara, Abi Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch and Jeff Bigham,
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Benjamin V. Hanrahan, Jeffrey P. Bigham and Chris Callison-Burch, Worker Demographics and Earnings on Amazon Mechanical Turk: An Exploratory Analysis. Abi Adams-Prassl, Chris Callison-Burch, Kotaro Hara, Kristy Milland, Saiph Savage and Jeffrey Bigham, (In Submission) The Gender Wage Gap in an Online Labour Market: The Cost of Interruptions |
| Tue, Feb 9, 2021 | Datasaur.ai - Data Labeling Startup [slides] [video] by Ivan Lee | |
| Tue, Feb 9, 2021 | An Analytic Framework for Human Computation [slides] [video] |
Alex Quinn and Ben Bederson,
Human Computation: A Survey and Taxonomy of a Growing Field
Panos Ipeirotis, Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers? |
| Thu, Feb 11, 2021 | Citizen Science [slides] [video] by Christopher Cieri |
James Fiumara, Christopher Cieri, Jonathan Wright, Mark Liberman,
LanguageARC: Developing Language Resources Through Citizen Linguistics
Christopher Cieri, James Fiumara, Mark Liberman, Chris Callison-Burch, Jonathan Wright, Introducing NIEUW: Novel Incentives and Workflows for Eliciting Linguistic Data Christopher Cieri, James Fiumara, LanguageARC – a tutorial |
| Thu, Feb 11, 2021 | Games with a Purpose [slides] [video] by Karën Fort |
John Chamberlain, Karën Fort, Ugo Kruschwitz, Mathieu Lafourcade, Massimo Poesio,
Using Games to Create Language Resources: Successes and Limitations of the Approach
Bruno Guillaume, Karën Fort, Nicolas Lefèbvre, Crowdsourcing Complex Language Resources: Playing to Annotate Dependency Syntax Karën Fort, Bruno Guillaume, Yann-Alan Pilatte, Mathieu Constant, Nicolas Lefèbvre, Rigor Mortis: Annotating MWEs with a Gamified Platform Alice Millour, Marianne Grace Araneta, Ivana Lazić Konjik, Annalisa Raffone, Yann-Alan Pilatte, Karën Fort, pKatana and Grand Guru: a Game of the Lost Words (DEMO) |
| Tue, Feb 16, 2021 | Python Review [slides] [video] | |
| Thu, Feb 18, 2021 | Python Review - part 2 [slides] [video] | |
| Fri, Feb 19, 2021 | Grade Type Change Deadline | |
| Tue, Feb 23, 2021 | Intro to Machine Learning [slides] [video] |
Jeremy Howard / fast.ai,
Practical Deep Learning for Coders
Daniel Jurafsky and James Martin, Logistic Regression |
| Thu, Feb 25, 2021 | How Crowdsourcing Enabled Computer Vision [slides] [video] |
Dave Gershgorn - Quartz,
The data that transformed AI research – and possibly the world
Fei-Fei Li, slides about ImageNet: Crowdsourcing, benchmarking & other cool things Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei, ImageNet: A Large-Scale Hierarchical Image Database Stanford University instructors, notes for Stanford's CS231n course "Convolutional Neural Networks for Visual Recognition" |
| Tue, Mar 2, 2021 | Crowdsourcing Translation [slides] [video] |
Omar Zaidan and Chris Callison-Burch,
Crowdsourcing Translation: Professional Quality from Non-Professionals
Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch, The Language Demographics of Amazon Mechanical Turk Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch, Machine Translation of Arabic Dialects Chris Callison-Burch, Fast, Cheap and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk |
| Thu, Mar 4, 2021 | Linguistic Wisdom From the Crowd [video] by Michael Tseng |
Nancy Chang, Russell Lee-Goldman, and Michael Tseng,
Linguistic Wisdom from the Crowd
Christina Funk, Michael Tseng, Ravindran Rajakumar, and Linne Ha, Community-Driven Crowdsourcing: Data Collection with Local Developers Jennimaria Palomaki, Olivia Rhinehart, and Michael Tseng, A Case for a Range of Acceptable Annotations Clark et al., TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages Kwiatkowski et al., Natural Questions: A Benchmark for Question Answering Research Min et al., AmbigQA: Answering Ambiguous Open-domain Questions Parikh et al., ToTTo: A Controlled Table-To-Text Generation Dataset Zhang et al., PAWS: Paraphrase Adversaries from Word Scrambling |
| Tue, Mar 9, 2021 | Quality Control part 1 [slides] [video] |
Luis von Ahn and Laura Dabbish,
Labeling Images with a Computer Game
Rion Snow, Brendan O’Connor, Daniel Jurafsky and Andrew Ng, Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks Winter Mason and Duncan Watts, Financial Incentives and the Performance of Crowds |
| Thu, Mar 11, 2021 | No class (Spring Break) | |
| Tue, Mar 16, 2021 | Quality Control part 2 [slides] [video] |
Anindya Ghose, Panos Ipeirotis, and Arun Sundararajan,
Opinion Mining Using Econometrics: A Case Study on Reputation Systems
Kazushi Ikeda and Michael Bernstein, Pay It Backward: Per-Task Payments on Crowdsourcing Platforms Reduce Productivity |
| Thu, Mar 18, 2021 | Quality Control part 3 [slides] [video] |
A.P. Dawid and A.M. Skene,
Maximum Likelihood Estimation of Observer Error-rates using the EM Algorithm
Panos Ipeirotis, Foster Provost, Jing Wang, Quality Management on Amazon Mechanical Turk |
| Tue, Mar 23, 2021 | Crowdsourcing and human computer interaction (HCI) design [slides] [video] |
Michael Bernstein, Greg Little, Rob Miller, Björn Hartmann, Mark Ackerman, David Karger, David Crowell, and Katrina Panovich,
Soylent: A Word Processor with a Crowd Inside
Jeff Bigham, Real-Time Crowd Support for People with Disabilities (video) Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Vizwiz: Nearly Real-time Answers to Visual Questions |
| Thu, Mar 25, 2021 | Crowdsourcing and HCI part 2 [slides] [video] |
Michael Bernstein, Joel Brandt, Rob Miller, and David Karger,
Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces
Nicolas Kokkalis, Thomas Köhn, Carl Pfeiffer, Dima Chornyi, Michael Bernstein, and Scott Klemmer, EmailValet: Managing Email Overload through Private, Accountable Crowdsourcing |
| Mon, Mar 29, 2021 | Last day to withdraw from a course | |
| Tue, Mar 30, 2021 | No class (Engagement Day) | |
| Thu, Apr 1, 2021 | Programming concepts for human computation [slides] [video] |
Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller,
TurKit: Human Computation Algorithms on Mechanical Turk
|
| Tue, Apr 6, 2021 | Toward Future Crowd-AI Systems [video] by Ting-Hao (Kenneth) Huang |
Ting-Hao K. Huang, Joseph Chee Chang, Jeffrey P. Bigham,
Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
Gimlet Media, Sandra (a sci-fi thriller podcast about crowd-powered conversational agents) Chieh-Yang Huang, Shih-Hong Huang, Ting-Hao 'Kenneth' Huang, Heteroglossia: In-Situ Story Ideation with the Crowd Ting-Hao 'Kenneth' Huang, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Yen-Chia Hsu, C. Lee Giles, CODA-19: Using a Non-Expert Crowd to Annotate Research Aspects on 10,000+ Abstracts in the COVID-19 Open Research Dataset |
| Thu, Apr 8, 2021 | Iterative and Parallel Processing [slides] [video] |
Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller,
Exploring Iterative and Parallel Human Computation Processes
|
| Tue, Apr 13, 2021 | Minimum Wage Pledge for Requesters by Six Silberman and Hannah Johnston |
WeAreDynamo,
Guidelines for Academic Requesters
Crowdsourcing Code of Conduct Oxford Internet Institute, Fairwork Principles Amy Glasmeier, MIT Living Wage Calculator |
| Thu, Apr 15, 2021 | Expert Crowds [slides] [video] |
Panos Ipeirotis and Evgeniy Gabrilovich,
Quizz: Targeted Crowdsourcing with a Billion (Potential) Users
Maria Christoforaki and Panos Ipeirotis, STEP: A Scalable Testing and Evaluation Platform Wikipedia, Item response theory |
| Tue, Apr 20, 2021 | What data can tell us about sex and relationships [slides] |
Rajesh Ranganath, Dan Jurafsky, and Dan McFarland,
Detecting Flirting and its Misperception in Speed-Dates
Christian Rudder, Dataclysm: Who We Are (When We Think No One's Looking) Christian Rudder - OKCupid, OKCupid's blog |
| Thu, Apr 22, 2021 | Prediction Markets [slides] |
Prediction Markets,
Justin Wolfers and Eric Zitzewitz
Using Prediction Markets to Track Information Flows: Evidence from Google, Bo Cowgill, Justin Wolfers and Eric Zitzewitz |
| Tue, Apr 27, 2021 | Extracting More Wisdom from the Crowd by John McCoy |
Dražen Prelec, H. Sebastian Seung, and John McCoy,
A Solution to the Single-Question Crowd Wisdom Problem
|
| Thu, Apr 29, 2021 | Understanding Human Happiness by Matt Killingsworth |
Matt Killingsworth,
trackyourhappiness.org
|