Skip to main content

Subject to change as the term progresses.

Date Topic Readings
Aug 27 Intro Lecture
  • Relationship between crowdsourcing, human computation, collective intelligence, data mining, social computing
  • Examples of crowdsourcing and human computation
Sep 01 Labor Day (no classes)
Sep 03 The Mechanical Turk crowdsourcing platform - part 1
  • Working on Mechanical Turk, demographics of Mechanical Turk workers, estimating the size of the Mechanical Turk Marketplace
  • Terminology and mechanics: Turkers, Requesters, HITs, micropayments
Sep 08 The Mechanical Turk crowdsourcing platform - part 2
  • Amazon Mechanical Turk from the Requester's perspective: Designing HITs, qualifications, pricing HITs, approving/rejecting
Sep 10 Taxonomy of crowdsourcing and human computation
  • Categorization system: motivation, quality control, aggregation, human skill, process flow
Sep 15 Machine Learning part 1
  • Examples of machine learning applications
  • Bayes Rule
  • Reasoning with Probabilities
Sep 17 Machine Learning - part 2
  • Labeled training data - what it is, where to get it, and how to put it in a format that a classifier will be able to use
  • Walkthrough of building a classifier in Python
  • We do the homework in front of your eyes
  • Experiment design - training / test splits, overfitting, baselines, evaluation
Sep 22 Programming concepts for human computation
  • People as function calls, decomposing complex tasks into simpler subtasks
  • Memoizataion of expensive function calls, "Crash and Re-Run"
  • Quicksort for kittens
Sep 24 Iterative and Parallel Processing
  • Experiments with TurKit
  • Iterative versus parallel processing
Sep 29 Firearm Violence: A Public Health Issue by Douglas Wiebe, Associate Professor of Epidemiology, University of Pennsylvania Perelman School of Medicine
  • Firearm injury as a public health issue
  • Importance of data in epidemiology
Oct 01 Crowdsourcing and human computer interaction (HCI) design by Ellie Pavlick (Instructor will be traveling)
  • Next generation interfaces
  • Soylent word processor ("it's made of people")
  • The Wisdom of Crowds: Chapter 6 ("Society Does Exist: Taxes, Tipping, Television and Trust")
  • Soylent: A Word Processor with a Crowd Inside by Michael Bernstein, Greg Little, Rob Miller, Björn Hartmann, Mark Ackerman, David Karger, David Crowell, and Katrina Panovich
Oct 06 Crowdsourcing and HCI part 2
  • E-mail Valet
  • Adrenaline
Oct 08 In-class presentations: Pitch your term project ideas
Oct 13 In-class presentations: Pitch your term project ideas
Oct 15 No class today. Instructor and TA will be traveling for a DARPA meeting.
Oct 20 Quality Control part 1
  • Agreement-based methods
  • Embedded gold standard
Oct 22 Quality Control part 2
  • Economic incentives
  • Reputation systems
Oct 27 Quality Control part 3
  • More on reputation systems
  • The EM algorithm
Oct 29 Decision-Theoretic Crowdsourcing by Mausam
Nov 03 Prediction Markets by Ellie Pavlick
Nov 05 Distilling Collective Intelligence from Twitter part 1
  • Topic Detection and Tracking
  • Approximate algorithms for scaling to large data sets
Nov 10 Distilling Collective Intelligence from Twitter part 2
  • Public health and Twitter
Nov 12 Crowdsourcing Translation
  • The language demographics of Mechanical Turk
  • Using crowdsourcing to collect data to train SMT systems
Nov 17 Crowdsourcing for Automatic Speech Recognition
  • A high level overview of automatic speech recognition
  • Using crowdsourcing to train ASR systems
Nov 19 Social Sciences and Human Subjects Research
  • Human subjects research
  • Social Science experiments
Nov 26 No class (Friday class schedule today)
Dec 01 A/B Testing
  • Active versus passive crowdsourcing
  • Making money with statistics
  • Optimizing web sites using A/B testing
  • Are you part of a social experiment?
Dec 03 Expert Crowds
  • Finding people with obscure knowledge via ads
  • Creating and evaluating test questions
Dec 08 Course wrap-up
Dec 15 Final Presentations from 3pm-5pm in Berger Auditorium - room 101 in Skirkanich Hall