Skip to main content

Subject to change as the term progresses.

Date Topic Readings
Aug 27 Intro Lecture
  • Relationship between crowdsourcing, human computation, collective intelligence, data mining, social computing
  • Examples of crowdsourcing and human computation
Sep 01 Labor Day (no classes)
Sep 03 The Mechanical Turk crowdsourcing platform - part 1
  • Working on Mechanical Turk, demographics of Mechanical Turk workers, estimating the size of the Mechanical Turk Marketplace
  • Terminology and mechanics: Turkers, Requesters, HITs, micropayments
Sep 08 The Mechanical Turk crowdsourcing platform - part 2
  • Amazon Mechanical Turk from the Requester's perspective: Designing HITs, qualifications, pricing HITs, approving/rejecting
Sep 10 Taxonomy of crowdsourcing and human computation
  • Categorization system: motivation, quality control, aggregation, human skill, process flow
Sep 15 Machine Learning part 1
  • Examples of machine learning applications
  • Bayes Rule
  • Reasoning with Probabilities
Sep 17 Machine Learning - part 2
  • Labeled training data - what it is, where to get it, and how to put it in a format that a classifier will be able to use
  • Walkthrough of building a classifier in Python
  • We do the homework in front of your eyes
  • Experiment design - training / test splits, overfitting, baselines, evaluation
Sep 22 Programming concepts for human computation
  • People as function calls, decomposing complex tasks into simpler subtasks
  • Memoizataion of expensive function calls, "Crash and Re-Run"
  • Quicksort for kittens
Sep 24 Iterative and Parallel Processing
  • Experiments with TurKit
  • Iterative versus parallel processing
Sep 29 Firearm Violence: A Public Health Issue by Douglas Wiebe, Associate Professor of Epidemiology, University of Pennsylvania Perelman School of Medicine
  • Firearm injury as a public health issue
  • Importance of data in epidemiology
Oct 01 Crowdsourcing and human computer interaction (HCI) design by Ellie Pavlick (Instructor will be traveling)
  • Next generation interfaces
  • Soylent word processor ("it's made of people")
  • The Wisdom of Crowds: Chapter 6 ("Society Does Exist: Taxes, Tipping, Television and Trust")
  • Soylent: A Word Processor with a Crowd Inside by Michael Bernstein, Greg Little, Rob Miller, Björn Hartmann, Mark Ackerman, David Karger, David Crowell, and Katrina Panovich
Oct 06 Crowdsourcing and HCI part 2
  • E-mail Valet
  • Adrenaline
Sep 29 Quality Control part 1
  • Agreement-based methods
  • Embedded gold standard
Oct 01 Quality Control part 2
  • Reputation systems
  • Economic incentives
Oct 06 Quality Control part 3
  • The EM algorithm
Oct 08 In-class presentations: Pitch your term project ideas
Oct 13 In-class presentations: Pitch your term project ideas
Oct 15 No class today. Instructor and TA will be traveling for a DARPA meeting.
Oct 20 Machine Learning part 2
  • A high level perspective on how machine learning works and why it can fail
Oct 22 Machine Learning part 3
  • Integrating machine learning and crowdsourcing
Oct 27 Distilling Collective Intelligence from Twitter part 1
  • Topic Detection and Tracking
  • Approximate algorithms for scaling to large data sets
Oct 29 Distilling Collective Intelligence from Twitter part 2 by Michael Paul
  • Public health and Twitter
Nov 05 Detecting Emotional Contagion in Massive Social Networks
Nov 10 A/B Testing
  • Active versus passive crowdsourcing
  • Optimizing web sites using A/B testing
Nov 12 Prediction Markets by Ellie Pavlick
Nov 17 Crowdsourcing Translation
  • The language demographics of Mechanical Turk
  • Using crowdsourcing to collect data to train SMT systems
Nov 19 Crowdsourcing for Automatic Speech Recognition by Scott Novotney
  • A high level overview of automatic speech recognition
  • Using crowdsourcing to train ASR systems
Nov 24 What big data can tell us about sex and relationships
Nov 26 No class (Friday class schedule today)
Dec 01 Review of machine learning homework by Ellie Pavlick
Dec 03 Personality, Gender and Age in the Language of Social Media by Andy Schwartz
Dec 08 Course wrap-up
Dec 15 Final Presentations from 3pm-5pm in Berger Auditorium - room 101 in Skirkanich Hall