Skip to main content

The lecture schedule is subject to change as the term progresses.

Date Topic Readings
Wednesday, January 13, 2016 Intro Lecture
  • Relationship between crowdsourcing, human computation, collective intelligence, data mining, and sharing economy companies
  • Examples of crowdsourcing and human computation
Friday, January 15, 2016 Friday hands-on exercise (become a Crowd Worker)
Monday, January 18, 2016 No classes - Martin Luther King Day
Wednesday, January 20, 2016 Crowd Workers
  • Ethics of Crowdsourcing
  • Why the Mechanical Turk user interface sucks
  • External tools designed by workers
  • Design principles for a fairer workplace
Friday, January 22, 2016 Friday hands-on activity by Sierra Yit
  • HW2 Walkthrough
  • Vimeo Tutorial
  • IPython Installation
Monday, January 25, 2016 Machine Learning part 1
  • Examples of machine learning applications
  • Bayes Rule
  • Reasoning with Probabilities
Wednesday, January 27, 2016 Python bootcamp by Ellie Pavlick
  • A rare Wednesday hands-on, for credit in-class exercise
  • Bring your laptop
  • Be sure to install IPython Notebook before the class
Friday, January 29, 2016 Friday hands-on activity: Machine Learning - part 2
  • Labeled training data - what it is, where to get it, and how to put it in a format that a classifier will be able to use
  • Walkthrough of building a classifier in Python
  • We do the homework in front of your eyes
  • Experiment design - training / test splits, overfitting, baselines, evaluation
Monday, February 1, 2016 The Amazon Mechanical Turk crowdsourcing platform
  • Amazon Mechanical Turk from the Requester's perspective: Designing HITs, qualifications, pricing HITs, approving/rejecting
Wednesday, February 3, 2016 The Amazon Mechanical Turk crowdsourcing platform (continued)
  • Amazon Mechanical Turk from the Requester's perspective: Designing HITs, qualifications, pricing HITs, approving/rejecting
Friday, February 5, 2016 Friday hands-on activity by Kate Miller
  • HW5 Walkthrough
Monday, February 8, 2016 Taxonomy of crowdsourcing and human computation
  • Categorization system: motivation, quality control, aggregation, human skill, process flow
Wednesday, February 10, 2016 Programming concepts for human computation
  • People as function calls, decomposing complex tasks into simpler subtasks
  • Memoizataion of expensive function calls, "Crash and Re-Run"
  • Quicksort for kittens
Friday, February 12, 2016 Iterative and Parallel Processing
  • Experiments with TurKit
  • Iterative versus parallel processing
Monday, February 15, 2016 Firearm Violence: A Public Health Issue by Douglas Wiebe, Associate Professor of Epidemiology, University of Pennsylvania Perelman School of Medicine
  • Firearm injury as a public health issue
  • Importance of data in epidemiology
Wednesday, February 17, 2016 Public health and Twitter by Ellie Pavlick
  • Distilling Collective Intelligence from Twitter
Friday, February 19, 2016 Drop deadline
Friday, February 19, 2016 Friday hands-on activity
Monday, February 22, 2016 Inferring individual and county level traits from social media by Lyle Ungar
Wednesday, February 24, 2016 In-class activity: find a group
Friday, February 26, 2016 Distilling Collective Intelligence from Twitter
  • Topic Detection and Tracking
  • Approximate algorithms for scaling to large data sets
Monday, February 29, 2016 Quality Control part 1
  • Agreement-based methods
  • Embedded gold standard
Wednesday, March 2, 2016 Quality Control part 2
  • Economic incentives
Friday, March 4, 2016 Friday hands-on activity
Monday, March 7, 2016 No classes - Spring Break
Wednesday, March 9, 2016 No classes - Spring Break
Friday, March 11, 2016 No classes - Spring Break
Monday, March 14, 2016 Quality Control part 3
  • Reputation systems
Wednesday, March 16, 2016 Quality Control part 4
  • The EM algorithm
Friday, March 18, 2016 Friday hands-on activity
Monday, March 21, 2016 Crowdsourcing and human computer interaction (HCI) design
  • Next generation interfaces
  • Soylent word processor ("it's made of people")
Wednesday, March 23, 2016 Crowdsourcing and HCI part 2
  • E-mail Valet
  • Adrenaline
Friday, March 25, 2016 Withdraw deadline
Friday, March 25, 2016 Friday hands-on activity
Monday, March 28, 2016 Crowdsourcing Translation
  • The language demographics of Mechanical Turk
  • Using crowdsourcing to collect data to train SMT systems
Wednesday, March 30, 2016 The Wisdom of the Crowd Comes to Poetry by Al Filreis
Friday, April 1, 2016 Friday hands-on activity
Monday, April 4, 2016 Crowdsourcing for Automatic Speech Recognition
  • A high level overview of automatic speech recognition
  • Using crowdsourcing to train ASR systems
Wednesday, April 6, 2016 A/B Testing
  • Active versus passive crowdsourcing
  • Making money with statistics
  • Optimizing web sites using A/B testing
  • Are you part of a social experiment?
Friday, April 8, 2016 The Best Practices of the Best Requesters by Kristy Milland
Monday, April 11, 2016 Social Sciences and Human Subjects Research
  • Human subjects research
  • Social Science experiments
Wednesday, April 13, 2016 Expert Crowds
  • Finding people with obscure knowledge via ads
  • Creating and evaluating test questions
Friday, April 15, 2016 Crowdsourcing A Meeting of Minds: Designing the Future of Work by Michael Bernstein
Wednesday, April 20, 2016 What data can tell us about sex and relationships
Wednesday, April 20, 2016 How to make better forecasts by Lyle Ungar
  • What traits do good forecasters share?
  • How important is domain expertise?
  • Can training improve forecasts?
  • How should forecasts be elicited--surveys or prediction markets?
  • How should that wisdom be combined? What clever statistical aggregations work best?
Friday, April 22, 2016 Prediction Markets
Monday, April 25, 2016 Real-time Crowdsourcing by Walter S. Lasecki, Assistant Professor, University of Michigan
Wednesday, April 27, 2016 Course wrap-up
Thursday, May 5, 2016 Final Presentations will be held during the final exam slot on Thursday May 5th, 2016 from 9:00 to 11:00 (there are locations - LRSM Auditorium and Skirkanich Auditorium)