Skip to main content
This assignment has multiple deliverables. The writeup is due before 11:59PM on Tuesday, October 29, 2024. The video is due before 11:59PM on Thursday, October 31, 2024.

Market Research : Assignment 2

This homework will look at the wealth of companies and platforms that use crowdsourcing as part of their business. Here’s a list of companies and platforms that you could research:

  1. Microtask and Freelance Platforms
  2. AI Data Creation and Training
    • Scale AI: AI training data platform
    • Appen (formerly Figure Eight): AI training data and content relevance services
    • Labelbox: Data labeling platform for AI
    • Hive: AI solutions and data labeling
    • Sama (formerly Samasource): Training data for AI
    • Snorkel AI: Data programming platform using weak supervision
    • CloudFactory: Data processing for AI and automation
    • Amazon SageMaker Ground Truth: Data labeling for machine learning
    • Dataloop: Combines human and machine intelligence for data annotation
    • Surge AI: Network of annotators for AI data labeling
    • Invisible AI: Human-in-the-loop processes for high-quality AI training data
  3. Prediction Markets and Forecasting
    • Metaculus: Community forecasting platform
    • Good Judgment Open: Crowdsourced forecasting of geopolitical events
    • Kalshi: A prediction market that operates under the approval of the Commodity Futures Trading Commission (CFTC)
    • PredictIt: Online marketplace for event predictions
    • Polymarket: Prediction market for politics, providing real-time odds for election results
  4. Citizen Science and Research
    • Zooniverse: Platform hosting various citizen science projects
    • Foldit: Online puzzle game about protein folding
    • BOINC: Volunteer computing and grid computing platform
    • EyeWire: Neuron mapping game
    • Apple ResearchKit: Platform for building health research apps
  5. Crowdsourced Content Creation and Curation
  6. Distributed Problem-Solving
    • InnoCentive: Platform for posting and solving challenges
    • Kaggle: Data science and machine learning competition platform
    • Topcoder: Competitive programming and design platform
  7. Crowdfunding and Financial Services
    • Kickstarter: Crowdfunding platform for creative projects
    • GoFundMe: Crowdfunding platform for various causes
    • Experiment: Crowdfunding platform for scientific research
    • Unbound: Crowdfunding publisher for authors
    • Kiva: Microlending platform
    • Robinhood: Commission-free stock trading platform
  8. Peer-to-Peer Services
    • Airbnb: Peer-to-peer lodging platform
    • Uber: Ride-sharing and delivery platform
    • Lyft: Ride-sharing platform
    • DoorDash: Food delivery platform
    • Instacart: Grocery delivery platform
    • Postmates: Food and goods delivery platform
  9. Crowdsourced Design and Creativity
    • 99designs: Platform connecting clients with graphic designers
    • Threadless: Creative community for art and design
    • iStockPhoto: Stock media marketplace
  10. Education and Online Learning
    • Coursera: Online education platform
    • edX: Online education platform
    • Duolingo: Language learning app with crowdsourced translations
  11. Review and Recommendation Platforms
  12. Mapping and Navigation

This assignment has two deliverables:

  1. Answering the Homework 2 questionnaire on Gradescope about the company or project that you are researching. This is due before 11:59PM on Tuesday, October 29, 2024. If you are working with a partner, only one of you needs to submit the questionnaire. Please save your survey answers in a file on your own computer, so that you can have a copy to use for your presentation.
  2. A presentation video about the company. Several of the best videos will be selected for in-class presentations. Teams whose videos are selected will receive extra credit. This is due before 11:59PM on Thursday, October 31, 2024. You can submit a link to your video on the Gradescope presentation form.

This assignment can be done in pairs.

Part 1: Researching a Company

First, please sign up for a company or project. Up to two teams can sign up for a company. If two teams have already signed up for a company that you’re interested in then please pick a different company. If there’s a company that you’d like to research that isn’t on the list, you are welcome to ask the instructor or the TA to ask if it is OK. Please post your request to Piazza.

Next, please read the academic paper Human Computation: A Survey and Taxonomy of a Growing Field. The paper introduces several dimensions that we’ll use to categorize the companies that you’ll be doing research – dimensions like motivation, quality control, aggregation, and so on. We will ask you to write a short summary of the paper.

You and your partner should research the company together, and then fill in your answers about it in the Homework 2 questionnaire on Gradescope. Your short answers to these questions on Tuesday, October 29, 2024. Remember to save your survey answers on your own computer, so that you can have a copy to use when you do your video profile.

Questionnaire

Below are the questions that you will be asked to answer about the company or project that you are profiling. Please upload a PDF with your answers to Gradescope.

  1. Please give a summary of the academic paper Human Computation: A Survey and Taxonomy of a Growing Field, and how it relates to this assignment.
  2. What company are you profiling?
  3. What online resources did you use in researching it? Include URLs.
  4. Did you use any other resources? Did you conduct any interviews, did you try the company’s service?
  5. Give a one sentence description of the company. Be sure to use the name of the company in your sentence.
  6. Give a URL for the company’s website
  7. Give a URL for the company’s logo
  8. When was the company started?
  9. Who were the founders?
  10. Does it have an interesting origin story?
  11. What kind of organization is it?
    • Publicly traded company
    • Privately held company
    • Non-profit organization
    • Other
  12. If the company type is none of the above, please specify.
  13. What service does the company provide?
  14. Does this update a previous service or business model, or is it completely new?
  15. If it updates something, what does it replace?
  16. What other new companies provide services that are similar to your company’s?
  17. What is an example of how someone uses this service? What sort of people use the service? If this is a service that you have used, then describe your experience.
  18. Compare the number of users to contributors.
    • More users than contributors
    • More contributors than users
    • Roughly equal numbers
  19. Who are the people who contribute services?
  20. How does the company incentivize them to contribute, or what motivates them to participate?
  21. Is this a service that was previously provided by experts / professionals?
  22. Are the contributors experts / professionals?
  23. How does the company generate revenue?
  24. How does the company ensure the quality of the services it provides?
  25. Is its service something that is typically regulated by the government?
  26. If so, what are the intents of the regulations and does your company meet those standards?
  27. Is a reputation system used by your company?
  28. If so, how does it work?
  29. If the service is provided by many contributors, how are are their contributions aggregated?
  30. Describe the workflow for how the service is advertised, and how the contributors contribute, and what the users get in the end.
  31. What is the scale of the services that your company provides, in terms of users?
  32. What is the scale of the services that your company provides, in terms of dollars?
  33. If your company were to scale up to 10 or 100 times its current size, how well do you think its business model would work? How well would the incentive scheme scale? How about the quality/aggregation model?
  34. How do you think AI will affect this company?
  35. Have there been any controversies about the company or the service that it provides?
  36. Is there anything else you’d like to say about the company?

Part 2: Creating a Video

You will prepare a short presentation in the form a 5 to 7 minute video presentation about your company. Your presentation should address the following questions:

  • What incentives does it offer to get people to participate?
  • How does it aggregate the information provided by the crowd?
  • What are the quality concerns, and how does the company do quality control?
  • How does the company benefit from user contributions?
  • Are there any controversies about the company?

Your presentation is due on Thursday, October 31, 2024. To turn in your video, please upload it to YouTube or Vimeo. Give us the link to your presentation or video using the Homework 2 presentation form. Make sure that your video is publicly viewable or that you give us a password to view it on the web form.

FAQ

  • What is the difference between Users and Contributors?

    Many of you seemed confused about what constitutes a user vs. a contributor. In many crowdsourcing companies, this line is blurry- or nonexistent! Many companies you looked at fit what can be thought of as a “data-mining” model (e.g. Yelp, Foursquare), in which the primary service being provided depends on using data and modeling observed patterns of behavior– e.g. to target ads or to recommend products. In these cases, the users are the contributors– everyone who participates provides data, and everyone uses everyone else’s data. This is an awesome crowdsourcing model because it is (ideally) self-sustaining and self-incentivizing. The more a person contributes (by providing more data about themselves), the better product they receive (e.g. better recommendations).

  • How does my company do Aggregation?

    Many of the companies we look at fit into the “match-making” or “marketplace” model (e.g. Uber, Airbnb, Etsy). This is a very common case in which the company is simply working to match supply with demand, where the suppliers (e.g. drivers, in the case of Uber) and the consumers (riders) are distributed all over, and may have trouble finding one another otherwise. Here, rating systems etc. are one piece of the platform that might require aggregating, but arguably the more interesting aggregation problem is how to match a supplier with a consumer. Is this through preference-based recommendation systems (as in the data-mining model discussed above), through location-based matching, through bidding/price? Making these matches is non-trival, but crucial to making the company function.

  • Does this update a previous service or business model, or is it completely new?

    For this question, more than a third of you said “completely new.” Try to think broadly when you are considering where these crowdsourcing companies fit into the economy as a whole. (Think Silicon Valley’s favorite buzzword: “disruption.”) Most of these services being provided are not completely new, but they are directly competing with a service that was traditionally supplied by a brick-and-mortar company in a more centralized manner. Uber competes with taxi companies, Airbnb with hotel chains, Coursera with higher education universities. This is part of what makes crowdsourcing so exciting!

  • Is getting venture capital funding is a business model?

    No. Smart investors don’t invest in a company who says their plan for generating revenue is to “get money from investors.” When in doubt about the company’s business plan, it is probably “ads”. :-)

  • How to make a group submission on Gradescope?

    One group member can submit answers just as you do in previous homeworks. Then at the submitted page where you could see all your answers and points distribution, there is an option on the top right for you to edit group member. You can add your partner there with his or her Gradescope Id inside this course. Both of you with share the same submission and get the same grading. The submission is required to be made by only one team member.

Grading Rubric

The written survey is worth approximately 10% of your overall grade in the course. Please answer the survey questions thoughtfully. Importantly, you must discuss aspects of the company to crowdsourcing, and the themes of the class like aggregation, quality control, machine learning, etc. Extra credit would worth approximately 5% of the overall grade in the course.