Skip to main content
This assignment has multiple deliverables. The written survey is due before 11:59PM on Tuesday, February 5, 2019. The presentation is due before 11:59PM on Tuesday, February 12, 2019.

Market Research : Assignment 2

This week we will be looking at the wealth of companies and projects that use crowdsourcing as part of their business. Here’s a list of companies and projects that you could research:

5 Calls, 99Designs, Airbnb, Apple HealthKit/ResearchKit, Benefunder,, CivilServant, Clickworker, CloudFactory, Couchsurfing, Coursera,, CrowdMed, Crowds On Demand, Daemo, Dataturks, Defined Crowd, Duolingo, EatWith, eBay buyer/seller ratings, edX, EyeWire, Facewatch, Field Agent, Figure Eight, Flattr, Freelancer, Genius, GoFundMe, Good Judgment Open, Google Crowdsource app, Iceland’s Crowdsourced Constitution, The iLabour Project, IndieGoGo, Indivisible, Innocentive, Instacart, Invisible Boyfriend, iStockPhoto, Intrade, Iowa Election Markets, Jovoto, Kaggle, Kickstarter, Kiva, leadGenius, Lending Club, Lyft, Microworkers, Mighty AI, Netflix prize, New York Times, OpenStreetMap, Orchestra, PatientsLikeMe, Patreon, Polis, Postmates, PredictIt, PredictWise, Premise, Prolific, PublicStuff, Quora, Qualtrics, Samasource, Scale, Scistarter, Sunshine, TaskRabbit, Threadless, Thumbtack, Topcoder, TurkOpticon, TurkPrime, TurkServer, Uber, Upwork, Ushahidi, vTaiwan, Waze, We the People, Wikipedia’s AI helper, XPRIZE, Yelp, Zensors, Zooniverse.

This assignment has two deliverables:

  1. Answering the Homework 2 questionnaire on Gradescope about the company or project that you are researching. This is due before 11:59PM on Tuesday, February 5, 2019. If you are working with a partner, only one of you needs to submit the questionnaire. Please save your survey answers in a file on your own computer, so that you can have a copy to use for your presentation.
  2. A presentation, optionally a video, about the company. If you’re not submitting a vidoe, Google Slides (or equivalent slide deck) are required. Several of the best videos will be selected for in-class presentations. Teams whose videos are selected will receive extra credit. This is due before 11:59PM on Tuesday, February 12, 2019. You can submit a link to your video/slides on the Gradescope presentation form.

This assignment should be done in pairs.

First, please sign up for a company or project. Please do not pick a company that another team has already signed up for. If there’s a company that you’d like to research that isn’t on the list, you are welcome to ask the instructor or the TA to ask if it is OK.

You should independently research the company, and then fill in your answers about it in the Homework 2 questionnaire on Gradescope. Your short answers to these questions on Tuesday, February 5, 2019. Remember to save your survey answers on your own computer, so that you can have a copy to use when you do your video profile.

You will prepare a short presentation, optionally a 5 to 7 minute video presentation, about your company. Your presentation should address the following questions:

  • What incentives does it offer to get people to participate?
  • How does it aggregate the information provided by the crowd?
  • What are the quality concerns, and how does the company do quality control?
  • How does the company benefit from user contributions?
  • Are there any controversies about the company?

Your presentation is due on Tuesday, February 12, 2019. To turn in your video, please upload it to Vimeo. Give us the link to your presentation or video using the Homework 2 presentation form. Make sure that your video is publicly viewable or that you give us a password to view it on the web form.

If you have managed to make it this far in life without having to sign up for accounts for things on the internet, here are more detailed instructions:

Instructions for uploading your video to Vimeo

  1. Go to Vimeo.
  2. Create an account by clicking the enormous blue “JOIN” button.
  3. You will receive and email with a link to verify your account. You have to verify before you can upload videos.
  4. Once you are signed in, click “Upload” at the top of the page.
  5. Click the “Choose a Video to Upload” button and choose your video
  6. Once it is uploading, you can change the privacy settings. If you are soon to be on the job market, be careful. You probably don’t want potential employers to know how intelligently and elegantly you are able to analyze and present on the potential market value of technology companies, so maybe don’t use your real name.
  7. That’s it! Fill in the title and tags and what-have-you in the survey to tell us where to find it.


  • What is the difference between Users and Contributors?

    Many of you seemed confused about what constitutes a user vs. a contributor. In many crowdsourcing companies, this line is blurry- or nonexistent! Many companies you looked at fit what can be thought of as a “data-mining” model (e.g. Yelp, Foursquare), in which the primary service being provided depends on using data and modeling observed patterns of behavior– e.g. to target ads or to recommend products. In these cases, the users are the contributors– everyone who participates provides data, and everyone uses everyone else’s data. This is an awesome crowdsourcing model because it is (ideally) self-sustaining and self-incentivizing. The more a person contributes (by providing more data about themselves), the better product they receive (e.g. better recommendations).

  • How does my company do Aggregation?

    Many of the companies we look at fit into the “match-making” or “marketplace” model (e.g. Uber, Airbnb, Etsy). This is a very common case in which the company is simply working to match supply with demand, where the suppliers (e.g. drivers, in the case of Uber) and the consumers (riders) are distributed all over, and may have trouble finding one another otherwise. Here, rating systems etc. are one piece of the platform that might require aggregating, but arguably the more interesting aggregation problem is how to match a supplier with a consumer. Is this through preference-based recommendation systems (as in the data-mining model discussed above), through location-based matching, through bidding/price? Making these matches is non-trival, but crucial to making the company function.

  • Does this update a previous service or business model, or is it completely new?

    For this question, more than a third of you said “completely new.” Try to think broadly when you are considering where these crowdsourcing companies fit into the economy as a whole. (Think Silicon Valley’s favorite buzzword: “disruption.”) Most of these services being provided are not completely new, but they are directly competing with a service that was traditionally supplied by a brick-and-mortar company in a more centralized manner. Uber competes with taxi companies, Airbnb with hotel chains, Coursera with higher education universities. This is part of what makes crowdsourcing so exciting!

  • Is getting venture capital funding is a business model?

    No. Smart investors don’t invest in a company who says their plan for generating revenue is to “get money from investors.” When in doubt about the company’s business plan, it is probably “ads”. :-)

  • How to make a group submission on Gradescope?

    One group member can submit answers just as you do in previous homeworks. Then at the submitted page where you could see all your answers and points distribution, there is an option on the top right for you to edit group member. You can add your partner there with his or her Gradescope Id inside this course. Both of you with share the same submission and get the same grading. The submission is required to be made by only one team member.

Survey Questions

Below are the questions that you will be asked to answer about the company or project that you are profiling. Please turn in your answers using the Homework 2 survey on Gradescope.

We recommend that you save your survey answers in a file on your own computer, typing them directly into Gradescope, so that you’ll have a copy after you press the submit button.

  1. What company are you profiling?
  2. If you choose a company that is not in the list above, please specify the company name.
  3. What online resources did you use in researching it? List URLs, one per line.
  4. Did you use any other resources? Did you conduct any interviews, did you try the company’s service?
  5. Give a one sentence description of the company. Be sure to use the name of the company in your sentence.
  6. Give a URL for the company’s website
  7. Give a URL for the company’s logo
  8. When was the company started?
  9. Who were the founders?
  10. Does it have an interesting origin story?
  11. What kind of organization is it?
    • Publicly traded company
    • Privately held company
    • Non-profit organization
    • Other
  12. If the company type is none of the above, please specify.
  13. What service does the company provide?
  14. Does this update a previous service or business model, or is it completely new?
  15. If it updates something, what does it replace?
  16. What other new companies provide services that are similar to your company’s?
  17. What is an example of how someone uses this service? What sort of people use the service? If this is a service that you have used, then describe your experience.
  18. Compare the number of users to contributors.
    • More users than contributors
    • More contributors than users
    • Roughly equal numbers
  19. Who are the people who contribute services?
  20. How does the company incentivize them to contribute, or what motivates them to participate?
  21. Is this a service that was previously provided by experts / professionals?
  22. Are the contributors experts / professionals?
  23. How does the company generate revenue?
  24. How does the company ensure the quality of the services it provides?
  25. Is its service something that is typically regulated by the government?
  26. If so, what are the intents of the regulations and does your company meet those standards?
  27. Is a reputation system used by your company?
  28. If so, how does it work?
  29. If the service is provided by many contributors, how are are their contributions aggregated?
  30. Describe the workflow for how the service is advertised, and how the contributors contribute, and what the users get in the end.
  31. What is the scale of the services that your company provides, in terms of users?
  32. What is the scale of the services that your company provides, in terms of dollars?
  33. If your company were to scale up to 10 or 100 times its current size, how well do you think its business model would work? How well would the incentive scheme scale? How about the quality/aggregation model?
  34. Have there been any controversies about the company or the service that it provides?
  35. Is there anything else you’d like to say about the company?

Grading Rubric

The written survey is worth approximately 10% of your overall grade in the course. Please answer the survey questions thoughtfully. Extra credit would worth approximately 5% of the overall grade in the course.

Examples profiles from last year

Who were the founders? Peter Weijmarshausen, Robert Schouwenburg and Marleen Vogelaar
When was it started? Company was started in 2007-2008 in the Netherlands
Does it have any interesting origin story? It started as a spinoff from Royal Philips Electronics in the Netherlands in 2007.
What services does Shapeways provide? Shapeways provides a 3D printing service in many materials and finishes. Anyone can upload a 3D model to the website and someone may order to print that object. Users can also sell those designs and find designers to collaborate with to create products. Shapeways will print those objects for a fee on top of the price they set. Recently they have partnered with businesses offering 3D scanning services in order to make 3D scanning technology accessible to their customers. They also offer tutorials and tools to help customers design 3D models.
Who uses the services? A Jewelry designer who uploads my models on Shapeways' marketplace. People can buy his/her designs in different printed materials and he/she can charge any price for a design. For each sale he/she makes his/her listed price minus the quoted print cost from Shapeways.

Designers and entrepreneurs design and sell products while anyone interested in non mass-manufactured products buy from the marketplace. These tend to be products that are not yet offered by any online retailer and solve a very specific issue for the user.

Who are the people who contribute the services? Designers or entrepreneurs who want to sell a few of their designs or create their own shop online. Shapeways helps them test those and check if there is demand for them.

How does Shapeways incentivize them to contribute, or what is their motivation? Creators are incentivised by getting paid when others purchase their printed designs.

Is this a service that was previously provided by experts? Yes.
Are the contributors experts/professionals? No.
How does Shapeways ensure the quality of the services it provides? The 3D printing process they use is very thorough. They use expensive and modern machines to print those objects and the people they hire ensure that the quality of the products is exceptional.

Compare the number of users to contributors: More users than contributors
If its service is provided by many contributors, how are their contributions aggregated? The marketplace is sorted conveniently into categories such as Gadgets, Accessories, Jewlery etc and contributors set up shops which group their items in. As a user, you can browse those shops for goods from the same designer.

Describe the workflow for how the service is advertised, how the contributors contribute, and what the users get in the end: (1) Designer uploads a 3D model to the website. (2) Random user is interested and buys the product (3) Shapeways prints the object in one of their factories (4) Shapeways ships the product to the buyer (5) Designer and Shapeways gets some money and the user all the fun!

What is the scale of Shapeways in terms of users? In 2015, the company had 500,000 "shapies", which are the contributors of 3D model designs.
What is the scale of Shapeways in terms of dollars? Since it's privately held, there is no data on revenues/profits. But the company has received $76.3 million in funding since it started in 2008.
If Shapeways were to scale up to 10-100 times its current size, how well do you think would its business model would work? If Shapeways were to scale up by a significant percentage it would have to expand its 3D printing facilities accordingly, as well as labor. It could potentially move printing to China or India where costs are lower.

The incentive scheme would still work, because it would mean that there would be more buyers and, therefore, more money for the sellers. Quality may decrease, but they could potentially offer different printing qualities.

What kind of organization is it? Privately held company
How does Shapeways generate revenue? Shapeways generates revenue through production fees from items that they print.

Who were the founders? Leigh Drogen
When was it started? 2011
Does it have any interesting origin story? Leigh Drogen was previously an analyst at a quantitative hedge fund that used many strategies around earnings events. Every quarter, Leigh would see the same thing - companies would beat their earnings estimates given by Wall Street almost 100% of the time. Upon further inspection, he saw that Wall Street analysts are incentivized to low ball their estimates for companies due to the relationships that they have, and he saw the opportunity to make the estimates a more fair representation of what will truly happen. He quit his job at the hedge fund and started Estimize, with the drive to make estimates more democratic and transparent for everyone.
What services does Estimize provide? Estimize provides multiple services through earnings estimates. The first is the sheer raw data of estimates. Estimize collects estimates for earnings per share (EPS) and revenue for nearly every US Exchange listed company. The raw data that the company collects comes from individuals who sign up for the company's website. These individuals can be professionals, students, independent investors, research analysts, etc. Then, Estimize generates a consensus, which is an average of every valid estimate, into one singular number. To sell the data, Estimize provides 3 spreadsheets- the first includes a list of every user on estimize along with their biographical data, the second includes every estimate that has been made on the website, and the third includes every release, or every earnings event that has occured, with both the estimize consensus for EPS and revenue, as well as the Wall Street estimates.

The second main service that Estimize provides is a stock screener. On this screener, individuals are able to screen US listed stocks for specific analytics in their earnings estimates. They are able to search for analytics such as % of the time Estimize is more accurate than the Wall Street consensus, or % of the time company beats Estimize consensus. This stock screener is unlike any other stock screener already available, because no other data sets exist like Estimize's, so it is novel.

If this updates a previous service or business model, what does it replace? The previous model Estimize replaces is Thomson Reuters I/B/E/S estimates. These estimates typically come from 20-30 of the same research analysts at investment banks that make estimates on the companies time and time again. The Estimize data set draws from a broader representation of the population, by polling individuals who are both directly involved in financial markets and those who are not.

Who uses the services? Hedge funds that use quantitative strategies to trade will use the Estimize data. The Estimize data contains more accurate representations of what the market is expecting, so hedge funds will be able to make more accurate trades and generate larger alpha than they will with data from Thomson Reuters. Whether this results in them engaging in post-earnings trading strategies, pre-earnings strategies, or even holding through earnings announcements, the data has been academically proven to produce alpha and create returns.
Who are the people who contribute the services? The people who contribute are any individuals who sign up for the website to submit estimates. These people can range from hedge fund managers, to research analysts at investments, to students, to academics, to even professionals that are not in the field in finance. Anyone is eligible to sign up and contribute, and the biographies of those who contribute is very diverse.
How does Estimize incentivize them to contribute, or what is their motivation? The data set is blind to any individual who does not contribute. That is, if you do not contribute, you will not be able to see the estimates for any given stock. The company also incentivizes individuals to contribute by giving an assessment of the individual's history, and providing a track record for individual analysts to show to employers, clients, etc. Finally, there is a leagues feature (think Fantasy football, except for estimates), in which individuals are able to win prizes (super bowl tickets, gift cards, etc.) if they are the most accurate in the league.

Despite all of these incentives, when polled, the contributors cited wanting to build a more accurate data set as the
Is this a service that was previously provided by experts? Yes.
Are the contributors experts/professionals? Yes.
How does Estimize ensure the quality of the services it provides? Estimize runs many algorithms on the front end of its website in order to ensure the quality of its data. The first is a general algorithm that compares any individual's estimate to the Estimize consensus for that estimate. If any person's estimate is plus or minus 2 standard deviations from the estimate, that person's estimate will automatically be flagged and removed from the system.

Secondly, Estimize runs many more complex algorithms to further ensure the quality of its data. Estimize runs algorithms that detect various details, such as the amount of time spent on the website before making the estimate, how accurate the user has been on the past, what other parts of Estimize has the user recently clicked to before submitting this estimate, did the user only change the default estimate by a little before submitting, etc. All of this results in a simple boolean, true or false, as to whether the estimate was flagged. An analysis of their data shows that nearly 1 in 10,000 estimates needs to be flagged, so users typically do not submit false estimates. Of those that are reported, nearly 999/1000 are due to mechanical errors by humans - someone submits $15 instead of $1.50 for earnings per share. Barely any estimates are truly submitted with the intent of gaming the system.
If Estimize uses a reputation system, how does it work? Users are rated on the accuracy of their estimate for every estimate that they submit. They get a certain amount of either positive or negative points, for EPS and revenue, for every submission. This, along with the number of estimates the user has submitted, are turned into a Confidence level 1-10, that is displayed on the user's profile to show how accurate they are. As you make more estimates that are accurate, your confidence level goes up. If your estimates are consistently inaccurate, then your confidence level will go down.
Compare the number of users to contributors: More contributors than users
If its service is provided by many contributors, how are their contributions aggregated? The users contributions are aggregated through the backend of the website. Users submit estimates through the front end, and an algorithm runs on the backend to calculate the simple average of every individual's estimate. Recently (~ 2 weeks ago), the company released a new consensus, called the Estimize select consensus. This consensus weights individuals' estimates based off of how accurate they have been in the past. This is an extra fee on top of the base data, and it has been shown to be more accurate than Wall Street 80% of the time (as opposed to the usual 70% for Estimize).

Graphs display individuals' estimates on the website, along with a list that shows every estimate made.
Describe the workflow for how the service is advertised, how the contributors contribute, and what the users get in the end: 1) The service is not advertised for directly. Advertisements occur through media appearances by company executives, and through word of mouth in the financial industry

2) Users contribute (as said above) through the front end of the website, or through software that directly hooks to their financial terminal (although this is about 2% of users).

3) Users get to see what the Estimize consensus is for any given stock, along with being able to build a track record for their history. They are also able to win prizes or money.
What is the scale of Estimize in terms of users? 17063
What is the scale of Estimize in terms of dollars? Currently revenue is around $1 million per year, but growing (only a guess given how many clients)
If Estimize were to scale up to 10-100 times its current size, how well do you think would its business model would work? Very well. The incentive scheme doesn't change much, given that people are still able to view the consensus estimates, and the top analysts still receive prizes. In fact, the data has shown to be even more accurate the more people that create estimates (supported by the Wisdom of the Crowds), so if the company were to scale up 10 or 100 times, it would be extremely beneficial. Right now the company is currently trying to figure out digital marketing strategies in order to grow its user base.
What kind of organization is it? Privately held company
How does Estimize generate revenue? The company generates revenue in multiple ways - namely, they generate revenue by selling their data to quantitative hedge fund.

Other ways the company generates revenue is through the screener, which allows individuals to manage lists of stocks with given characteristics around earnings (see previous answer), and this screener is sold for a monthly fee depending on the assets the individual or company manages.

Finally, Estimize generates revenue through its leagues feature - companies pay Estimize both in cash as well as through prizes to sponsor a league and have their name appear on the page.