Final Project : The End

Below are the questions that you should address in your final report. lease submit your answers in a PDF that you submit to Gradescope.

You must answer the questions in all of the sections below, but you only need to create graphs for 2 sections.

Basic project information

• Name of your project
• Name of your teammates
• Give a one sentence description of your project. Please use the name of the project in your description.
• Logo for your project.
• What problem does it solve?
• What similar projects exist?
• What type of project is it?
  • Human computation algorithm
  • Social science experiment with the crowd
  • A tool for crowdsourcing
  • A business idea that uses crowdsourcing
  • Other
• What was the main focus of your team’s effort
  • Engineering a complex system
  • Conducting an in depth analysis of data
  • Something in between
• How does your project work? Describe each of the steps involved in your project. What parts are done by the crowd, and what parts will be done automatically.
• Provide a link to your final presentation video. Give the full path to your Vimeo video, and the password, if it is not public.

The Crowd

• Who are the members of your crowd?
• For your final project, did you simulate the crowd or run a real experiment?
  • Simulated crowd
  • Real crowd
• If the crowd was simulated, how did you collect this set of data?
• If the crowd was simulated, how would you change things to use a real crowd?
• If the crowd was real, how did you recruit participants?
• How many unique participants did you have?

Incentives

• What motivation does the crowd have for participating in your project?
  • Pay
  • Altruism
  • Enjoyment
  • Implicit work
  • Reputation
  • Other
• How do you incentivize the crowd to participate? Please write 1-3 paragraphs giving the specifics of how you incentivize the crowd. If your crowd is simulated, then what would you need to do to incentivize a real crowd?
• Did you perform any analysis comparing different incentives?
• If you compared different incentives, what analysis did you perform? If you have a graph analyzing incentives, include the graph here.

What the crowd gives you

• What does the crowd provide for you?
• Is this something that could be automated?
• If it could be automated, say how. If it is difficult or impossible to automate, say why.
• Did you train a machine learning component from what the crowd gave you?
• If you trained a machine learning component, describe what you did.
• Did you analyze the quality of the machine learning component? For instance, did you compare its quality against crowd workers using an n-fold cross validation?
• If you have a graph analyzing a machine learning component, include the graph here.
• Did you create a user interface for the crowd workers? Answer yes even if it’s something simple like a HTML form on CrowdFlower.
• If yes, please include a screenshot of the crowd-facing user interface in your report. You can include multiple screenshots if you want.
• Describe your crowd-facing user interface. This can be a short caption for the screenshot. Alternately, if you put a lot of effort into the interface design, you can give a longer explanation of what you did.

Ethics

Your report should include a discussion of ethics. One method of address this requirement is to nominate one of your teammates to ask ethics question of the whole group.

• Should my application exist at all?
• Does this task potentially expose workers to harm (for example, content moderation)? What effect can it have on them?
• Are you fairly compensating the workers for their time?
• If you are creating a dataset for machine learning: ** Are you allowed to access the data that you’re labeling (have you ensured that it isn’t private data)? ** Is there the potential for introducing bias in your machine learning classifer (unintended discrimination against a group because of underrepresentaiton)? ** ML algorithms sometimes amplify bias in the training data. What would be the consequences if this happens in your ML system?
• Is your evaluation sound? Do the conclusions you reach stand up to scientific scruitny?

Skills

• Do your crowd workers need specialized skills?
• What sort of skills do they need?
• Do the skills of individual workers vary widely?
• If skills vary widely, what factors cause one person to be better than another?
• Did you analyze the skills of the crowd?
• If you analyzed skills, what analysis did you perform? How did you analyze their skills? What questions did you investigate? Did you look at the quality of their results? Did you analyze the time it took individuals to complete the task? What conclusions did you reach?
• Do you have a graph analyzing skills? If you have a graph analyzing skills, include the graph here.

Quality Control

• Is the quality of what the crowd gives you a concern?
• How do you ensure the quality of the crowd provides?
• If quality if a concern, then what did you do for quality control? If it is not a concern, then what about the design of your system obviates the need for explicit QC? This answer should be substantial (several paragraphs long).
• Did you analyze the quality of what you got back? For instance, did you compare the quality of results against a gold standard? Did you compare different QC strategies?
• What analysis did you perform on quality?
• What questions did you investigate? What conclusions did you reach?
• Do you have a graph analyzing quality? If you have a graph analyzing quality, include the graph here.

Aggregation

• How do you aggregate the results from the crowd?
• Did you analyze the aggregated results?
• What analysis did you perform on the aggregated results? What questions did you investigate? Did you compare aggregated responses against individual responses? What conclusions did you reach?
• Do you have a graph analyzing the aggregated results? If you have a graph analyzing the aggregated results, include the graph here.
• Did you create a user interface for the end users to see the aggregated results? If yes, pleaseinclude a screenshot of the user interface for the end user in your final report. You can include multiple screenshots, if you want.
• Describe what your end user sees in this interface. This can be a short caption for the screenshot. Alternately, if you put a lot of effort into the interface design, you can give a longer explanation of what you did.

Scaling Up

• What is the scale of the problem that you are trying to solve?
• Would your project benefit if you could get contributions from thousands of people?
• If it would benefit from a huge crowd, how would it benefit?
• What challenges would scaling to a large crowd introduce?
• Did you perform an analysis about how to scale up your project? For instance, a cost analysis?
• What analysis did you perform on the scaling up?
• What questions did you investigate? What conclusions did you reach?
• Do you have a graph analyzing scaling? If you have a graph analyzing scaling, include the graph here.

Project Analysis

• Did your project work? How do you know? Analyze some results, discuss some positive outcomes of your project.
• Do you have a graph analyzing your project? If you have a graph analyzing your project, include the graph here.
• What were the biggest challenges that you had to deal with?
• Where there major changes between what you originally proposed and your final product?
• If so, what changed between your original plan and your final product?
• What are some limitations of your product? If yours is an engineering-heavy project, what would you need to overcome in order to scale (cost/incentives/QC…)? If yours was a scientific study, what are some sources of error that may have been introduced by your method.
• Did your results deviate from what you would expect from previous work or what you learned in the class?
• If your results deviated, why might that be?

Technical Challenges

• Did your project require a substantial technical component? Did it require substantial software engineering? Did you need to learn a new language or API?
• If project required a substantial technical component, describe the largest technical challenge you faced.
• How did you overcome this challenge? What new tools or skills were required? Feel free to nerd out a bit, to help us understand the amount of work that was required.
• Do you have any screen shots or flow diagrams to illustrate the technical component you described? If so, include the graph here.

Other info (optional)

• Is there anything else you’d like to say about your project?
• If you have additional information about your project that didn’t fit into the above questions, put it here.