Warning: this assignment is out of date. It may still need to be updated for this year's class. Check with your instructor before you start working on this assignment.
This assignment is before 11:59PM due on Thursday, April 29, 2021. Late days *are* allowed for this assignment.
Final Project : Part 4 (Optional)
This will be a preliminary analysis for your final report. Your final report and final presentation are both due on the day of the final. You’ll have to perform some sort of quantitative analysis on your project. Please look over the questions that you’ll have to answer for your final report, so that you get a sense of what is expected of you for the final writeup.
In this final project milestone, you will perform a preliminary analysis on the data that you collected from your classmates or through a crowdsourcing platform, if you’ve already begun collecting real data.
- How your system compare against an existing non-crowdsourcing method? For instance, you could compare the quality of non-expert annotations versus expert annotations.
- How accurate is your machine learning once it is trained on crowd-sourced training data? You should compare against a random and a majority-class baseline.
- How well did your quality control mechanism work at filtering out low quality work?
- How well could your method scale in terms of time/money?
- What is the latency of the crowd response, if your project has a large human-computer interaction component? How did you improve it?
- If you are doing a creative project, how much is the crowd-created project preferred to a sole-effort? How well does an iterative approach to creating something compare to a parallel approach?
What do for this project milestone
- Collect the data that was generated by your classmates and/or a real crowd
- Decide what analysis you want to use in your final writeup
- Do a preliminary analysis of the data
- Generate a figure, and write an explanation of what it shows (and say whether it matches your expectations or not)
- Link to the code that you used to generate the figure, tell us how to run it.
What to submit
Please create a PDF that answers these questions:
- Details about your team members
- Name of your project
- Who contributed to your preliminary data collection
- Classmates
- Crowd workers
- Both
- How many people made contributions to your project so far?
- How many data points have you collected so far?
- Do you plan to collect more data for your final project?
- What kind of analysis did you perform for this milestone?
- Comparison of different incentives
- Analysis of worker skills
- Investigation of quality control
- Comparison of aggregated results versus individual efforts
- Accuracy of a machine learning component
- Analysis of scaling
- Other
- Describe what analysis you performed
- Where the results what you expected?
- Include a figure that shows a visualization of your preliminary analysis (a graph, a chart, a table)
- Write a caption that describes the figure. Your caption should says how to interpret it, and gives your understanding of what it shows.
- What will you do differently for your final analysis?
- Give a link to the data files that you analyzed
- Give a link to the code, scripts, or spreadsheets that you used to generate the figure
- What is the command line argument that we would need to run to produce your analysis?
Grading Rubric
This assignment is worth 5 points of your overall grade in the course. The rubric for the assignment is given below.
- 1 point for assembling a data file from the crowd’s contributions
- 1 point for picking an analysis that makes sense for your project, and fits with the themes of the class
- 1 points for generating a well-designed figure
- 1 point for writing a good explanation of the figure
- 1 point for uploading and documenting your code