This directory contains example code for running a paraphrase judgement HIT with embedded quality controls. The directory contains the following files:

- input/instructions.txt: instructions for the HIT. This file is read in by post.py in order to add instructions to the HIT interface.
- input/wikipedia.tsv: data to be labeled, taken from Wikipedia. The file is tab-separated with the following columns: an id, a syntactic category, a phrase, a list of paraphrases (delimted with |||), a context in which the phrase appears.
- input/controls.tsv: gold-standard control questions derived from WordNet. Here, the sentences are example usage sentences associated with wordnet synsets. The format is as above, with two additional columns: known positive controls (words in the synset, which should be good paraphrases for the phrase in the context), and known negative controls (randomly chosen words, which should be bad paraphrases of the phrase in the context)

- src/settings.py: parameter settings used in post.py and other scripts, including access keys, host site (sandbox or regular mturk), and price for HITs
- src/post.py: post the HIT to MTurk
- src/download.py: download the workers results and save to a csv file
- src/grade.py: grade the downloaded HITs, and approve/reject according to workers' performance on the controls
- src/remove.py: remove the HITs from MTurk

Usage:

   $ python src/post.py //posts HIT to MTurk using settings in settings.py
   $ python src/download.py //download the HITs that workers have submitted, writing them to a file called ungraded.TIMESTAMP.csv
   $ python src/grade.py ungraded.TIMESTAMP.csv //grade controls and approve/reject accordingly, writing results to a file called graded.TIMESTAMP.csv
