Once you click on the Multi-rater Reliability Calculator, you will be taken to its input page (see the image below). This is the interface you will use to provide all the necessary information for the calculator. You will need to provide some key information about the data you will be using: the Patient ID, the Script Name, and the language of the script. After this, you will copy and paste the full target script of the patient in the field “Target Script”. Then, you will copy and paste the transcription that was written by Rater 1 in “Rater 1 Input”. Lastly, you will copy and paste the transcription that was written by Rater 2 in “Rater 2 Input”.

If you have additional raters or additional scripts, you can click the button “Add Script Set”. If you do not have additional raters or additional scripts, you can click the “Calculate” button.

Figure 1. Input for VISTA Multi-Rater Reliability Calculator.

When you click the Calculate button, you will see a Results page. This will provide you with key information that will be used to determine the consistency of both raters’ transcriptions with one another and the target script. The Results page will show you the same information you entered into the Calculator in the prior step. It will also show you fractions that report the performance of the patient in saying the target script. Here is a quick description of each value:

Rater 1 Ratio: The number of words that matched between Rater 1 Input and the target script out of the total number of words in the target script.

Rater 2 Ratio: The number of words that matched between Rater 2 Input and the target script out of the total number of words in the target script.

Overall Ratio: The number of words in the target script that appeared in both Rater 1 Input and Rater 2 Input.

Agreed upon script words: The words in the target script that appeared in both Rater 1 Input and Rater 2 Input.

Agreed upon missed words: The words in the target script that did not appear in both Rater 1 Input and Rater 2 Input.

Disagreed upon script words: The words in the target script that only appeared in either Rater 1 Input or Rater 2 Input.

Difference in rating: The difference in Rater 1 Ratio and Rater 2 Ratio.

Figure 2. Output from VISTA Multi-rater Reliability Calculator.

If you would like this information to be built into a spreadsheet format, you can click “Export to Spreadsheet” at the bottom of the Results page. This will simply transform all of the above information into a spreadsheet.

Figure 3. Any example of what can be exported to a spreadsheet after the Results page.

VISTA POM Consensus Calculator - Guide