VISTA POM Consensus Calculator - Guide
In order to determine that the clinician rater (R1) for VISTA POM is a reliable rater, their (clinician R1) transcription and POM score must be compared with the new reliability rater’s (rater 2) transcription and score. By comparing the two raters we are able to see how similar or discrepant they are. Discrepancies can arise at the level of the transcription or at the level of the manual scoring of items that meet criteria (i.e. if the original script has a name but the transcript has a nickname, see POM Guide for more info).
The original VISTA POM Calculator is unable to do this directly because it only compares one rater’s scoring to the original script. For this reason, we have created a tool that allows us to compare the transcription from one rater to those of another. This tool is called the VISTA Multi-rater Reliability Calculator and can be found on our MADR Lab Tool page (link: https://comm-sg-p01.moody.utexas.edu/). Click on the Instructions tab to see a walkthrough on how to use this tool.
Once you click on the Multi-rater Reliability Calculator, you will be taken to its input page (see the image below). This is the interface you will use to provide all the necessary information for the calculator. You will need to provide some key information about the data you will be using: the Patient ID, the Script Name, and the language of the script. After this, you will copy and paste the full target script of the patient in the field “Target Script”. Then, you will copy and paste the transcription that was written by Rater 1 in “Rater 1 Input”. Lastly, you will copy and paste the transcription that was written by Rater 2 in “Rater 2 Input”.
If you have additional raters or additional scripts, you can click the button “Add Script Set”. If you do not have additional raters or additional scripts, you can click the “Calculate” button.
Figure 1. Input for VISTA Multi-Rater Reliability Calculator.
When you click the Calculate button, you will see a Results page. This will provide you with key information that will be used to determine the consistency of both raters’ transcriptions with one another and the target script. The Results page will show you the same information you entered into the Calculator in the prior step. It will also show you fractions that report the performance of the patient in saying the target script. Here is a quick description of each value:
Rater 1 Ratio: The number of words that matched between Rater 1 Input and the target script out of the total number of words in the target script.
Rater 2 Ratio: The number of words that matched between Rater 2 Input and the target script out of the total number of words in the target script.
Overall Ratio: The number of words in the target script that appeared in both Rater 1 Input and Rater 2 Input.
Agreed upon script words: The words in the target script that appeared in both Rater 1 Input and Rater 2 Input.
Agreed upon missed words: The words in the target script that did not appear in both Rater 1 Input and Rater 2 Input.
Disagreed upon script words: The words in the target script that only appeared in either Rater 1 Input or Rater 2 Input.
Difference in rating: The difference in Rater 1 Ratio and Rater 2 Ratio.
Figure 2. Output from VISTA Multi-rater Reliability Calculator.
If you would like this information to be built into a spreadsheet format, you can click “Export to Spreadsheet” at the bottom of the Results page. This will simply transform all of the above information into a spreadsheet.
Figure 3. Any example of what can be exported to a spreadsheet after the Results page.