VISTA Reliability - Blinding Procedure
Because our projects have multiple steps in which lab members must transcribe patients' speech, we have decided to use a system that takes advantage of work that has already been done. Checking the reliability of transcriptions normally requires comparing the transcriptions of two (or more) individuals and identifying any discrepancies between them. As we derive two independent pipelines for VISTA data (POM & Connected Speech), this would typically take 4 individuals transcribing patients' data (2 for POM and 2 for Connected Speech). However, we elected to use a reliability structure that only requires 3 individuals by using the transcription of one transcriber (who we denote as Rater 2) for both POM and Connected Speech reliability. This allows us to minimize the work that is required for assessing reliability.
For a diagram, please see here.
In order to use Rater 2’s transcriptions for both POM and Connected Speech, we need to ensure that we control for potential biases similarly across POM and Connected Speech. In other words, we need to be careful about the amount of information we give the Raters/transcribers for any given observation that they need to transcribe. For information that must be withheld from the transcribers to prevent bias, we must blind
Patient ID
Language (Spanish, Catalan)
Observation (pre1, pre2, post1, tx session, post2, 3mo, 6mo, 12mo)
Script (1, 2, 3, 4, 5, 6, 7, 8)
Script training status (trained, untrained)
Our goal is to harmonize the two VISTA Reliability procedures as much as possible. For blinding, there are some differences that need to be addressed. For POM, the SLP will always be Rater 1. As such, they will have access to the Patient ID, the language, the observation, the script and the script training status. However, for Connected Speech, both transcribers must be blinded to the observation time point of the audio. We have decided that the most straightforward way to unify these processes is to have VISTA POM Reliability use clipped audio samples (rather than session videos like previously) that come directly from the current transcription pipeline along with different transcripts to be used as the base for the transcribers' coding. For Transcriber 1, the Reliability Supervisor will provide them with the SLP’s POM transcript to use as a base. For Transcriber 2, the Reliability Supervisor will provide them with the Whisper transcript. This means that the clipping and whisper teams will carry on like normal for the sessions for VISTA Connected Speech with the addition of the two treatment sessions needed for VISTA POM Reliability.