VISTA Reliability Overview
Overview of VISTA
One part of our project uses Video Implemented Script Training for Aphasia (VISTA), a form of rehabilitation in which patients practice with scripts. What is unique about VISTA, though, is that the patients receive video recordings of a native speaker saying their script so that they can they practice along as they (re-)watch it. This ensures that they have the means to consistently train with their scripts rather than needing to wait until their next session with the clinician. The video itself only shows the nose, mouth, and chin of the native speaker being recorded. This is important because it visually demonstrates the movements that are necessary for the successful production of all the sounds of each word in the script.
Script Personalization & Procedure
Another notable aspect of our project is that the scripts are not pre-selected for the patients. Instead, we work with the patients to identify different parts of their life that they would like to practice with (e.g., family, trips, food, etc.). From there, we work with each patient to create a set of scripts that they will practice with for the entirety of the study.
The patients’ ability to produce the scripts will be probed at specific timepoints throughout the course of the experiment. We measure their performance at each probe by having the clinician transcribe what the patient produces for the target script. We then quantify the number of target words that were produced by using a specially designed calculator. The output from this calculation then acts as the primary outcome measure (POM) of the experiment. This provides an in-depth look at the maintenance or loss of language ability over the duration of the study.
Ensuring Data Reliability
However, we must be sure that when we later analyze our data that the measures are accurate and reliable. For this reason, we have developed a system to verify the accuracy of our POM. We have a second rater listen to the audio recording of the probe, and then transcribe and assess the POM for each patient’s probes. It is crucial that this rater does not know any information about the data they are given. That means they do not know which patient it is or the specific session (e.g., pre-treatment or post-treatment). This ensures that they are not biased as they transcribe and assess the output. By including 2 separates assessments, one from the clinician (rater 1) and the new rater (rater 2), we can then compare the transcription (and thus the POM) to assess accuracy and reliability of our data.
This process is critical and must involve careful planning by a reliability supervisor to ensure that the proper steps are taken in the right order. It is the reliability supervisor that will compare the two raters’ transcriptions.
Multiple Pipelines
Though our implementation of VISTA occurs as a single method, it provides us with two distinct pipelines for analysis: POM and Connected Speech. POM data allow us to examine patients' longitudinal performance with their scripts. The Connected Speech data let us identify how different linguistic features are impacted in patients' speech.
Multiple Reliability Team Members
As the Reliability workflow involves a great deal of interaction with moving parts across multiple projects, teams, etc., we have multiple team members that are involved with reliability. As of today (9/12/2025), the setup is that there are two reliability supervisors, and one undergraduate research assistant. Each reliability supervisor has a specific workflow for reliability that they are in charge of while the undergraduate RA helps the supervisors with organizing documents and populating cells within the reliability smartsheets. Recall that each workflow involves lots of interaction with other members of the research team, so this divides the work into manageable chunks.
For example, one reliability supervisor is in charge of VICS. This includes assigning and managing the transcribers, the transcriptions/coded documents as they go through the process. The other is in charge of POM. This includes keeping tabs on the SLPs' transcripts and progress with patients as they go through their sessions.
This page can be used as a reference to better understand these pipelines, the reliability procedures for these projects, and the responsibilities for the reliability team members.
VISTA Connected Speech (VICS) Reliability
VICS
This table is for the Reliability Team to better understand the responsibilities pertaining to reliability for VICS and to better understand the individuals steps that are part of this process.
Related Reliability Pages
These are the responsibilities for the VICS Reliability Supervisor and the Undergraduate Assistant:
Reliability Supervisor:
At regular intervals, check the Reliability Admin Smartsheets for both Spanish and Catalan to identify which transcripts are ready to be assigned to the transcribers.
If the transcript and accompanying documents (e.g., audio files) are correctly blinded and ready, use the Transcriber 1 and Transcriber 2 columns to properly choose a transcriber to begin working on the transcript.
At regular intervals, check that the undergraduate assistant is successfully blinding the materials as is described in this page.
Ensure that file names are properly blinded and labeled, check that the links are working correctly, and that all files are in the proper folders.
Also check that the status of blinding is correctly labeled to ensure that you are aware of the progress of each stage of this workflow.
As needed, update the undergraduate assistant when patients have reached different milestones (e.g., done pre-tx, first half of treatment, etc.) so they are aware of when they might be able to expect incoming transcripts that are ready for blinding.
As needed, meet with the undergraduate assistant and/or other Reliability team members to discuss problems that arise and/or the overall progress of this process.
Undergraduate Assistant:
Once or twice a week, check the Reliability Admin Smartsheets for both Spanish and Catalan to determine if new transcripts will be ready for blinding.
Specifically check the Whisper team columns to see if there are new transcripts that are ready.
For new transcripts that are identified as being ready, begin the blinding procedure by copying over the transcripts, audio files, etc.
Be sure to follow the proper steps to ensure that the naming of files is appropriate, the files are saved in the correct locations, and that any missing files, etc., are correctly labeled in the Reliability Admin Smartsheets.
Correctly and accurately document blinding procedure status updates for each transcript using the Reliability Admin Smartsheets.
This is a very important step because it easily allows the team to see the progress and understand where we all are in this process.
As needed, please reach out to the Reliability Supervisor if there are unexpected problems or issues with completing the blinding procedure.
This can range from unexpectedly missing files, transcripts that misnamed in the original folders, etc.
Be sure to maintain proper organization to this workflow.
There are a lot of moving parts and files, and it is very easy to become confused or lose your place when working along in the Smartsheets and Box folders.
VISTA Connected Speech (VICS) Reliability
Note that because neither the pipeline for Transcriber 1 or Transcriber 2 relies on the other Transcriber, there is no specific order to how this process can happen.
In other words, Transcriber 2 can begin before Transcriber 1 or the reverse. This might need to happen in cases where the SLP’s POM transcript is temporarily inaccessible, which is the first step for Transcriber 1’s pipeline but has no bearing on Transcriber 2.
These are the steps that must be completed for Transcriber 1’s inclusion in the Reliability process:
These are the steps that must be completed for Transcriber 2’s inclusion in the Reliability process: