VISTA Reliability Overview

Overview of VISTA

One part of our project uses Video Implemented Script Training for Aphasia (VISTA), a form of rehabilitation in which patients practice with scripts. What is unique about VISTA, though, is that the patients receive video recordings of a native speaker saying their script so that they can they practice along as they (re-)watch it. This ensures that they have the means to consistently train with their scripts rather than needing to wait until their next session with the clinician. The video itself only shows the nose, mouth, and chin of the native speaker being recorded. This is important because it visually demonstrates the movements that are necessary for the successful production of all the sounds of each word in the script.

Script Personalization & Procedure

Another notable aspect of our project is that the scripts are not pre-selected for the patients. Instead, we work with the patients to identify different parts of their life that they would like to practice with (e.g., family, trips, food, etc.). From there, we work with each patient to create a set of scripts that they will practice with for the entirety of the study.

The patients’ ability to produce the scripts will be probed at specific timepoints throughout the course of the experiment. We measure their performance at each probe by having the clinician transcribe what the patient produces for the target script. We then quantify the number of target words that were produced by using a specially designed calculator. The output from this calculation then acts as the primary outcome measure (POM) of the experiment. This provides an in-depth look at the maintenance or loss of language ability over the duration of the study.

Ensuring Data Reliability

However, we must be sure that when we later analyze our data that the measures are accurate and reliable. For this reason, we have developed a system to verify the accuracy of our POM. We have a second rater listen to the audio recording of the probe, and then transcribe and assess the POM for each patient’s probes. It is crucial that this rater does not know any information about the data they are given. That means they do not know which patient it is or the specific session (e.g., pre-treatment or post-treatment). This ensures that they are not biased as they transcribe and assess the output. By including 2 separates assessments, one from the clinician (rater 1) and the new rater (rater 2), we can then compare the transcription (and thus the POM) to assess accuracy and reliability of our data.

This process is critical and must involve careful planning by a reliability supervisor to ensure that the proper steps are taken in the right order. It is the reliability supervisor that will compare the two raters’ transcriptions.

Multiple Pipelines

Though our implementation of VISTA occurs as a single method, it provides us with two distinct pipelines for analysis: POM and Connected Speech. POM data allow us to examine patients' longitudinal performance with their scripts. The Connected Speech data let us identify how different linguistic features are impacted in patients' speech.

Multiple Reliability Team Members

As the Reliability workflow involves a great deal of interaction with moving parts across multiple projects, teams, etc., we have multiple team members that are involved with reliability. As of today (9/12/2025), the setup is that there are two reliability supervisors, and one undergraduate research assistant. Each reliability supervisor has a specific workflow for reliability that they are in charge of while the undergraduate RA helps the supervisors with organizing documents and populating cells within the reliability smartsheets. Recall that each workflow involves lots of interaction with other members of the research team, so this divides the work into manageable chunks.

For example, one reliability supervisor is in charge of VICS. This includes assigning and managing the transcribers, the transcriptions/coded documents as they go through the process. The other is in charge of POM. This includes keeping tabs on the SLPs' transcripts and progress with patients as they go through their sessions.

This page can be used as a reference to better understand these pipelines, the reliability procedures for these projects, and the responsibilities for the reliability team members.