To ensure the reliability of transcriptions for connected speech samples, the following structured process will be implemented:
Randomized Sample Selection:
A subset equivalent to 10% of the total transcription samples will be randomly selected for reliability testing. This selection will be project-specific (for example DrSantos_Spa_WABs_FTLD_2024, Kesha_nfvPPA_lvPPA_2024)
Initial Comparison for Reliability:
The selected samples will be assessed for transcription accuracy using the CLAN software and using Rely (How to use RELY on CLAN). Two key metrics will be evaluated:
Percentage of Utterances with Matching Codes: Calculated as the proportion of all utterances where the assigned codes match perfectly between transcribers.
Percentage of Words with Matching Codes: Calculated as the proportion of individual words with identical coding across transcriptions.
Discrepancy Resolution:
If 100% agreement is not achieved:
A meeting will be scheduled between the involved transcribers to review discrepancies. It’s recommended to have this meeting after running through Rely all the samples used for reliability.
Consensus will be reached on the appropriate transcription for all disputed elements.
Finalization of Reliable Transcriptions:
Following the consensus meeting:
Edits will be made to the coded transcriptions based on agreed-upon revisions in the same meeting.
The transcription will then be considered finalized for reliability and all the samples of that project will be considered reliable enough to extract the linguistic measurements.
Documentation of Process:
All reliability calculations and consensus resolutions will be documented in the Connected Speech Reliability Smartsheet Reports for project records, contributing to transparency and repeatability in the transcription process.
See steps that are needed for the reliability procedures:
File location:
/Users/skm3435/Library/CloudStorage/Box-Box/SLHS_Grasso/BCN Participants - Recordings and Materials/MADR_HSP R01/5. Data_Raw/B1--Connected Speech Reliability
RELY Function 4:
“The fourth function of the RELY command is to estimate the overall match between two transcripts on the main line. It is very difficult to define this type of comparison precisely. Instead, RELY uses a rough-and-ready "bag of words" comparison method that simply looks at the overall match of the main line items in the two versions. The command for this type of analysis adds the +d switch, and the output is the percentage of overall overlap.” (This is directly sourced from the CLAN manual: https://talkbank.org/manuals/CLAN.pdf)
This function provides two output calculations:
% of all utterances with matching codes
% % of all words examined with matching codes
How to run this function:
Step 1:
Prepare Files: Make sure you have two coded files for comparison (e.g., two raters coding the same language sample) in the CLAN-compatible .cha format. Since the files should have the same title, it is helpful to modify each file name to differentiate between the two.
Step 2:
Type in the following code into the command window: rely +d sample.cha samplea.cha
Select “file in” and add the two coded files you wish to process. Make sure that the CLAN more library is linked on BOX. Here are the instructions to complete this step.
Once you’ve done so, select “done”.
Step 3:
After running, RELY provides a report showing percent agreement across your chosen categories. This breakdown helps identify areas where raters agree most and where further clarification might be needed.
Here is an example of what the reliability report looks like:
