To ensure the reliability of transcriptions for connected speech samples, the following structured process will be implemented:

Randomized Sample Selection:

A subset equivalent to 10% of the total transcription samples will be randomly selected for reliability testing. This selection will be project-specific (for example DrSantos_Spa_WABs_FTLD_2024, Kesha_nfvPPA_lvPPA_2024).

All samples used for reliability will be saved here 102--Connected Speech Reliability under the corresponding Project name folder. And only copied from the original file source 101--Connected Speech_Data

Connected Speech reliability roles

• Connected Speech reliability supervisor: usually project lead of the research project, responsible to assign, track sample completion, compare the samples and run reliability analysis.

• Connected Speech transcriber 1: first transcriber

• Connected Speech transcriber 2: second transcriber, reliability supervisor assigns the samples they have to transcribe

Reliability procedures

Once the samples are selected, the reliability supervisor will complete the following steps to begin the reliability process.

Reliability supervisor steps to complete for transcriber 1 sample:

1. Make sure sample is final by looking at the 1.2 SPANISH CS3 Transcription Status, 2.2 CATALAN CS3 Transcription Status reports

2. Copy the original transcript from its folder to this folder: 3. Picnic Scene_Rely (transcriber 1)

3. Edit the title of the transcript by adding the initials of the first transcriber (e.g. Sonia Marques, SM): CODE001_BACC001_CatRescue_Spa_Pre_20230914_Coded_SM.cha​

Reliability supervisor steps to complete for transcriber 2:

1. Assign samples using the reliability reports (1.3 SPANISH CS Reliability, 2.3 CATALAN CS Reliability)

   1. Use “TaskName_ProjectRely” column to filter the rows relevant to the project specific samples that need to be transcribed for reliability

   2. In “Rely rater transcriber 2” column add name of the 2nd transcriber, for the samples assigned to be transcribed for reliability. Sort or filter by this column so the transcriber 2 can use the report more functionally.

2. Create folder within 102--Connected Speech Reliability with this folder structure:

Open

3. Copy assigned audios from 101--Connected Speech_Data audio folders to “1.TaskName_Audios_Reliability”

4. Copy assigned whisper from 101--Connected Speech_Data audio folders to “1.TaskName_Whisper_Reliability”

5. Inform transcriber 2 about the assigned samples using the reliability reports, transcriber 2 starts next steps

Steps for transcriber 2 (same steps followed in general transcription projects)

For more detailed information see https://cloud.wikis.utexas.edu/wiki/spaces/MADRWiki/pages/56197560 , https://cloud.wikis.utexas.edu/wiki/x/xoFZAw , https://cloud.wikis.utexas.edu/wiki/spaces/MADRWiki/pages/56198263

• Here is a summary to share with transcriber 2:

1. Create transcription (.cha) file by copying the template that exists within this folder: 4. Picnic Scene_Rely (transcriber 2)​

2. Copy the whisper output CODE001_BACC001_PicnicScene_Spa_Pre_20230914.txt and paste it in the template (.cha)​

3. Fill out some particular fields in the headers of the file (@) (language, participant code, etc.,)​

4. Segment in utterances following the transcription protocol rules CHAT

5. Code with the transcription protocol rules​

6. Use CLAN to detect typos or spelling mistakes (command CHECK and command MOR)

7. Save in 4. Picnic Scene_Rely (transcriber 2) and make sure naming is correct by adding your initials (e.g. Ana Quinonez, AQ): CODE001_BACC001_CatRescue_Spa_Pre_20230914_Coded_AQ.cha​

Running Rely Initial Comparison for Reliability (reliability supervisor)

1. Run transcriber 1 and transcriber 2 Coded_Rely samples through RELY

2. Save RELY output to this folder: 5. Picnic Scene_Rely output. The name of the file should be the original file name (excluding Coded_Rely_Initials), as provided here: BISE010_PreTX_PicnicDescription_Castellano.rely

3. Add score in Smartsheet CS Data Analysis in the 2 columns (report)reliability reports (1.3 SPANISH CS Reliability, 2.3 CATALAN CS Reliability)

4. The selected samples will be assessed for transcription accuracy using the CLAN software and using Rely (How to use RELY on CLAN). Two key metrics will be evaluated:

• Percentage of Utterances with Matching Codes: Calculated as the proportion of all utterances where the assigned codes match perfectly between transcribers.

• Percentage of Words with Matching Codes: Calculated as the proportion of individual words with identical coding across transcriptions.

• Discrepancy Resolution:
  If 80% agreement is NOT achieved:

  • Anything below 80% we have to re-transcribe (with transcriber 3) and we delete the unreliable transcription from the parent/original folder.

  • To identify the unreliable transcription, we compare transcriber 3 to both transcriber 1 and transcriber 2 to identify who was most correct. Update the most congruent transcriber pair in the Smartsheet with the final reliability values.

    • Recall, that in the case that pair 1+3 are less congruent, Rater 2 will replace Rater 1.

  • If 80% is then achieved between a pair of raters, we proceed to the steps below.

  • If 80% is not achieved between any rater pair, we need to discuss and review.

    • Likely this file will need to go through consensus because something may be particularly challenging about the transcript (this has only happened on one occasion thus far 0422026)

  • IMPORTANT! If transcriber 1 is not included in the pair that reached 80%, we’ll need to replace the ORIGINAL transcript following consensus procedures outlined below

    • See the section below regarding where the final file can be saved: Finalization of Reliable Transcriptions

• If 80% agreement is achieved (or when it is achieved following re-transcription):

  • If reliability falls between 80 to 100% for both reliability measures we have the consensus meeting, we look at the samples side by side and update the final version of the transcription. However, the exception to this rule is that if either agreement value (words or utterances), reaches 100% agreement, we will not conduct consensus and we will always select Rater 1’s transcription.

    • A meeting will be scheduled between the involved transcribers to review discrepancies. It’s recommended to have this meeting after running through Rely for all the samples used for reliability for a given project.

  • Consensus will be reached on the appropriate transcription for all disputed elements. The date of the meeting will be added in CS Data Analysis Smartsheets (Catalan: https://app.smartsheet.com/reports/96Xjp4G54hwCXM3pVVHP5xcrvVFfPXgh9wq49g81; Spanish: https://app.smartsheet.com/reports/qJP8g3XRmJ2mqw5RgWwrxVCjxHFR2Xp3Pqg5MPf1)

• Consensus Meeting:

  • When possible, the two raters will schedule a meeting and each transcriber will have their transcript open and they will review discrepancies between the samples by listening to the sample

  • A third transcript will be generated that reflects the final version of the sample representing the agreed upon transcript between the two transcribers

    • In some cases, one or both transcribers may no longer be available to the lab. In these cases, the single transcriber who is still with the lab, should coordinate a meeting with a third party who speaks the language they are making decisions about. If both transcribers are not part of the lab, then two lab members should review the transcripts and follow the steps we have outlined.

• Finalization of Reliable Transcriptions:
  Following the consensus meeting:

  • Edits will be made to the original transcription located in its original folder (e.g. B--Connected Speech_Data) based on agreed-upon revisions in the same meeting (i.e., the final transcript).

  • Delete any old or unreliable transcript versions from the original folder so only the final version remains.

  • The transcription will then be considered finalized for reliability and all the samples of that project will be considered reliable enough to extract the linguistic measurements.

• Documentation of Process:

  • All final reliability calculations will be documented in the Connected Speech Reliability Smartsheet Reports.

  • Project-specific aspects of the reliability process are always kept in the following folder: 102--Connected Speech Reliability Folder

    • Project-specific aspects are defined in section X

Overview of Reliability Folder Structure:

Open

See scheme here: https://cloud.wikis.utexas.edu/wiki/spaces/MADRWiki/whiteboard/246612214

How to use RELY on CLAN

RELY Function 4:

“The fourth function of the RELY command is to estimate the overall match between two transcripts on the main line. It is very difficult to define this type of comparison precisely. Instead, RELY uses a rough-and-ready "bag of words" comparison method that simply looks at the overall match of the main line items in the two versions. The command for this type of analysis adds the +d switch, and the output is the percentage of overall overlap.” (This is directly sourced from the CLAN manual: https://talkbank.org/manuals/CLAN.pdf)

This function provides two output calculations:

• % of all utterances with matching codes

• % % of all words examined with matching codes

How to run this function:

Step 1:

• Prepare Files: Make sure you have two coded files for comparison (e.g., two raters coding the same language sample) in the CLAN-compatible .cha format. Since the files should have the same title, it is helpful to modify each file name to differentiate between the two.

Step 2:

• Type in the following code into the command window: rely +d sample.cha samplea.cha

• Select “file in” and add the two coded files you wish to process. Make sure that the CLAN more library is linked on BOX. Here are the instructions to complete this step.

  Open

  

   

• Once you’ve done so, select “done”.

Open

Step 3:

• After running, RELY provides a report showing percent agreement across your chosen categories. This breakdown helps identify areas where raters agree most and where further clarification might be needed.

• Here is an example of what the reliability report looks like:

Open