Versions Compared

Version Old Version 26 New Version 27
Changes made by

Sonia Marqués

Sonia Marqués

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

To ensure the reliability of transcriptions for connected speech samples, the following structured process will be implemented:

Randomized Sample Selection:

A subset equivalent to 10% of the total transcription samples will be randomly selected for reliability testing. This selection will be project-specific (for example DrSantos_Spa_WABs_FTLD_2024, Kesha_nfvPPA_lvPPA_2024).

All samples used for reliability will be saved here 102--Connected Speech Reliability

...

under the corresponding Project name folder. And only copied from the original file source 101--Connected Speech_Data

Connected Speech reliability roles

  • Connected Speech reliability supervisor: usually project lead of the research project, responsible to assign, track sample completion, compare the samples and run reliability analysis

  • Connected Speech transcriber 1: first transcriber

  • Connected Speech transcriber 2: second transcriber, reliability supervisor assigns the samples they have to transcribe

Reliability procedures

Once the samples are selected, transcribers the reliability supervisor will complete the following steps to begin the reliability process. 1. Reliability procedures

Reliability supervisor steps to complete for transcriber 1 sample:

  1. Make sure sample is final by looking at the 1.2 SPANISH CS3 Transcription Status, 2.2 CATALAN CS3 Transcription Status reports

  2. Copy the original transcript from its folder to this folder: 3. Picnic Scene_Rely (transcriber 1)

  3. Edit the title of the transcript by adding your the initials of the first transcriber (e.g. Sonia Marques, SM): CODE001_BACC001_CatRescue_Spa_Pre_20230914_Coded_SM.cha​

...

Reliability supervisor steps to complete for transcriber 2:

  1. Locate the audio files in the folder/link to the audio files: 1. S3_PicnicScene_Picture Description_Audios_Reliability

  2. Locate the whisper output files in this folder: 2. PicnicScene_whisper_output_ReliabilityAssign samples using the reliability reports (1.3 SPANISH CS Reliability, 2.3 CATALAN CS Reliability)

    1. Use “TaskName_ProjectRely” column to filter the rows relevant to the project specific samples that need to be transcribed for reliability

    2. In “Rely rater transcriber 2” column add name of the 2nd transcriber, for the samples assigned to be transcribed for reliability. Sort or filter by this column so the transcriber 2 can use the report more functionally. (!! find a way to randomize it within smartsheet)

  3. Create folder within 102--Connected Speech Reliability with this folder structure:

...

  1. Copy assigned audios from 101--Connected Speech_Data audio folders to “1.TaskName_Audios_Reliability”

  2. Copy assigned whisper from 101--Connected Speech_Data audio folders to “1.TaskName_Whisper_Reliability”

  3. Meet with transcriber 2 and explain next steps

Steps for transcriber 2 (same steps followed in general transcription projects)

For more detailed information see https://cloud.wikis.utexas.edu/wiki/spaces/MADRWiki/pages/56197560/4.+Formatting+transcription+process+transcribers?atlOrigin=eyJpIjoiOTEzMGVmMmZjNGJjNGQ2ZDhlYjgwYjY2NjU2NDQ0OGMiLCJwIjoiYyJ9 , https://cloud.wikis.utexas.edu/wiki/x/xoFZAw , https://cloud.wikis.utexas.edu/wiki/spaces/MADRWiki/pages/56198263/6.+Coding+process+transcribers?atlOrigin=eyJpIjoiYTFkYzMzZTc4MDM1NDEwZDk1YTRjYTU1YzExNTVjODEiLCJwIjoiYyJ9

  • Here is a summary to share with transcriber 2:

  1. Create transcription (.cha) file by copying the template that exists within this folder: 4. Picnic Scene_Rely (transcriber 2)​

  2. Copy the whisper output CODE001_BACC001_PicnicScene_Spa_Pre_20230914.txt and paste it in the template (.cha)​

  3. Fill out some particular fields in the headers of the file (@) (language, participant code, etc.,)​

  4. Segment in utterances following the transcription protocol rules CHAT

  5. Code with the transcription protocol rules​

  6. Use CLAN to detect typos or spelling mistakes (command CHECK and command MOR)

  7. Save in 4. Picnic Scene_Rely (transcriber 2) and make sure naming is correct by adding your initials (e.g. Ana Quinonez, AQ): CODE001_BACC001_CatRescue_Spa_Pre_20230914_Coded_AQ.cha​

...

Running Rely Initial Comparison for Reliability

...

(reliability supervisor)

  1. Run transcriber 1 and transcriber 2 Coded_Rely samples through RELY

  2. Save RELY output to this folder: 5. Picnic Scene_Rely output. The name of the file should be the original file name (excluding Coded_Rely_Initials), as provided here: BISE010_PreTX_PicnicDescription_Castellano.rely

  3. Add score in Smartsheet CS Data Analysis in the 2 columns (report)

  4. Spanish CS Data Analysis Report: https://app.smartsheet.com/reports/qJP8g3XRmJ2mqw5RgWwrxVCjxHFR2Xp3Pqg5MPf1

    5. Catalan CS Data Analysis Report: https://app.smartsheet.com/reports/96Xjp4G54hwCXM3pVVHP5xcrvVFfPXgh9wq49g81

    reliability reports (1.3 SPANISH CS Reliability, 2.3 CATALAN CS Reliability)

  5. The selected samples will be assessed for transcription accuracy using the CLAN software and using Rely (How to use RELY on CLAN). Two key metrics will be evaluated:

  • Percentage of Utterances with Matching Codes: Calculated as the proportion of all utterances where the assigned codes match perfectly between transcribers.

  • Percentage of Words with Matching Codes: Calculated as the proportion of individual words with identical coding across transcriptions.

...