Reasoning Beyond Transcription
Can audio-language models reason over disfluent child speech?
Reasoning Beyond Transcription: Audio Language Models on Child Stuttering Speech
University of Florida
Direct cross-modal audio processing without intermediate human-written transcript
01 / Evaluation Challenge
Why Disfluent Child Speech Challenges Audio Models
Existing speech-language models are primarily trained on fluent adult data. Evaluating ALMs on child stuttering speech presents three compound acoustic and semantic obstacles.
1. Child Speech
Distinct acoustic pitch, fundamental frequency (F0), unique vocal tract length, and developmental prosodic variations compared to adult benchmarks.
2. Disfluency
Repetitions, prolongations, silent blocks, and filled pauses interrupt word temporal bounds and break standard language model context tokens.
3. Speaker Interference
Interviewer prompts, overlapping dialogue, background scaffolding, and mixed turn-taking disrupt child-focused focus.
02 / Evaluation Tasks
Two Benchmark Tasks for Semantic Evaluation
Instead of standard word error rate (WER), we formulate two end-to-end tasks to probe whether models truly reason over disfluent spoken audio.
Child-Focused Semantic Summarization
Given a raw audio recording of a child talking (often mixed with an interviewer), generate a concise summary strictly representing the child's intentions while filtering out adult prompt leakage.
Child Speech Entailment (NLI)
Given raw child audio (premise) and a textual hypothesis, determine whether the hypothesis is entailed by, neutral to, or contradicted by the child's spoken message.
Standardized prompts ensure zero-shot evaluation without task-specific fine-tuning.
Can You Reason About This Audio?
Test your intuition against state-of-the-art Audio Language Models. Listen to the spoken prompt (or inspect the schematic context), evaluate the hypothesis, and see where ALMs succeed or fail.
“The child enjoys playing sports in large noisy groups.”
03 / Experimental Methodology
Three Diagnostic Reasoning Paths
By comparing end-to-end ALMs against cascaded pipelines and human transcript oracle baselines, our benchmark pinpoints whether failures stem from acoustic feature extraction or semantic reasoning.
Diagnostic Error Localization Strategy
By comparing Path A (Direct ALM) against Path B (ASR Cascade) and Path C (Oracle Transcript), we decouple whether reasoning failures are driven by Acoustic Processing Disruption or Semantic Reasoning Bottlenecks.
04 / Corpus & Benchmark
Benchmark at a Glance
Constructed from the clinical Voices of Children Who Stutter corpus (FluencyBank), systematically structured across acoustic density levels and reasoning difficulties.
Entailment Reasoning Complexity Granularity
Evidence can be identified directly from a single distinct utterance.
05 / Evaluated Systems
Model Landscape & Taxonomy
A comprehensive evaluation of state-of-the-art Audio Language Models alongside established ASR + Text LLM cascades.
Direct End-to-End Audio Language Models (ALMs)
Process continuous acoustic features directly without text transcription
06 / Headline Benchmark Findings
Direct Audio Models vs. Cascaded Pipelines
Evaluating whether end-to-end continuous audio processing matches traditional ASR transcription combined with high-capacity text LLMs.
Qwen2.5-Omni
Whisper + Qwen 2.5
Primary Scientific Message
Providing a strong text LLM with a transcript recovers substantial semantic reasoning performance (+5.8% accuracy), indicating that modern ALM failures stem primarily from disfluent acoustic/speaker processing rather than semantic reasoning capabilities.
High Accuracy Can Hide a Failure Mode
Standard aggregate accuracy metrics mask severe class-wise calibration imbalances. Several state-of-the-art ALMs collapse onto entailment predictions, achieving baseline accuracy while failing entirely on contradiction detection.
| Model System | Overall Acc | Macro F1 | Entailment Acc | Contradiction Acc |
|---|---|---|---|---|
| Qwen2.5-Omni | 68.1% | 0.683 | 81.2% | 55.0% |
| Kimi-Audio | 64.7% | 0.582 | 75.4% | 81.1% |
| Audio Flamingo 3 | 38.6% | 0.280 | 97.1% | 11.9% |
| SALMONN | 33.4% | 0.180 | 99.0% | 0.0% |
| LTU | 33.0% | 0.170 | 98.5% | 0.0% |
07 / Impact of Reasoning Complexity
Reasoning Difficulty Breakdown
Evaluating performance across Easy, Medium, and Hard inference demands.
One Utterance Direct Retrieval
Evidence can be identified directly from a single distinct utterance.
Local Context Integration
Requires paraphrasing and light inference across adjacent utterances.
Long-Range Integration & Speaker Filtering
Requires integrating multi-utterance narrative while filtering adult interviewer speech.
Performance changes modestly across difficulty tiers (Easy: 44.9% → Hard: 41.7%). The primary bottleneck is not a dramatic collapse on longer context, but rather that systemic class calibration biases (overpredicting entailment) persist uniformly across all difficulty tiers.
08 / Acoustic Stress Testing
Disfluency Stress Test & Density Analysis
Measuring model reasoning stability across disfluency types and acoustic disfluency density bins.
Performance Across Disfluency Density Bins
Ratio of disfluent events to total word count
Minimal disfluencies (isolated pauses)
Frequent repetitions / single prolongations
Dense repetitions and part-word blocks
Severe stuttering with multi-factor disfluencies
Diagnostic Failure Explorer
Filter case studies by error taxonomy to analyze acoustic processing breakdowns, speaker leakage, and contradiction calibration failures.
Case 01: Contradiction Calibration Failure
“The child enjoys playing sports in large noisy groups.”
Case 02: Adult Interviewer Interference Leakage
“The child travelled to Florida last summer.”
Case 03: Severe Disfluency Breakdown
“The child's favorite toy is a dinosaur.”
Case 04: Long-Range Narrative Inference
“The child owns a pet dog named Max.”
09 / Summarization Benchmark
Child-Focused Summarization Metrics
Evaluating free-form generation quality across Fluency, Faithfulness, and Purity (resisting adult interviewer prompt leakage).
“The child explains that they enjoy science experiments at school, specifically growing plants in small cups, and want to show their project to their grandmother.”
“The child likes science class experiments involving plant growth and wants to share the result with their grandmother.”
“The child enjoys science at school with plant cups and hopes to show their family.”
“The interviewer asks about science class. The child mentions planting seeds in cups and talking to their grandmother.”
Next Steps & Future Research Directions
Addressing current ALM diagnostic vulnerabilities requires joint acoustic modeling, calibration refinement, and speaker-targeted processing.
Disfluency-Aware Audio Pre-training
Incorporating clinical disfluent child speech acoustics (repetitions, prolongations, blocks) directly into self-supervised audio encoder pre-training loss functions rather than treating them as out-of-distribution noise.
Calibrated Entailment & Contrastive Alignment
Mitigating severe entailment overprediction bias through Direct Preference Optimization (DPO) and class-balanced contrastive alignment to ensure robust contradiction classification.
Target-Speaker Acoustic Extraction
Integrating frontend acoustic speaker separation and diarization directly into the audio encoder representations to prevent adult interviewer prompt leakage in mixed-speaker clinical interviews.
Long-Span Temporal Block Retention
Developing continuous temporal memory representations capable of maintaining narrative state across extended silent stuttering blocks and multi-second disfluent pauses.
10 / Summary & Conclusions
Key Research Takeaways
Three core conclusions from our diagnostic evaluation on child stuttering speech.
Useful High-Level Extraction Capabilities
Modern Audio Language Models can extract useful high-level semantic meaning directly from disfluent child speech without requiring clean intermediate text transcripts.
Acoustic & Speaker Interference Bottlenecks
Mixed adult-interviewer dialogue and disfluent acoustic inputs (repetitions, blocks) remain primary sources of failure, where cascades with strong ASR still hold an advantage.
Class-Wise Calibration Bias Hidden by Accuracy
Aggregate accuracy obscures severe calibration collapse. Models frequently overpredict entailment (achieving up to 99% entailment accuracy while scoring 0% on contradictions).
“Reasoning beyond transcription requires more than hearing the words.”
Interspeech 2026 • Audio Language Models on Child Stuttering Speech