DuoGesture

Method Overview

DuoGesture models co-speech gesture generation as the interaction of two coupled processes: a semantic stream for sparse lexical gestures and a beat stream for rhythmic prosody-aligned motion.

Highlights

DuoGesture addresses the tension between semantic expressivity and biomechanical plausibility in holistic co-speech gesture generation.

Abstract

Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures. This limits semantic grounding, speech-motion alignment, and kinematic smoothness. We propose DuoGesture, a neuro-inspired and biomechanically informed approach that decomposes co-speech gesture synthesis into semantic and beat streams. The two streams are coordinated by a Semantic Variational Information Bottleneck, a stochastic frame-level gate that learns when semantic gestures should override rhythmic beat motion. The semantic stream is controlled by Motion-Grounded Semantic Conditioning, which replaces purely linguistic word embeddings with motion-language representations to provide motion-aligned semantic priors for long-tailed lexical triggers of gestures. The beat stream is further regularised by an Inertial Beat Prior, an anthropometry-weighted arm-chain module that reduces jitter and improves rhythmic consistency without constraining semantic frames. Objective evaluations and subjective experiments show that DuoGesture outperforms strong baselines, while component ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.

Framework Overview

DuoGesture is a two-stage latent generator. Stage 1 uses a regional RVQ-VAE tokenizer. Stage 2 predicts latent gesture codes through a convex combination of semantic and beat branches controlled by a stochastic frame-level gate.

High-level dual-stream overview. DuoGesture decomposes co-speech gesture synthesis into a semantic stream and a beat stream, coupled by the Semantic Variational Information Bottleneck (S-VIB).

DuoGesture pipeline. The semantic stream is driven by motion-grounded semantic features (MGSC), while the beat stream is guided by audio, speaker identity, seed poses, and biomechanical inertial regularization (IBP). S-VIB learns a stochastic frame-level gate that dynamically fuses semantic and beat predictions.

Two-Stream Hierarchical Blender architecture. Residual VQ-VAE tokens from the semantic and beat branches are fused under S-VIB gate control before decoding.

Why Two Streams? ‐ Motivation for the IBP

Beat and semantic arm-swings have fundamentally different spectral signatures ‐ and that difference is what makes the Inertial Beat Prior both possible and necessary.

Arm-swing angular velocity analysis on 265 BEAT2 English test clips. (a) Energy decay from each clip’s spectral peak ‐ beat gestures lose energy sharply (half-BW = 0.46 Hz) while semantic gestures spread it broadly (0.89 Hz, 1.92× wider). (b) Violin plot of spectral peakedness ‐ beat gestures have a clearer dominant frequency (median 4.63 vs 3.78; 43% vs 29% show a clear peak).

Quantitative Results

On BEAT2, DuoGesture achieves the lowest Fréchet Gesture Distance (FGD) in both the one-speaker and all-speaker settings. It also leads on the two metrics that test our contributions directly: semantic-relevant gesture recall (SRGR) for MGSC, and smoothness relative to ground truth (ΔLDLJ) for IBP. On the secondary diagnostics (beat alignment, diversity, facial MSE and LVD) it remains competitive.

Ablation Studies

All ablations use the all-speaker setting. Removing MGSC costs the most (FGD 4.081 → 4.803), more than removing S-VIB or IBP. Replacing the text-to-motion encoder (TMR) with CLIP lowers semantic recall, and the anthropometric De Leva weighting is what brings motion smoothness close to ground truth: uniform smoothing over-smooths, while no prior leaves excess jitter.

User Study

We ran a perceptual study with 35-second clips from the BEAT2 test set across six narrated topics, comparing Ground Truth, DuoGesture, SemTalk and EMAGE. Thirty native English-speaking participants from the UK (balanced by self-reported gender) rated 24 videos on a five-point Likert scale for naturalness, motion diversity, and alignment with speech content and timing. The order of methods and clips was randomised for each participant.

User study results comparing Ground Truth, DuoGesture, SemTalk, and EMAGE. Ground Truth is rated highest, as expected; among the generated motions, DuoGesture receives the highest ratings on all three axes. Stars mark significant differences.

Qualitative Results

For semantic phrases such as “to get,” “I can share,” and “more drama,” DuoGesture produces clearer arm trajectories with a more visible, phrase-dependent structure. For the beat-dominant phrase “in this way,” the motion stays temporally coherent with the speech rhythm without excessive smoothing.

Qualitative comparison across semantic and beat-dominant speech contexts. SemTalk and GestureLSM tend to generate weaker or more ambiguous movements, while EMAGE often produces plausible but less semantically differentiated gestures.

Demo Videos

Featured videos from the user study, with one explicit comparison against Ground Truth.