Multilingual Sign Language Translation with Unified Datasets and Pose-Based Transformers

Pedro Dal Bianco, Oscar Stanchi, Facundo Quiroga, Franco Ronchetti

Workshop on Sign Language Processing (WSLP), IJCNLP-AACL 2025

Abstract

Sign languages are highly diverse across countries and regions, yet most Sign Language Translation (SLT) work remains monolingual. We explore a unified, multi-target SLT model trained jointly on four sign languages (German, Greek, Argentinian, Indian) using a standardized data layer. Our model operates on pose keypoints extracted with MediaPipe, yielding a lightweight and dataset-agnostic representation that is less sensitive to backgrounds, clothing, cameras, or signer identity while retaining motion and configuration cues. On RWTH-PHOENIX-Weather 2014T, Greek Sign Language Dataset, LSA-T, and ISLTranslate, naive joint training under a fully shared parameterization performs worse than monolingual baselines; however, a simple two-stage schedule: multilingual pre-training followed by a short language-specific fine-tuning, recovers and surpasses monolingual results on three datasets (PHOENIX14T: +0.15 BLEU-4; GSL: +0.74; ISL: +0.10) while narrowing the gap on the most challenging corpus (LSA-T: −0.24 vs. monolingual). Scores span from BLEU-4 ≈ 1 on open-domain news (LSA-T) to > 90 on constrained curricula (GSL), highlighting the role of dataset complexity. We release our code to facilitate training and evaluation of multilingual SLT models.

Citation

@inproceedings{dalbianco2025multilingual,
  title={Multilingual Sign Language Translation with Unified Datasets and Pose-Based Transformers},
  author={Dal Bianco, Pedro and Stanchi, Oscar and Quiroga, Facundo and Ronchetti, Franco},
  booktitle={Proceedings of the Workshop on Sign Language Processing (WSLP)},
  year={2025},
  publisher={Association for Computational Linguistics},
  doi={10.18653/v1/2025.wslp-main.5}
}