NVIDIA-NeMo/Speech
NVIDIA NEMO SPEECH — FRAMEWORK FOR SPEECH AI AND MULTIMODAL LLMS
Scalable framework for ASR, TTS, and speech LLMs. Enables creating, customizing, and deploying speech models with ready checkpoints.
// KEY FEATURES
- Nemotron 3.5 Transcribe: offline LLM for 40 languages up to 1 hour audio
- Streaming ASR with controllable latency 80 ms–1 s on FastConformer
- MagpieTTS v2607: 12 languages, Nemotron 3 VoiceChat — full-duplex dialogue
- Runs on top of your Python/PyTorch/CUDA without replacing the stack
// DEEP DIVE: HOW IT WORKS
NeMo Speech 3.0 is a PyTorch framework for training and inference of speech models: ASR, TTS, speaker diarization/recognition, and speech LLMs. Architecture is modular: YAML configs, module classes, distributed training support via PyTorch Lightning. Checkpoints distributed via Hugging Face and NGC containers.
Key feature — Nemotron family: unified checkpoints for offline/streaming modes with latency control (Nemotron-Speech-Streaming, Parakeet-unified). MagpieTTS covers 12 languages. VoiceChat implements full-duplex on Nemotron Nano v2 backbone. Production-ready NIM microservices available.
Requires Python 3.12+, PyTorch 2.7+, NVIDIA GPU with CUDA for training (recommended for inference). Installs on top of existing environment without replacing PyTorch. Tested builds: Python 3.13 + PyTorch 2.11/CUDA 12.9 or 2.12/CUDA 13.2. Repository since 2026 focuses only on audio/speech/multimodality.
// USE CASES
Best for: Researchers and developers of Speech AI, ASR/TTS, multimodal LLMs
- Train and deploy streaming ASR with sub-100 ms latency for call center
- Generate speech in 12 languages via MagpieTTS for multilingual product
- Run offline transcription of hour-long audio with speaker attribution and translation
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram