PyTorch consolidates media libraries into TorchCodec

PyTorch has restructured its media processing ecosystem by centralizing all decoding and encoding capabilities into a single library called TorchCodec. Previously, these tasks were scattered across TorchVision and TorchAudio, which relied on fragmented backends like PyAV, C++ FFmpeg, CUDA/NVCUVID, libsoundfile, and libsox. Under the new architecture, TorchCodec serves as the unified entry point for converting images, video, and audio into tensors, and vice versa.

With media input and output consolidated, TorchVision and TorchAudio have narrowed their scope to focus exclusively on transforming media tensors. Legacy components like models, datasets, and pipelines are no longer under active development, as the PyTorch team points to external alternatives like Hugging Face. This shift simplifies the libraries, though the transition required deprecating several APIs. However, popular tools like TorchAudio's MelSpectrogram and TorchVision's v2 transforms remain core features.

Centralizing media I/O in TorchCodec resolves significant maintenance challenges. Managing complex dependencies—including six major FFmpeg versions (4 through 9), NVIDIA's GPU codec SDK, and format-specific libraries like libjpeg and libpng—is now isolated to TorchCodec. This isolation allows developers to deliver targeted performance optimizations, particularly for CUDA video decoding, while keeping TorchVision and TorchAudio lightweight.

For developers, this reorganization means cleaner workflows and improved stability. Codecs like VideoDecoder, AudioDecoder, JpegEncoder, and PngEncoder now handle the initial tensor conversion, which then feeds directly into downstream transforms. Furthermore, TorchCodec, TorchVision, and TorchAudio are now ABI stable. They are no longer tied to specific PyTorch releases, meaning practitioners do not need to rebuild these libraries when upgrading the core PyTorch framework.