Ilpo Viertola$^1$, Vladimir Iashin$^1$, Sophie Tötterström$^1$, Esa Rahtu$^1$
$^1$Tampere University
Preprint
ArXiv link coming….
Code
https://github.com/ilpoviertola/EASE
Audio-Visual Semantic Segmentation ( AVSS) aims to identify, segment, and classify sound-emitting objects in video frames. Previous Transformer-based AVSS approaches largely inherit design principles from image segmentation models. Recent studies show that these image segmentation models contain redundant components that contribute little to the segmentation performance. Following this insight, we propose Encoder-only Audio-Visual Segmentation (EASE). EASE runs at up to 365 FPS, 3× faster than prior State-of-the-Art (SotA) AVS models at comparable accuracy, and trains in under 22 GPU-hours. Furthermore, we achieve SotA AVSS performance across different backbones and input resolutions. Our results demonstrate that AVSS can be both simpler and faster, providing a scalable foundation for future research and real-time applications.
Figure 1. Overview of Encoder-only Audio-Visual Segmentation (EASE) model.
EASE is a simplified approach for Audio-Visual Segmentation (AVS) built on top of a plain ViT architecture making it fully compatible with modern VFMs, unlike traditional Transformer-based AVS models. It also introduces new components: a Gumbel-Softmax-based audio feature enhancement module, a mid-backbone audio query injection strategy, and a bidirectional audio-visual fusion module with learned soft clustering.
Table 1. Quantitative comparison on AVSS [1] with efficiency evaluations. We compare performance on AVSS and throughput between EASE and SotA approaches with open-source code. †Retrained on the standard data split and benchmark metrics. ‡DDESeg uses HTSAT (29M) as its audio encoder, while all other methods use VGGish (72M). Parameter counts are reported inclusive of audio encoders.
| Method | Backbone | Params | Inp. | mIoU | F-Score | FPS | GFLOPs |
|---|---|---|---|---|---|---|---|
| AVSegFormer [1] | PVTv2 | 186M | $224^2$ | 36.7 | 42.0 | 27 | 99 |
| AVSegFormer [1] | PVTv2 | 186M | $512^2$ | 37.3 | 42.8 | 23 | 504 |
| AAVS [7] | Swin-B | 187M | $384^2$ | 48.5 | 53.2 | 69 | 151 |
| SelM [3] | Swin-B | 186M | $448^2$ | 41.3 | 46.9 | 71 | 258 |
| COMBO [6] | PVTv2 | 499M | $224^2$ | 42.1 | 46.1 | 73 | 346 |
| DDESeg† [4] | Transf. | 133M | $224^2$ | 43.3 | 48.7 | 27 | 179 |
| EASE | PVTv2 | 147M | $224^2$ | 42.3 | 47.6 | 159 | 40 |
| EASE | PVTv2 | 147M | $384^2$ | 46.1 | 50.6 | 141 | 105 |
| EASE | ViT-B | 187M | $224^2$ | 45.8 | 50.7 | 365 | 96 |
| EASE | ViT-B | 187M | $384^2$ | 49.8 | 54.1 | 181 | 291 |
| EASE | ViT-L | 424M | $224^2$ | 52.2 | 57.2 | 225 | 227 |
| EASE | ViT-L | 424M | $384^2$ | 56.5 | 61.0 | 86 | 721 |
We show more qualitative comparisons on the AVSS task between ground truth semantic segmentation masks, and segmentation predictions on AVSegFormer [1] with PvTv2 [2] and $224 \times 224$ input size, SelM [3] with PvTv2 [2] and $448 \times 448$ input size, DDESeg [4] with Transformer backbone and $224 \times 224$ input size, and our proposed method EASE with ViT-L [5] and $224 \times 224$ input size.
