Apple's StereoFoley Breakthrough: Generating Object-Aware Stereo Audio from Video

Apple Machine Learning Research has introduced StereoFoley, a groundbreaking video — to-audio generation framework. This innovative system, presented at the ICASSP conference in April 2026, is designed to produce semantically aligned, temporally synchronized, and spatially accurate stereo sound directly from video content, operating at a high-fidelity 48 kHz sample rate. Unveiled as a significant advancement, StereoFoley addresses longstanding limitations in artificial intelligence's ability to create rich, immersive audio experiences that truly match visual scenes.

Historically, generative video — to-audio models have shown considerable promise in achieving strong semantic accuracy and temporal synchronization, meaning the generated audio correctly corresponds to the events depicted in a video and aligns with their timing. However, a critical gap persisted: these systems were largely confined to generating monaural audio or struggled to produce truly object — aware stereo imaging. This limitation significantly hampered their capacity to create immersive and realistic soundscapes where individual objects emit distinct, spatially localized sounds.

To overcome these challenges, the StereoFoley development involved a multi — stage approach. The initial step focused on developing and meticulously training a robust base model. This foundational model was engineered to generate stereo audio directly from video input, and critically, it achieved state — of-the-art performance in both semantic accuracy and temporal synchronization, building upon the strengths of prior research while adding the crucial stereo dimension. Recognizing that the scarcity of real-world, high-quality spatial audio datasets would impede further progress, the research team then innovated by introducing a novel synthetic data generation pipeline.

The synthetic data generation pipeline is a cornerstone of StereoFoley's innovation. It meticulously combines several advanced techniques: detailed video analysis to understand the scene, precise object tracking to follow individual elements as they move within the frame, and sophisticated audio synthesis. Crucially, this synthesis incorporates dynamic panning, which simulates how sound sources move across the stereo field, and distance — based loudness controls, which adjust audio volume based on an object's perceived proximity to the listener.

Following the generation of this rich synthetic dataset, the base StereoFoley model underwent a critical fine-tuning phase. Training the model on this specialized data resulted in a clear and robust object — audio correspondence, meaning the generated sounds were accurately linked to specific objects and their movements within the video. Acknowledging the nascent nature of object — aware stereo audio generation, the researchers also took the proactive step of introducing new, dedicated stereo object — awareness measures. These novel metrics allowed for objective evaluation of the model's performance in accurately localizing and representing sounds in a stereo field.

StereoFoley marks a significant milestone in generative AI, establishing the first comprehensive, end-to-end framework specifically designed for stereo object — aware video — to-audio generation. This achievement addresses a critical gap that has long limited the realism and immersion potential of video — generated audio, setting a new benchmark for the field. The research was led by a team of distinguished researchers including Tornike Karchkhadze (affiliated with UC San Diego and having contributed while at Apple), Kuan — Lin Chen, Mojtaba Heydari, Robert Henzel, Alessandro Toso, Mehrez Souden, and Joshua Atkins from Apple Machine Learning Research. Their collective efforts have pushed the boundaries of what is possible in automated audio creation for visual media.

This work by Apple Machine Learning Research fits into a broader ongoing effort to advance spatial audio and machine perception of sound environments. Related initiatives include ImmerseDiffusion, a generative spatial audio latent diffusion model unveiled in February 2025, also at ICASSP, which focuses on producing 3D immersive soundscapes by generating first — order ambisonics (FOA) audio. Another related area of research, "Learning Spatially — Aware Language and Audio Embeddings" from December 2024 (NeurIPS), explores how machines can interpret natural language descriptions to understand and reconstruct acoustic environments, combining semantic and spatial attributes.