AV-Highlight: results

Joint audio-visual highlighting: given an 8-second clip and an object name, brighten the object in the video and make its sound stand out in the audio. Every number on this page is read from the evaluation JSONs when the page is built.

1 · Data process

1. Source clips. 8-s clips from filtered VGGSound and AudioSet (unbalanced). The same clip is both the audio and the video input.
2. Video target (SAM3). SAM3 segments the object named in the prompt (up to 3 instances + union), and the masks are interpolated over time. The target brightens the object's box (×2.0, skipped if the box covers under 5% of the frame) and applies a Gaussian-falloff blur outside 2.5× the box. Output is 24 fps.
3. Audio target (SAM-Audio, best of several seeds). Separation is guided by the mask when one exists (5 seeds × up to 4 masks) and by the text prompt otherwise (5 seeds). ImageBind picks the winner by the highest text–audio score, with ties broken by the lowest residual score. Target audio = peaknorm(1.5·sonority_boost(target) + 0.1·residual), 44.1 kHz mono.
4. Mask / prompt consistency fix. 23% of train+val ids had their mask and their audio built from different object names. A canonical text_prompt was restored, then SAM3 and SAM-Audio were re-run for those ids.
5. Object count (GroundingDINO). grounding-dino-base runs on 8 frames (threshold 0.35); the median detections per frame gives 0 / 1 / 2 / ≥3.
6. Highlight visual check. The highlight region is compared with the GroundingDINO boxes, giving one of: match, global (the whole frame is highlighted), mismatch, nodet, invisible, noop.
7. Cleaning (clean3). Three filters, applied in order: drop empty audio targets (target stem RMS < 0.005), then clips with more than 2 objects, then bad highlights (noop / invisible).

Model inputs: stable-audio VAE latents (64×345), CV-VAE video latents (16×48×32×32; 256², 192 frames), and per-frame Perception-Encoder features of the highlight video (192×1792). Two things are not filtered out: mismatch / global highlights, and weak or leaky audio separations (≈5%). The highlight data has no speech filter; voices are valid targets.

2 · Data distribution & statistics

Target objects per clip

Object type (regex group of the prompt)

Regex quirks, left unfixed so the numbers match the figure: “singing bowl” is counted as Bird, “fireworks” as Water / nature, and “hens” is unmapped.

Object type × count / highlight verdict (all splits)

Every nodet clip has 0 detected objects. Water / nature causes most global highlights: diffuse sources have no single box. Birds are the most common two-object type, which drives up their mismatch rate.

Top-20 prompts

The prompt distribution is long-tailed: 1,162 of 2,053 prompts appear only once. Only 56 val and 33 test prompts are unseen in train, and they are one-off paraphrases (e.g. resonant piano, snack foods, operator, high-speed motorized machine, emergency sirens, fast-paced percussive instrument), so there is no held-out object class.

3 · Main model & ablations

All three runs share the same data (0902 clean3), architecture (DiT with a Schrödinger-bridge flow, PE-FiLM conditioning, cross-attention with time-aligned RoPE), video_loss_weight 0.5, and sampler (Euler from t = 0.02, 10 steps). They differ only in the settings below; highlighted cells differ from the main model. Values are read from each run's config.yaml.

Eval set sub500: 500 random clips from the clean3 test split (seed 0). Audio: audioldm_eval against the highlight audio target; FAD from login-node VGGish with single-thread BLAS; LSD without the vocoder time offset. Video: metrics against the VAE-decoded highlight target (192 frames, 256²). Bold marks the best model in each column; the grey row is the unedited source.

Audio

Video

Paired Wilcoxon over the same 500 clips (per-clip video PSNR / LPIPS vs target). Δ = Main − other (PSNR: + favours Main; LPIPS: − favours Main).

Checkpoint steps are not matched (≈6.8k vs ≈10.9k steps). FVD ranks close arms differently from the per-clip metrics, so trust the paired tests more than FVD.

4 · Main model vs baselines (sub500)

No existing baseline edits audio and video together, so each baseline appears only in the table for the modality it edits. Video baselines get the prompt highlight {object}. VisAH is guided by the video and takes no text.

• Ditto: Wan2.1-VACE-14B with the Ditto LoRA (832×480, 73 frames, 50 steps, cfg 5). Video only.
• LGVI: language-driven video inpainting. It was trained only on removal, so it removes the object instead of highlighting it. Video only.
• VisAH: visually guided audio highlighting. Audio only.

Audio

Video (vs VAE-decoded target, n = 500)

Video vs raw highlight target (video-only baselines, n = 474)

Baselines output raw video, so this second table compares them against the raw (not VAE-decoded) highlight target. Clips whose raw target is shorter than 8 s are excluded.

5 · 20 test samples: single vs multiple objects

Metrics by object count (sub500)

All sub500 clips with 1 object (n = 380) vs 2 objects (n = 89); 0-object clips excluded. FAD is computed separately inside each group (login-node VGGish, 1-thread BLAS; smaller n → higher FAD, so compare models within a column, not single vs multi). Video = mean per-clip PSNR / LPIPS against the VAE-decoded target.

Samples

20 sub500 test clips whose highlight target passed the visual check (match): 10 with one object and 10 with two, drawn at random (seed 0) round-robin over object types. They are not cherry-picked. Each cell plays video with audio. Video-only baselines play with the source audio; VisAH plays with the source video. Captions give that clip's video PSNR / LPIPS against the target.

Single object (object_count = 1)

Multiple objects (object_count = 2)

Caveats

• FAD comes only from VGGish run with single-thread BLAS. Multi-threaded BLAS makes sqrtm unstable and can produce negative FAD.
• Every table here is on sub500 with LSD offset 0; numbers are not comparable with the older test-200 page (LSD offset 1120).
• IS ≈ 1.01 for every arm and barely separates them.
• No model beats pass-through on audio FAD: most targets differ only slightly from the source mix. The model's clear gains are on video.
• Each configuration is a single training run; run-to-run variance is unknown.