Ai! Jack!v2026.04.01-f12
← 홈HF Daily Papers

The Attention Triangle in Audio-Video Models

원문 보기
이스라엘

AI 분석

•기여: 오디오·비디오 diffusion 모델의 의미 누출을 텍스트·오디오·비디오 사이의 attention 삼각형으로 분석하고, 특정 경로의 편향을 겨냥한 추론 시점 개입 제안
•방법: 세 교차 attention 경로에서 의미 전달을 추적하고, attention 기반 신호로 누출을 진단·유도하여 개별 상호작용의 역할을 분리
•결과: 오디오와 비디오가 양방향으로 서로 영향을 주며, 학습된 사전 경향과 프롬프트가 충돌할 때 해당 경로가 의도한 조건을 덮어쓸 수 있음을 관찰. 생성 품질을 유지하면서 의미적 정합성을 개선했다고 보고
•한계: 초록에 정량적 개선 폭과 구체적인 개입 비용이 없어 효과의 크기와 실용적 부담 판단에 한계

검증 정보

출처
huggingface.co
피드
HF Daily Papers
수집/분석 기준
2026. 9. 8. 오전 9:00
원문 확인
원문 열기

본문

Audio-video diffusion models rely on cross-modal attention to coordinate text, sound, and visual content, yet this same mechanism can introduce subtle and systematic semantic leakage. We study these models by probing and analyzing the ``attention triangle,'' comprising the three cross-attention edges connecting the text, audio, and video streams, and examine how semantic information is routed across modalities during generation. Our analysis reveals that routing along the audio-video edge is bidirectional: audio can influence video generation, while video can influence audio generation. This edge is shaped by biases encoded in the model's parameters and emerges as a major contributor to leakage: when prompts are in tension with learned priors, cross-modal interactions may override the intended conditioning and reroute semantics toward visually canonical but incorrect outcomes. These effects suggest that semantic artifacts arise not merely from attention spreading beyond its intended target, but from structured, bias-driven interactions along specific pathways. Building on this perspective, we extract attention-derived signals that expose how semantics are distributed and grounded across modalities, and use them as a diagnostic tool to both analyze and deliberately incur leakage under controlled conditions. This enables us to probe the internal dynamics of cross-modal routing and isolate the role of individual interactions. We further leverage these signals to guide inference-time interventions that encourage more consistent cross-modal alignment. Extensive experiments support our analysis and demonstrate improved semantic grounding while preserving generation quality.

오디오·비디오 모델의 Attention 삼각형 — Ai! Jack! — Ai! Jack!