Video Diffusion Transformers (DiTs) have been synthesizing high-quality video with high fidelity from given text descriptions involving motion. However, understanding how Video DiTs convert motion words into video remains insufficient. Furthermore, while prior studies on interpretable saliency maps primarily target objects, motion-related behavior in Video DiTs remains largely unexplored. In this paper, we investigate concrete motion features that specify when and which object moves for a given motion concept. First, to spatially localize, we introduce GramCol, which adaptively produces per-frame saliency maps for any text concept, including both motion and non-motion. Second, we propose a motion-feature selection algorithm to obtain an Interpretable Motion-Attentive Map (IMAP) that localizes motion spatially and temporally. Our method discovers concept saliency maps without the need for any gradient calculation or parameter update. Experimentally, our method shows outstanding localization capability on the motion localization task and zero-shot video semantic segmentation, providing interpretable and clearer saliency maps for both motion and non-motion concepts.
Video DiT features extracted at early timesteps are noisy and semantically intractable, making it difficult to obtain clear boundaries. Also, memorization-related features (e.g., watermarks) emerge at these early steps, making the features blurry.
Average λ2: As the average λ2 increases, the extracted features become sharper and more interpretable.
QK-Matching visualizations per text token. While QKMatching yields somewhat unclear spatial localization, its peak (red dot) still accurately pinpoints the target concept.
Visualization of Motion Localization Score (MLS) versus the separation score (Calinski-Harabasz index, CHI). MLS measured from GramCol extracted across attention heads in layersL of CogVideoX-2B. Heads with higher CHI scores tend to exhibit higher MLS, with a Pearson correlation coefficient of 0.60.
Spatiotemporal localization pipeline. This pipeline obtains a video saliency map for any concept using Video DiTs. Given a concept, we first obtain a text-surrogate token via Query–Key Matching, and then compute the GramCol to derive its spatial saliency map. For motion concepts, we additionally identify motion heads before computing GramCol, thereby improving temporal localization.
Qualitative comparisons of motion localization results using CogVideoX-5B on the MeViS dataset
Qualitative ablation results exhibiting progressive improvement in motion localizability (concept: walking).
Qualitative results of zero-shot video semantic segmentation using interpretable map methods for Video DiTs. All these saliency maps are upsampled by AnyUp.
Feature-similar but conceptually irrelevant cases (Repeated textures).
Feature-similar but conceptually irrelevant cases (Different motions).
Table 1. Motion localization results on including five metrics.
Table 2. Ablation results showing the impact of our components on the performance over five metrics.
Quantitative results of zero-shot video semantic segmentation on the VSPW dataset.