Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search

Abstract

• aligment problem?
• this paper proposes diffusion latent beam search with estimator
  • which can select a better diffusion latent to maximize a given alignment reward at inference time.
• This paper point out that improving video quality with respect to alignment to prompts requires reward calibration by weighting existing metrics.

1 Introduction

• Recently, it has attracted a lot of attetion to 1) steering the output of diffusion models , 2) quantifying the goodness of the content.
  • which is studied as an alignment problem.
• This paper proposes Diffusion Latent Beam Search (DLBS) with lookahead estimator
  • which can select a better diffusion latent to maximize a given aligment reward

• This paper also point out that
  • the improvement of perceptual video quaility, considering the alignment to prompt, requires reward calibration of existing method.
  • Because many previous metric for quantifying video naturalness do not always correct between vision language and human raters
• We demonstrate that DLBS can induce high-quality outputs based on the calibrated reward

2 Preliminaries.

Latent Diffusion Models

1. forward process

$$
p_0(\mathbf{z})
\longrightarrow
p_t(\mathbf{z})
\longrightarrow
p_T(\mathbf{z}) \approx \mathcal{N}(0,I).
$$

• $p_0(\mathbf{z})$: clean latent data distrubution
• $p_t(\mathbf{z})$: partially noised distribution
• $p_T(\mathbf{z})$: nearly Gaussian-noise distribution

$$
z_0 = \operatorname{Enc}(x_0)
$$

$$
d\mathbf{z}=\mathbf{f}(\mathbf{z}, t),dt+g(t),d\mathbf{w},\quad\text{where }\mathbf{z}_0 \sim p_0(\mathbf{z}).
$$

$$
\underbrace{z_{t+1}-z_t}_{dz} = \underbrace{f(z_t,t)\Delta t}_{\text{drift}} + \underbrace{g(t)\sqrt{\Delta t}\,\epsilon}_{\text{Gaussian noise}}, \qquad \epsilon\sim\mathcal{N}(0,I)
$$

• Expression that I’m familiar

$$
z_t = \sqrt{\alpha_t},z_{t-1} + \sqrt{1-\alpha_t},\epsilon.
$$

• $\sqrt{\alpha_t},z_{t-1}$ : slightly weakens the existing latent signal → drift-like deterministic change
• $\sqrt{1-\alpha_t},\epsilon$ : adds Gaussian noise → random change

2. Reverse diffusion process

$$ p_T(\mathbf{z}) \approx \mathcal{N}(0,I) \longrightarrow p_t(\mathbf{z}) \longrightarrow p_0(\mathbf{z}) $$

• The reverse process starts from Gaussian noise and repeatedly uses the learned score to move the latent toward the prompt-conditioned video distribution.

$$
d\mathbf{z}=\left[\mathbf{f}(\mathbf{z},t)-g(t)^2 \nabla_{\mathbf{z}} \log p_t(\mathbf{z})\right]dt+g(t),d\mathbf{w},
$$

Original drift

$$
\mathbf{f}(\mathbf{z},t)
$$

Diffusion coefficient

$$
g(t)
$$

$g(t)$ is the diffusion coefficient, which determines the noise scale at diffusion time $t$.

Score function

$$
\nabla_{\mathbf{z}}\log p_t(\mathbf{z})
$$

This function tell the current noisy latent which direction lead more realistic video latents.

$$
z_t\xrightarrow{\nabla_{\mathbf{z}} \log p_t(\mathbf{z})}\text{high-probability region}
$$

• Why does $z_t$ have to move toward a high-probability region ?
  • Because high-density regions represent latent patterns that are more typical of the model’s traning data at time $t$.

Score-based correction

$$
-g(t)^2 \nabla_{\mathbf{z}} \log p_t(\mathbf{z})
$$

Reverse-time noise

$$
g(t),d\bar{\mathbf{w}}
$$

 

 

---

Reference

https://arxiv.org/abs/2501.19252

Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search