Latex Code for Diffusion Models Equations

Navigation

In this blog, we will summarize the latex code of equations for Diffusion Models, which are among the top-performing generative models, including GAN, VAE and flow-based models. The basic idea of diffusion models are to inject random noise to the feature vector in the forward process as markov chain models, and in the reverse process gradualy reconstruct the feature vector for generation. See below blogpost as reference for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. lilianweng.github.io/posts/2021-07-11-diffusion-models/.

• 1. Diffusion Model
• 1.1 Forward Process
• 1.2 Forward Process Reparameterization Trick
• 1.3 Reverse Process
• 1.4 Reverse Process Variational Lower Bound
• 1.5 Reverse Process Variational Lower Bound Loss Function

• 1.1 Forward Process

  Equation

  
  
  

  Latex Code

              q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I) \\q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1})
          

  Explanation

  See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

• 1.2 Forward Process Reparameterization Trick

  Equation

  
  
  

  Latex Code

              x_{t}=\sqrt{\alpha_{t}}x_{t-1}+\sqrt{1-\alpha_{t}} \epsilon_{t-1}\\=\sqrt{\alpha_{t}\alpha_{t-1}}x_{t-2} + \sqrt{1-\alpha_{t}\alpha_{t-1}} \bar{\epsilon}_{t-2}\\=\text{...}\\=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon \\\alpha_{t}=1-\beta_{t}, \bar{\alpha}_{t}=\prod_{t=1}^{T}\alpha_{t}
          

  Explanation

  See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

• 1.3 Reverse Process

  Equation

  
  
  

  Latex Code

              p_\theta(\mathbf{x}_{0:T}) = p(\mathbf{x}_T) \prod^T_{t=1} p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) \\
              p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t))
          

  Explanation

  See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

• 1.4 Reverse Process Variational Lower Bound

  Equation

  
  
  

  Latex Code

              \begin{aligned}
              - \log p_\theta(\mathbf{x}_0) 
              &\leq - \log p_\theta(\mathbf{x}_0) + D_\text{KL}(q(\mathbf{x}_{1:T}\vert\mathbf{x}_0) \| p_\theta(\mathbf{x}_{1:T}\vert\mathbf{x}_0) ) \\
              &= -\log p_\theta(\mathbf{x}_0) + \mathbb{E}_{\mathbf{x}_{1:T}\sim q(\mathbf{x}_{1:T} \vert \mathbf{x}_0)} \Big[ \log\frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T}) / p_\theta(\mathbf{x}_0)} \Big] \\
              &= -\log p_\theta(\mathbf{x}_0) + \mathbb{E}_q \Big[ \log\frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} + \log p_\theta(\mathbf{x}_0) \Big] \\
              &= \mathbb{E}_q \Big[ \log \frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} \Big] \\
              \text{Let }L_\text{VLB} 
              &= \mathbb{E}_{q(\mathbf{x}_{0:T})} \Big[ \log \frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} \Big] \geq - \mathbb{E}_{q(\mathbf{x}_0)} \log p_\theta(\mathbf{x}_0)
              \end{aligned}
          

  Explanation

  See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

• 1.5 Reverse Process Variational Lower Bound Loss Function

  Equation

  
  
  

  Latex Code

              \begin{aligned}
              L_\text{VLB} &= L_T + L_{T-1} + \dots + L_0 \\
              \text{where } L_T &= D_\text{KL}(q(\mathbf{x}_T \vert \mathbf{x}_0) \parallel p_\theta(\mathbf{x}_T)) \\
              L_t &= D_\text{KL}(q(\mathbf{x}_t \vert \mathbf{x}_{t+1}, \mathbf{x}_0) \parallel p_\theta(\mathbf{x}_t \vert\mathbf{x}_{t+1})) \text{ for }1 \leq t \leq T-1 \\
              L_0 &= - \log p_\theta(\mathbf{x}_0 \vert \mathbf{x}_1)
              \end{aligned}
          

  Explanation

  See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.