Latex Code for Diffusion Models Equations
Navigation
In this blog, we will summarize the latex code of equations for Diffusion Models, which are among the top-performing generative models, including GAN, VAE and flow-based models. The basic idea of diffusion models are to inject random noise to the feature vector in the forward process as markov chain models, and in the reverse process gradualy reconstruct the feature vector for generation. See below blogpost as reference for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. lilianweng.github.io/posts/2021-07-11-diffusion-models/.
- 1. Diffusion Model
- 1.1 Forward Process
- 1.2 Forward Process Reparameterization Trick
- 1.3 Reverse Process
- 1.4 Reverse Process Variational Lower Bound
- 1.5 Reverse Process Variational Lower Bound Loss Function
1. Diffusion Model
-
1.1 Forward Process
Equation
Latex Code
q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I) \\q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1})
Explanation
See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
-
1.2 Forward Process Reparameterization Trick
Equation
Latex Code
x_{t}=\sqrt{\alpha_{t}}x_{t-1}+\sqrt{1-\alpha_{t}} \epsilon_{t-1}\\=\sqrt{\alpha_{t}\alpha_{t-1}}x_{t-2} + \sqrt{1-\alpha_{t}\alpha_{t-1}} \bar{\epsilon}_{t-2}\\=\text{...}\\=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon \\\alpha_{t}=1-\beta_{t}, \bar{\alpha}_{t}=\prod_{t=1}^{T}\alpha_{t}
Explanation
See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
-
1.3 Reverse Process
Equation
Latex Code
p_\theta(\mathbf{x}_{0:T}) = p(\mathbf{x}_T) \prod^T_{t=1} p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) \\ p_\theta(\mathbf{x}_{t-1} \vert \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \boldsymbol{\Sigma}_\theta(\mathbf{x}_t, t))
Explanation
See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
-
1.4 Reverse Process Variational Lower Bound
Equation
Latex Code
\begin{aligned} - \log p_\theta(\mathbf{x}_0) &\leq - \log p_\theta(\mathbf{x}_0) + D_\text{KL}(q(\mathbf{x}_{1:T}\vert\mathbf{x}_0) \| p_\theta(\mathbf{x}_{1:T}\vert\mathbf{x}_0) ) \\ &= -\log p_\theta(\mathbf{x}_0) + \mathbb{E}_{\mathbf{x}_{1:T}\sim q(\mathbf{x}_{1:T} \vert \mathbf{x}_0)} \Big[ \log\frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T}) / p_\theta(\mathbf{x}_0)} \Big] \\ &= -\log p_\theta(\mathbf{x}_0) + \mathbb{E}_q \Big[ \log\frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} + \log p_\theta(\mathbf{x}_0) \Big] \\ &= \mathbb{E}_q \Big[ \log \frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} \Big] \\ \text{Let }L_\text{VLB} &= \mathbb{E}_{q(\mathbf{x}_{0:T})} \Big[ \log \frac{q(\mathbf{x}_{1:T}\vert\mathbf{x}_0)}{p_\theta(\mathbf{x}_{0:T})} \Big] \geq - \mathbb{E}_{q(\mathbf{x}_0)} \log p_\theta(\mathbf{x}_0) \end{aligned}
Explanation
See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
-
1.5 Reverse Process Variational Lower Bound Loss Function
Equation
Latex Code
\begin{aligned} L_\text{VLB} &= L_T + L_{T-1} + \dots + L_0 \\ \text{where } L_T &= D_\text{KL}(q(\mathbf{x}_T \vert \mathbf{x}_0) \parallel p_\theta(\mathbf{x}_T)) \\ L_t &= D_\text{KL}(q(\mathbf{x}_t \vert \mathbf{x}_{t+1}, \mathbf{x}_0) \parallel p_\theta(\mathbf{x}_t \vert\mathbf{x}_{t+1})) \text{ for }1 \leq t \leq T-1 \\ L_0 &= - \log p_\theta(\mathbf{x}_0 \vert \mathbf{x}_1) \end{aligned}
Explanation
See blog for more details: Weng, Lilian. (Jul 2021). What are diffusion models? Lilâ??Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
Comments
-
It's enormous that you are getting ideas from this piece of writing as well as from our discussion made here.
Il est important de lire attentivement les termes et conditions associés à ces bonus pour éviter toute déception. Also visit my website Dane (https://www.salon.com/crypto/gambling/best/bitcoin-casinos)
Great looking site. Assume you did a great deal of your very own coding. My blog post - web page (https://www.salon.com/crypto/gambling/best/bitcoin-casinos)
Reply