Denoising Diffusion Policy Optimization DDPO

Tags: #AI #nlp #llm #RLHF

Equation

$${\mathcal{J}}_{\text{DDRL}}(\theta )={\mathbb{E}}_{c\sim p(c),x_0\sim p_{\theta }(x_0|c)}[r(x_0,c)]$$ $$w_{\text{RWR}}(x_0,c)=\frac{1}{Z}\exp (\beta r(x_0,c))$$ $$w_{\text{sparse}}(x_0,c)=\mathbf{1}[r(x_0,c)\ge C]$$ $${\mathrm{\nabla }}_{\theta }{\mathcal{J}}_{\text{DDRL}}=\mathbb{E}\sum _{t=0}^T{\mathrm{\nabla }}_{\theta }\log p_{\theta }(x_{t-1}\mid x_t,c)r(x_0,c)$$ $${\mathrm{\nabla }}_{\theta }{\mathcal{J}}_{\text{DDRL}}=\mathbb{E}\sum _{t=0}^T\frac{p_{\theta }(x_{t-1}\mid x_t,c)}{p_{{\theta }_{\text{old}}}(x_{t-1}\mid x_t,c)}{\mathrm{\nabla }}_{\theta }\log p_{\theta }(x_{t-1}\mid x_t,c)r(x_0,c)$$

Latex Code

Have Fun

Let's Vote for the Most Difficult Equation!

Introduction

REINFORCEMENT LEARNING TRAINING OF DIFFUSION MODELS. The denoising diffusion RL objective is to maximize a reward signal r defined on the samples and contexts for some context distribution p(c) of our choosing.
paper: TRAINING DIFFUSION MODELS WITH REINFORCEMENT LEARNING
huggingface: DDPO Trainer