Equation Database

EQUATION LIST

nlp

Binary Cross Entropy Optimization BCO

#AI #nlp #llm #RLHF

$$E_{(x, y_w, y_l) \sim \mathcal{D}} [-\log \sigma \left( r_\theta (x, y_w) - r_\theta(x, y_l) \right) ] < E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \sigma (r_\theta(x, y_w))] + E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \left( 1 - \sigma (r_\theta (x, y_l)) \right)] $$ $$ E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \sigma(r_\theta(x, y_w) - \delta) - \log \sigma(- (r_\theta(x, y_l) - \delta))] $$ $$ \mathcal{L}_\text{BCO}(\theta) = - E_{(x, y) \sim \mathcal{D}^+} [\log \sigma (r_\theta (x, y) - \delta)] - E_{(x, y) \sim \mathcal{D}^-} \left[ \frac{p_\psi (f = 1 \mid x)}{p_\psi (f = 0 \mid x)} \log \sigma (- (r_\theta (x, y) - \delta)) \right] $$

READ MORE

Contrastive Preference Optimization CPO

#AI #nlp #llm #RLHF

$$\mathcal{L}(\pi_\theta;\pi_{\text{ref}}) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \log \frac{\pi_{\theta}(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_{\theta}(y_l | x)}{\pi_{\text{ref}} (y_l | x)} \Big) \Big] $$ $$ \mathcal{L}(\pi_\theta;U) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \log \pi_{\theta}(y_w | x) \nonumber \\ - \beta \log \pi_{\theta}(y_l | x) \Big) \Big] $$ $$ \min_\theta \mathcal{L}(\pi_\theta, U) \notag \text{ s.t. } \mathbb{E}_{(x,y_w) \sim \mathcal{D}}\Big [ \mathbb{KL}(\pi_w(y_w|x)||\pi_\theta(y_w|x))\Big] < \epsilon $$ $$ \min_\theta\underbrace{ \mathcal{L}(\pi_\theta, U)}_{\mathcal{L}_\text{prefer}} \underbrace{-\mathbb{E}_{(x,y_w) \sim \mathcal{D}} [\log \pi_\theta(y_w| x)]}_{\mathcal{L}_\text{NLL}} $$

READ MORE

Denoising Diffusion Policy Optimization DDPO

#AI #nlp #llm #RLHF

$$\mathcal{J}_\text{DDRL}(\theta) = \mathbb{E}_{c \sim p(c), x_{0} \sim p_{\theta} (x_{0} | c)} [r(x_{0}, c)] $$ $$ w_{\text{RWR}}(x_0, c) = \frac{1}{Z} \exp\big(\beta r(x_0, c) \big) $$ $$ w_{\text{sparse}} (x_0, c) = \mathbf{1} \big[ r(x_0, c) \geq C \big] $$ $$ \nabla_\theta \mathcal{J}_\text{DDRL} = \mathbb{E} {\; \sum_{t=0}^{T} \nabla_\theta \log p_\theta(x_{t-1} \mid x_t, c) \; r(x_0, c)} $$ $$ \nabla_\theta \mathcal{J}_\text{DDRL} = \mathbb{E} {\; \sum_{t=0}^{T} \frac{p_\theta (x_{t-1} \mid x_t, c)}{p_{\theta_\text{old}} (x_{t-1} \mid x_t, c)} \; \nabla_\theta \log p_\theta(x_{t-1} \mid x_t, c) \; r(x_0, c)} $$

READ MORE

Generalized Knowledge Distillation GKD

#AI #nlp #llm #RLHF

$$L_\mathrm{GKD}(\theta) := (1 - \lambda) \mathbb{E}_{(x, y) \sim (X, Y)} \big[ \mathcal{D}(p_{T} \| p_{S}^\theta)(y|x) \big] + \lambda \mathbb{E}_{x\sim X} \Big[\mathbb{E}_{y \sim p_{S} (\cdot|x)} \big[\mathcal{D}(p_{T} \| p_{S}^\theta)(y|x)\big]\Big] $$

READ MORE

Group Relative Policy Optimization GRPO

#AI #Machine Learning #NLP

$$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}{[q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(O|q)]} $$ $$ \frac{1}{G}\sum_{i=1}^G\frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left\{ \min \left[ \frac{\pi_\theta(o_{i,t} | q, o_{i,\lt t})}{\pi_{\theta_{old}}(o_{i,t} | q, o_{i,\lt t})} \hat{A}_{i,t}, \text{clip} \left( \frac{\pi_\theta(o_{i,t} | q, o_{i,\lt t})}{\pi_{\theta_{old}}(o_{i,t} | q, o_{i,\lt t})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_{i,t} \right] - \beta \mathbb{D}_{KL}\left[\pi_{\theta} || \pi_{ref}\right]\right\}$$

READ MORE

KTO Kahneman-Tversky Optimisation Equation

#nlp #llm #AI

$$f(\pi_\theta, \pi_\text{ref}) = \mathbb{E}_{x,y\sim\mathcal{D}}[ a_{x,y} v(r_\theta(x,y) - \mathbb{E}_{Q}[r_\theta(x, y')])] + C_\mathcal{D}$$

READ MORE

LOW RANK ADAPTATION LORA

#AI #nlp #llm #RLHF

$$W_{0} + \Delta W_{0} = W_{0} + BA, h=W_{0}x + \Delta W_{0}x = W_{0}x + BAx, \text{Initialization:} A \sim N(0, \sigma^{2}), B = 0$$

READ MORE

Odds Ratio Preference Optimization ORPO

#AI #nlp #llm #RLHF

$$\mathcal{L}_{ORPO} = \mathbb{E}_{(x, y_w, y_l)}\left[ \mathcal{L}_{SFT} + \lambda \cdot \mathcal{L}_{OR} \right] $$ $$ \mathcal{L}_{OR} = -\log \sigma \left( \log \frac{\textbf{odds}_\theta(y_w|x)}{\textbf{odds}_\theta(y_l|x)} \right) $$

READ MORE

RLHF Reinforcement Learning from Human Feedback

#AI #nlp #LLM #equation

$$p^*(y_w \succ y_l|x) = \sigma(r^*(x,y_w) - r^*(x,y_l)) $$ $$ \mathcal{L}_R(r_\phi) = \mathbb{E}_{x,y_w,y_l \sim D}[- \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))] $$ $$ \mathbb{E}_{x \in D, y \in \pi_\theta} [r_\phi(x,y)] - \beta D_{\text{KL}}(\pi_\theta(y|x) \| \pi_{\text{ref}}(y|x)) $$

READ MORE

ai

Binary Cross Entropy Optimization BCO

#AI #nlp #llm #RLHF

$$E_{(x, y_w, y_l) \sim \mathcal{D}} [-\log \sigma \left( r_\theta (x, y_w) - r_\theta(x, y_l) \right) ] < E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \sigma (r_\theta(x, y_w))] + E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \left( 1 - \sigma (r_\theta (x, y_l)) \right)] $$ $$ E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \sigma(r_\theta(x, y_w) - \delta) - \log \sigma(- (r_\theta(x, y_l) - \delta))] $$ $$ \mathcal{L}_\text{BCO}(\theta) = - E_{(x, y) \sim \mathcal{D}^+} [\log \sigma (r_\theta (x, y) - \delta)] - E_{(x, y) \sim \mathcal{D}^-} \left[ \frac{p_\psi (f = 1 \mid x)}{p_\psi (f = 0 \mid x)} \log \sigma (- (r_\theta (x, y) - \delta)) \right] $$

READ MORE

Contrastive Preference Optimization CPO

#AI #nlp #llm #RLHF

$$\mathcal{L}(\pi_\theta;\pi_{\text{ref}}) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \log \frac{\pi_{\theta}(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_{\theta}(y_l | x)}{\pi_{\text{ref}} (y_l | x)} \Big) \Big] $$ $$ \mathcal{L}(\pi_\theta;U) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \log \pi_{\theta}(y_w | x) \nonumber \\ - \beta \log \pi_{\theta}(y_l | x) \Big) \Big] $$ $$ \min_\theta \mathcal{L}(\pi_\theta, U) \notag \text{ s.t. } \mathbb{E}_{(x,y_w) \sim \mathcal{D}}\Big [ \mathbb{KL}(\pi_w(y_w|x)||\pi_\theta(y_w|x))\Big] < \epsilon $$ $$ \min_\theta\underbrace{ \mathcal{L}(\pi_\theta, U)}_{\mathcal{L}_\text{prefer}} \underbrace{-\mathbb{E}_{(x,y_w) \sim \mathcal{D}} [\log \pi_\theta(y_w| x)]}_{\mathcal{L}_\text{NLL}} $$

READ MORE

Denoising Diffusion Policy Optimization DDPO

#AI #nlp #llm #RLHF

$$\mathcal{J}_\text{DDRL}(\theta) = \mathbb{E}_{c \sim p(c), x_{0} \sim p_{\theta} (x_{0} | c)} [r(x_{0}, c)] $$ $$ w_{\text{RWR}}(x_0, c) = \frac{1}{Z} \exp\big(\beta r(x_0, c) \big) $$ $$ w_{\text{sparse}} (x_0, c) = \mathbf{1} \big[ r(x_0, c) \geq C \big] $$ $$ \nabla_\theta \mathcal{J}_\text{DDRL} = \mathbb{E} {\; \sum_{t=0}^{T} \nabla_\theta \log p_\theta(x_{t-1} \mid x_t, c) \; r(x_0, c)} $$ $$ \nabla_\theta \mathcal{J}_\text{DDRL} = \mathbb{E} {\; \sum_{t=0}^{T} \frac{p_\theta (x_{t-1} \mid x_t, c)}{p_{\theta_\text{old}} (x_{t-1} \mid x_t, c)} \; \nabla_\theta \log p_\theta(x_{t-1} \mid x_t, c) \; r(x_0, c)} $$

READ MORE

Generalized Knowledge Distillation GKD

#AI #nlp #llm #RLHF

$$L_\mathrm{GKD}(\theta) := (1 - \lambda) \mathbb{E}_{(x, y) \sim (X, Y)} \big[ \mathcal{D}(p_{T} \| p_{S}^\theta)(y|x) \big] + \lambda \mathbb{E}_{x\sim X} \Big[\mathbb{E}_{y \sim p_{S} (\cdot|x)} \big[\mathcal{D}(p_{T} \| p_{S}^\theta)(y|x)\big]\Big] $$

READ MORE

Group Relative Policy Optimization GRPO

#AI #Machine Learning #NLP

$$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}{[q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(O|q)]} $$ $$ \frac{1}{G}\sum_{i=1}^G\frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left\{ \min \left[ \frac{\pi_\theta(o_{i,t} | q, o_{i,\lt t})}{\pi_{\theta_{old}}(o_{i,t} | q, o_{i,\lt t})} \hat{A}_{i,t}, \text{clip} \left( \frac{\pi_\theta(o_{i,t} | q, o_{i,\lt t})}{\pi_{\theta_{old}}(o_{i,t} | q, o_{i,\lt t})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_{i,t} \right] - \beta \mathbb{D}_{KL}\left[\pi_{\theta} || \pi_{ref}\right]\right\}$$

READ MORE

KTO Kahneman-Tversky Optimisation Equation

#nlp #llm #AI

$$f(\pi_\theta, \pi_\text{ref}) = \mathbb{E}_{x,y\sim\mathcal{D}}[ a_{x,y} v(r_\theta(x,y) - \mathbb{E}_{Q}[r_\theta(x, y')])] + C_\mathcal{D}$$

READ MORE

LOW RANK ADAPTATION LORA

#AI #nlp #llm #RLHF

$$W_{0} + \Delta W_{0} = W_{0} + BA, h=W_{0}x + \Delta W_{0}x = W_{0}x + BAx, \text{Initialization:} A \sim N(0, \sigma^{2}), B = 0$$

READ MORE

Odds Ratio Preference Optimization ORPO

#AI #nlp #llm #RLHF

$$\mathcal{L}_{ORPO} = \mathbb{E}_{(x, y_w, y_l)}\left[ \mathcal{L}_{SFT} + \lambda \cdot \mathcal{L}_{OR} \right] $$ $$ \mathcal{L}_{OR} = -\log \sigma \left( \log \frac{\textbf{odds}_\theta(y_w|x)}{\textbf{odds}_\theta(y_l|x)} \right) $$

READ MORE

Proximal Policy Optimization PPO

#machine learning #AI #LLM

$$\arg\max\limits_{\pi}{ E_{p \sim D,g \sim \pi} [R(g|p)] }, R(g|p) = \tilde{R}_{c}(g|p) - \beta D_{KL}( \pi_{\theta} (g|p) || \pi_{0} (g|p))$$

READ MORE

RLHF Reinforcement Learning from Human Feedback

#AI #nlp #LLM #equation

$$p^*(y_w \succ y_l|x) = \sigma(r^*(x,y_w) - r^*(x,y_l)) $$ $$ \mathcal{L}_R(r_\phi) = \mathbb{E}_{x,y_w,y_l \sim D}[- \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))] $$ $$ \mathbb{E}_{x \in D, y \in \pi_\theta} [r_\phi(x,y)] - \beta D_{\text{KL}}(\pi_\theta(y|x) \| \pi_{\text{ref}}(y|x)) $$

READ MORE

machine learning

Group Relative Policy Optimization GRPO

#AI #Machine Learning #NLP

$$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}{[q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(O|q)]} $$ $$ \frac{1}{G}\sum_{i=1}^G\frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left\{ \min \left[ \frac{\pi_\theta(o_{i,t} | q, o_{i,\lt t})}{\pi_{\theta_{old}}(o_{i,t} | q, o_{i,\lt t})} \hat{A}_{i,t}, \text{clip} \left( \frac{\pi_\theta(o_{i,t} | q, o_{i,\lt t})}{\pi_{\theta_{old}}(o_{i,t} | q, o_{i,\lt t})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_{i,t} \right] - \beta \mathbb{D}_{KL}\left[\pi_{\theta} || \pi_{ref}\right]\right\}$$

READ MORE

Proximal Policy Optimization PPO

#machine learning #AI #LLM

$$\arg\max\limits_{\pi}{ E_{p \sim D,g \sim \pi} [R(g|p)] }, R(g|p) = \tilde{R}_{c}(g|p) - \beta D_{KL}( \pi_{\theta} (g|p) || \pi_{0} (g|p))$$

READ MORE