Equation Database

EQUATION LIST

nlp

BLEU Bilingual Evaluation Understudy

#nlp #BLEU #evaluation

$$ \text{BLEU}_{w}(\hat{S},S)=BP(\hat{S};S) \times \exp{\sum^{\infty}_{n=1}w_{n} \ln p_{n}(\hat{S};S)}, p_{n}(\hat{S};S)=\frac{\sum^{M}_{i=1}\sum_{s \in G_{n}(\hat{y})} \min(C(s,\hat{y}),\max_{y \in S_{i}} C(s,y))}{\sum^{M}_{i=1}\sum_{s \in G_{n}(\hat{y})}C(s, \hat{y})}, p_{n}(\hat{y};y)=\frac{\sum_{s \in G_{n}(\hat{y})} \min(C(s,\hat{y}), C(s,y))}{\sum_{s \in G_{n}(\hat{y})}C(s, \hat{y})}, BP(\hat{S};S) = e^{-(r/c-1)^{+}}$$

READ MORE

Binary Cross Entropy Optimization BCO

#AI #nlp #llm #RLHF

$$E_{(x, y_w, y_l) \sim \mathcal{D}} [-\log \sigma \left( r_\theta (x, y_w) - r_\theta(x, y_l) \right) ] < E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \sigma (r_\theta(x, y_w))] + E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \left( 1 - \sigma (r_\theta (x, y_l)) \right)] $$ $$ E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \sigma(r_\theta(x, y_w) - \delta) - \log \sigma(- (r_\theta(x, y_l) - \delta))] $$ $$ \mathcal{L}_\text{BCO}(\theta) = - E_{(x, y) \sim \mathcal{D}^+} [\log \sigma (r_\theta (x, y) - \delta)] - E_{(x, y) \sim \mathcal{D}^-} \left[ \frac{p_\psi (f = 1 \mid x)}{p_\psi (f = 0 \mid x)} \log \sigma (- (r_\theta (x, y) - \delta)) \right] $$

READ MORE

Conditional Random Field CRF

#machine learning #nlp

$$P(y|x)=\frac{1}{Z(x)}\exp(\sum_{i,k}\lambda_{k}t_{k}(y_{i-1},y_{i},x,i))+\sum_{i,l}\mu_{l}s_{l}(y_{i},x,i)) \\ Z(x)=\sum_{y}\exp(\sum_{i,k}\lambda_{k}t_{k}(y_{i-1},y_{i},x,i))+\sum_{i,l}\mu_{l}s_{l}(y_{i},x,i))$$

READ MORE

Contrastive Preference Optimization CPO

#AI #nlp #llm #RLHF

$$\mathcal{L}(\pi_\theta;\pi_{\text{ref}}) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \log \frac{\pi_{\theta}(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_{\theta}(y_l | x)}{\pi_{\text{ref}} (y_l | x)} \Big) \Big] $$ $$ \mathcal{L}(\pi_\theta;U) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \log \pi_{\theta}(y_w | x) \nonumber \\ - \beta \log \pi_{\theta}(y_l | x) \Big) \Big] $$ $$ \min_\theta \mathcal{L}(\pi_\theta, U) \notag \text{ s.t. } \mathbb{E}_{(x,y_w) \sim \mathcal{D}}\Big [ \mathbb{KL}(\pi_w(y_w|x)||\pi_\theta(y_w|x))\Big] < \epsilon $$ $$ \min_\theta\underbrace{ \mathcal{L}(\pi_\theta, U)}_{\mathcal{L}_\text{prefer}} \underbrace{-\mathbb{E}_{(x,y_w) \sim \mathcal{D}} [\log \pi_\theta(y_w| x)]}_{\mathcal{L}_\text{NLL}} $$

READ MORE

Denoising Diffusion Policy Optimization DDPO

#AI #nlp #llm #RLHF

$$\mathcal{J}_\text{DDRL}(\theta) = \mathbb{E}_{c \sim p(c), x_{0} \sim p_{\theta} (x_{0} | c)} [r(x_{0}, c)] $$ $$ w_{\text{RWR}}(x_0, c) = \frac{1}{Z} \exp\big(\beta r(x_0, c) \big) $$ $$ w_{\text{sparse}} (x_0, c) = \mathbf{1} \big[ r(x_0, c) \geq C \big] $$ $$ \nabla_\theta \mathcal{J}_\text{DDRL} = \mathbb{E} {\; \sum_{t=0}^{T} \nabla_\theta \log p_\theta(x_{t-1} \mid x_t, c) \; r(x_0, c)} $$ $$ \nabla_\theta \mathcal{J}_\text{DDRL} = \mathbb{E} {\; \sum_{t=0}^{T} \frac{p_\theta (x_{t-1} \mid x_t, c)}{p_{\theta_\text{old}} (x_{t-1} \mid x_t, c)} \; \nabla_\theta \log p_\theta(x_{t-1} \mid x_t, c) \; r(x_0, c)} $$

READ MORE

Direct Policy Optimization DPO

#nlp #llm #RLHF

$$\pi_{r} (y|x) = \frac{1}{Z(x)} \pi_{ref} (y|x) \exp(\frac{1}{\beta} r(x,y) ) , r(x,y) = \beta \log \frac{\pi_{r} (y|x)}{\pi_{ref} (y|x)} + \beta \log Z(x) , p^{*}(y_{1} > y_{2} |x) = \frac{1}{1+\exp{(\beta \frac{\pi^{*} (y_{2}|x)}{\pi_{ref} (y_{2}|x)} - \beta \frac{\pi^{*} (y_{1}|x)}{\pi_{ref} (y_{1}|x)} )}} , \mathcal{L}_{DPO}(\pi_{\theta};\pi_{ref}) = -\mathbb{E}_{(x, y_{w},y_{l}) \sim D } [\log \sigma (\beta \frac{\pi_{\theta} (y_{w}|x)}{\pi_{ref} (y_{w}|x)} - \beta \frac{\pi_{\theta} (y_{l}|x)}{\pi_{ref} (y_{l}|x)} )] , \nabla \mathcal{L}_{DPO}(\pi_{\theta};\pi_{ref}) = - \beta \mathbb{E}_{(x, y_{w},y_{l}) \sim D } [ \sigma ( \hat{r}_{\theta} (x, y_{l}) - \hat{r}_{\theta} (x, y_{w})) [\nabla_{\theta} \log \pi (y_{w}|x) - \nabla_{\theta} \log \pi (y_{l}|x) ] ] , \hat{r}_{\theta} (x, y) = \beta \log (\frac{\pi_{\theta} (y|x)}{\pi_{ref} (y|x)})$$

READ MORE

Direct Preference Optimization DPO

#nlp #llm #RLHF

$$\pi_{r} (y|x) = \frac{1}{Z(x)} \pi_{ref} (y|x) \exp(\frac{1}{\beta} r(x,y) ) $$ $$ r(x,y) = \beta \log \frac{\pi_{r} (y|x)}{\pi_{ref} (y|x)} + \beta \log Z(x) $$ $$ p^{*}(y_{1} > y_{2} |x) = \frac{1}{1+\exp{(\beta \frac{\pi^{*} (y_{2}|x)}{\pi_{ref} (y_{2}|x)} - \beta \frac{\pi^{*} (y_{1}|x)}{\pi_{ref} (y_{1}|x)} )}} $$ $$ \mathcal{L}_{DPO}(\pi_{\theta};\pi_{ref}) = -\mathbb{E}_{(x, y_{w},y_{l}) \sim D } [\log \sigma (\beta \frac{\pi_{\theta} (y_{w}|x)}{\pi_{ref} (y_{w}|x)} - \beta \frac{\pi_{\theta} (y_{l}|x)}{\pi_{ref} (y_{l}|x)} )] $$ $$ \nabla \mathcal{L}_{DPO}(\pi_{\theta};\pi_{ref}) = - \beta \mathbb{E}_{(x, y_{w},y_{l}) \sim D } [ \sigma ( \hat{r}_{\theta} (x, y_{l}) - \hat{r}_{\theta} (x, y_{w})) [\nabla_{\theta} \log \pi (y_{w}|x) - \nabla_{\theta} \log \pi (y_{l}|x) ] ] $$ $$ \hat{r}_{\theta} (x, y) = \beta \log (\frac{\pi_{\theta} (y|x)}{\pi_{ref} (y|x)})$$

READ MORE

Generalized Knowledge Distillation GKD

#AI #nlp #llm #RLHF

$$L_\mathrm{GKD}(\theta) := (1 - \lambda) \mathbb{E}_{(x, y) \sim (X, Y)} \big[ \mathcal{D}(p_{T} \| p_{S}^\theta)(y|x) \big] + \lambda \mathbb{E}_{x\sim X} \Big[\mathbb{E}_{y \sim p_{S} (\cdot|x)} \big[\mathcal{D}(p_{T} \| p_{S}^\theta)(y|x)\big]\Big] $$

READ MORE

Hidden Markov Model

#machine learning #nlp

$$Q=\{q_{1},q_{2},...,q_{N}\}, V=\{v_{1},v_{2},...,v_{M}\} \\ I=\{i_{1},i_{2},...,i_{T}\},O=\{o_{1},o_{2},...,o_{T}\} \\ A=[a_{ij}]_{N \times N}, a_{ij}=P(i_{t+1}=q_{j}|i_{t}=q_{i}) \\ B=[b_{j}(k)]_{N \times M},b_{j}(k)=P(o_{t}=v_{k}|i_{t}=q_{j})$$

READ MORE

KTO Kahneman-Tversky Optimisation Equation

#nlp #llm #AI

$$f(\pi_\theta, \pi_\text{ref}) = \mathbb{E}_{x,y\sim\mathcal{D}}[ a_{x,y} v(r_\theta(x,y) - \mathbb{E}_{Q}[r_\theta(x, y')])] + C_\mathcal{D}$$

READ MORE

LOW RANK ADAPTATION LORA

#AI #nlp #llm #RLHF

$$W_{0} + \Delta W_{0} = W_{0} + BA, h=W_{0}x + \Delta W_{0}x = W_{0}x + BAx, \text{Initialization:} A \sim N(0, \sigma^{2}), B = 0$$

READ MORE

Odds Ratio Preference Optimization ORPO

#AI #nlp #llm #RLHF

$$\mathcal{L}_{ORPO} = \mathbb{E}_{(x, y_w, y_l)}\left[ \mathcal{L}_{SFT} + \lambda \cdot \mathcal{L}_{OR} \right] $$ $$ \mathcal{L}_{OR} = -\log \sigma \left( \log \frac{\textbf{odds}_\theta(y_w|x)}{\textbf{odds}_\theta(y_l|x)} \right) $$

READ MORE

Perplexity of Language Model

#nlp #LLM #metric

$$\text{PPL}(X) = \exp \{- \frac{1}{t} \sum^{t}_{i} \log p_{\theta} (x_{i} | x_{ \lt i}) \}$$

READ MORE

RLHF Reinforcement Learning from Human Feedback

#AI #nlp #LLM #equation

$$p^*(y_w \succ y_l|x) = \sigma(r^*(x,y_w) - r^*(x,y_l)) $$ $$ \mathcal{L}_R(r_\phi) = \mathbb{E}_{x,y_w,y_l \sim D}[- \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))] $$ $$ \mathbb{E}_{x \in D, y \in \pi_\theta} [r_\phi(x,y)] - \beta D_{\text{KL}}(\pi_\theta(y|x) \| \pi_{\text{ref}}(y|x)) $$

READ MORE

Transformer

#machine learning #nlp #gpt

$$\text{Attention}(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V$$

READ MORE

ai

Binary Cross Entropy Optimization BCO

#AI #nlp #llm #RLHF

$$E_{(x, y_w, y_l) \sim \mathcal{D}} [-\log \sigma \left( r_\theta (x, y_w) - r_\theta(x, y_l) \right) ] < E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \sigma (r_\theta(x, y_w))] + E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \left( 1 - \sigma (r_\theta (x, y_l)) \right)] $$ $$ E_{(x, y_w, y_l) \sim \mathcal{D}} [- \log \sigma(r_\theta(x, y_w) - \delta) - \log \sigma(- (r_\theta(x, y_l) - \delta))] $$ $$ \mathcal{L}_\text{BCO}(\theta) = - E_{(x, y) \sim \mathcal{D}^+} [\log \sigma (r_\theta (x, y) - \delta)] - E_{(x, y) \sim \mathcal{D}^-} \left[ \frac{p_\psi (f = 1 \mid x)}{p_\psi (f = 0 \mid x)} \log \sigma (- (r_\theta (x, y) - \delta)) \right] $$

READ MORE

Contrastive Preference Optimization CPO

#AI #nlp #llm #RLHF

$$\mathcal{L}(\pi_\theta;\pi_{\text{ref}}) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \log \frac{\pi_{\theta}(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_{\theta}(y_l | x)}{\pi_{\text{ref}} (y_l | x)} \Big) \Big] $$ $$ \mathcal{L}(\pi_\theta;U) = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \log \pi_{\theta}(y_w | x) \nonumber \\ - \beta \log \pi_{\theta}(y_l | x) \Big) \Big] $$ $$ \min_\theta \mathcal{L}(\pi_\theta, U) \notag \text{ s.t. } \mathbb{E}_{(x,y_w) \sim \mathcal{D}}\Big [ \mathbb{KL}(\pi_w(y_w|x)||\pi_\theta(y_w|x))\Big] < \epsilon $$ $$ \min_\theta\underbrace{ \mathcal{L}(\pi_\theta, U)}_{\mathcal{L}_\text{prefer}} \underbrace{-\mathbb{E}_{(x,y_w) \sim \mathcal{D}} [\log \pi_\theta(y_w| x)]}_{\mathcal{L}_\text{NLL}} $$

READ MORE

Denoising Diffusion Policy Optimization DDPO

#AI #nlp #llm #RLHF

$$\mathcal{J}_\text{DDRL}(\theta) = \mathbb{E}_{c \sim p(c), x_{0} \sim p_{\theta} (x_{0} | c)} [r(x_{0}, c)] $$ $$ w_{\text{RWR}}(x_0, c) = \frac{1}{Z} \exp\big(\beta r(x_0, c) \big) $$ $$ w_{\text{sparse}} (x_0, c) = \mathbf{1} \big[ r(x_0, c) \geq C \big] $$ $$ \nabla_\theta \mathcal{J}_\text{DDRL} = \mathbb{E} {\; \sum_{t=0}^{T} \nabla_\theta \log p_\theta(x_{t-1} \mid x_t, c) \; r(x_0, c)} $$ $$ \nabla_\theta \mathcal{J}_\text{DDRL} = \mathbb{E} {\; \sum_{t=0}^{T} \frac{p_\theta (x_{t-1} \mid x_t, c)}{p_{\theta_\text{old}} (x_{t-1} \mid x_t, c)} \; \nabla_\theta \log p_\theta(x_{t-1} \mid x_t, c) \; r(x_0, c)} $$

READ MORE

Generalized Knowledge Distillation GKD

#AI #nlp #llm #RLHF

$$L_\mathrm{GKD}(\theta) := (1 - \lambda) \mathbb{E}_{(x, y) \sim (X, Y)} \big[ \mathcal{D}(p_{T} \| p_{S}^\theta)(y|x) \big] + \lambda \mathbb{E}_{x\sim X} \Big[\mathbb{E}_{y \sim p_{S} (\cdot|x)} \big[\mathcal{D}(p_{T} \| p_{S}^\theta)(y|x)\big]\Big] $$

READ MORE

KTO Kahneman-Tversky Optimisation Equation

#nlp #llm #AI

$$f(\pi_\theta, \pi_\text{ref}) = \mathbb{E}_{x,y\sim\mathcal{D}}[ a_{x,y} v(r_\theta(x,y) - \mathbb{E}_{Q}[r_\theta(x, y')])] + C_\mathcal{D}$$

READ MORE

LOW RANK ADAPTATION LORA

#AI #nlp #llm #RLHF

$$W_{0} + \Delta W_{0} = W_{0} + BA, h=W_{0}x + \Delta W_{0}x = W_{0}x + BAx, \text{Initialization:} A \sim N(0, \sigma^{2}), B = 0$$

READ MORE

Odds Ratio Preference Optimization ORPO

#AI #nlp #llm #RLHF

$$\mathcal{L}_{ORPO} = \mathbb{E}_{(x, y_w, y_l)}\left[ \mathcal{L}_{SFT} + \lambda \cdot \mathcal{L}_{OR} \right] $$ $$ \mathcal{L}_{OR} = -\log \sigma \left( \log \frac{\textbf{odds}_\theta(y_w|x)}{\textbf{odds}_\theta(y_l|x)} \right) $$

READ MORE

RLHF Reinforcement Learning from Human Feedback

#AI #nlp #LLM #equation

$$p^*(y_w \succ y_l|x) = \sigma(r^*(x,y_w) - r^*(x,y_l)) $$ $$ \mathcal{L}_R(r_\phi) = \mathbb{E}_{x,y_w,y_l \sim D}[- \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))] $$ $$ \mathbb{E}_{x \in D, y \in \pi_\theta} [r_\phi(x,y)] - \beta D_{\text{KL}}(\pi_\theta(y|x) \| \pi_{\text{ref}}(y|x)) $$

READ MORE

machine learning

Conditional Random Field CRF

#machine learning #nlp

$$P(y|x)=\frac{1}{Z(x)}\exp(\sum_{i,k}\lambda_{k}t_{k}(y_{i-1},y_{i},x,i))+\sum_{i,l}\mu_{l}s_{l}(y_{i},x,i)) \\ Z(x)=\sum_{y}\exp(\sum_{i,k}\lambda_{k}t_{k}(y_{i-1},y_{i},x,i))+\sum_{i,l}\mu_{l}s_{l}(y_{i},x,i))$$

READ MORE

Hidden Markov Model

#machine learning #nlp

$$Q=\{q_{1},q_{2},...,q_{N}\}, V=\{v_{1},v_{2},...,v_{M}\} \\ I=\{i_{1},i_{2},...,i_{T}\},O=\{o_{1},o_{2},...,o_{T}\} \\ A=[a_{ij}]_{N \times N}, a_{ij}=P(i_{t+1}=q_{j}|i_{t}=q_{i}) \\ B=[b_{j}(k)]_{N \times M},b_{j}(k)=P(o_{t}=v_{k}|i_{t}=q_{j})$$

READ MORE

Transformer

#machine learning #nlp #gpt

$$\text{Attention}(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V$$

READ MORE