4. Algorithms

FairLangProc provides a comprehensive suite of bias mitigation processors for Large Language Models.

4.1. Supported processors

Fairness processors can be classified according to their stage in the Machine Learning pipeline:

• Pre-processors: Fairness processors that modify the model inputs (data augmentation, embedding projection,…).

• In-processors: Fairness processors that modify the training process (regularizers, adapters,…).

• Intra-processors: Fairness processors that modify the model’s behavior without further training (attention scaling,…).

The supported methods are:

• Counterfactual Data Augmentation (CDA) (Webster et al. 2020).

• Projection based debiasing (Bolukbasi et al., 2023).

• Bias removaL wIth No Demographics (BLIND) (Orgad et al., 2023).

• Adapter-based DEbiasing of LanguagE models (ADELE) (Lauscher et al., 2021).

• Modular Debiasing with Diff Subnetworks (Hauzenberger et al., 2023).

• Entropy Attention Temperature (EAT) scaling (Zayed et al., 2023).

• Entropy Attention Regularizer (EAR) (Attanasio et al., 2022).

• Embedding based regularizer (Liu et al., 2020).

• Selective unfreezing (Gira et al., 2024).

Note

Different algorithms have different trade-offs. See our tutorials for more detailed comparisons of fairness-accuracy tradeoffs.

• 4.1.1. Pre-processing
  • 4.1.1.1. Counterfactual Data Augmentation (CDA)
  • 4.1.1.2. Projection-based debiasing
  • 4.1.1.3. BLIND debiasing
• 4.1.2. In-processing
  • 4.1.2.1. ADELE
  • 4.1.2.2. Embedding based regularizer
  • 4.1.2.3. EAR
  • 4.1.2.4. Selective unfreezing
• 4.1.3. Intra-processing
  • 4.1.3.1. MoDDiffy
  • 4.1.3.2. EAT

See also

• Tutorials - Interactive Jupyter notebooks (DemoProcessors notebook, DemoDebiasing notebook) for debiasing techniques