4.1.2. In-processing

In-processors are fairness processors that modify the training process.

The supported methods are:

4.1.2.1. ADELE

The ADELE procedure (Lauscher et al., 2021) is based on the adapter framework. A single adapter module is included to each transformer layer after the feed-forward sub-layer, where the outputs are compressed to a bottleneck dimension \(m\) and then decompressed back to the hidden size of the transformer, \(d_L\). The adapter module itself consists of a two-layer feed-forward network:

\[\text{Adapter}(\mathbf{h}, \mathbf{r}) = U \cdot g(D\cdot \mathbf{h) + \mathbf{r}}\]

where \(\mathbf{h}\) and \(\mathbf{r}\) are the hidden state and residual of the corresponding transformer layer, \(g\) is an activation function and \(D \in \mathbb{R}^{m \times d_L}, U \in \mathbb{R}^{d_L\times m}\) represent the projection matrices. The idea behind the adapter layer is to introduce an information bottleneck which compresses the latent representation of the inputs, forcing the model to discard all irrelevant information.

class FairLangProc.algorithms.inprocessors.adapter.DebiasAdapter(*args: Any, **kwargs: Any)[source]

Implements ADELE debiasing based on bottleneck adapter.

Example

>>> from adapters import AdapterTrainer
>>> from FairLangProc.algorithms.inprocessors import DebiasAdapter
>>>
>>> DebiasAdapter = DebiasAdapter(
...     model = AutoModel.from_pretrained('bert-base-uncased'),
...     adapter_config = "seq_bn"
...     )
>>> AdeleModel = DebiasAdapter.get_model()
>>>
>>> trainer = AdapterTrainer(
...     model=AdeleModel,
...     args=training_args,
...     train_dataset=train_CDA,
...     eval_dataset=val_dataset,
...     optimizers=(
...         AdamW(AdeleModel.parameters(),lr=1e-5, weight_decay=0.1),
...         None
...         )
... )
>>> trainer.train()
>>> results = trainer.evaluate()
>>> print(results)
__init__(model: torch.nn.Module, adapter_name: str = 'debias_adapter', adapter_config: str | dict = 'seq_bn') → None[source]

Constructor of the DebiasAdapter class.

Parameters:

  • model (nn.Module) – Pretrained model (e.g., BERT, GPT-2)

  • adapter_name (str) – Tensor with ids of text with demographic information of group A

  • adapter_config (Union[str, dict]) – Name or dictionary of the desired configuration for the adapter (bottleneck by default)

4.1.2.2. Embedding based regularizer

Embedding based regularizers (Liu et al., 2020) are based on the distance between the embeddings of counterfactual pairs given by $A$,

\[\mathcal{R} = \sum_{(a_i, a_j)\in A} || M(a_i) - M(a_j)||_2 .\]
class FairLangProc.algorithms.inprocessors.regularizers.EmbeddingBasedRegularizer(*args: Any, **kwargs: Any)[source]

Class for adding a regularizer based on the embeddings of counterfactual pairs. Requires the implementation of the _get_embedding method

Example

>>> from FairLangProc.algorithms.inprocessors import EmbeddingBasedRegularizer
>>> class BERTEmbedingReg(EmbeddingBasedRegularizer):
...     def _get_embedding(self, inputs):
...         return self.model(**inputs).last_hidden_state[:,0,:]
>>> model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased')
>>> tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
>>> words = [('he', 'she'), ('his', 'hers'), ('monk', 'nun')]
>>> EmbRegularizer = EARModel(
...      model = model,
...      tokenizer = tokenizer,
...      word_pairs = words,
...      ear_reg_strength = 0.01
... )
>>>
>>> trainer = Trainer(
...     model=EARRegularizer,
...     args=training_args,
...     train_dataset=train_dataset,
...     eval_dataset=val_dataset,
...     optimizers=(
...         AdamW(EARRegularizer.parameters(), lr=1e-5, weight_decay=0.1),
...         None
...         )
... )
>>> trainer.train()
>>> results = trainer.evaluate()
>>> print(results)
__init__(model: torch.nn.Module, tokenizer: TokenizerType, word_pairs: list[tuple[str]], ear_reg_strength: float = 0.01) → None[source]

Constructor of the EmbeddingBasedRegularizer class.

Parameters:

  • model (nn.Module) – A language model

  • tokenizer (TokenizerType) – Tokenizer of the model

  • word_pairs (list[tuple[str]]) – List of tuples of counterfactual pairs whose embeddings should be close together (e.g. daughter and son, he and she,…).

  • ear_reg_strength (float) – Hyper-parameter containing the strength of the regularization term.

4.1.2.3. EAR

EAR (Attanasio et al., 2022) tries to maximize the entropy of the attention weights to encourage attention to the broader context of the input,

\[\mathcal{R} = - \sum_{l=1}^L \text{entropy}_l(\mathbf{A})\]

where \(\text{entropy}_l(\cdot)\) denotes the entropy of the l-th layer.

class FairLangProc.algorithms.inprocessors.regularizers.EARModel(*args: Any, **kwargs: Any)[source]

Class for adding a regularizer based on entropy attention.

Example

>>> from FairLangProc.algorithms.inprocessors import EARModel
>>>
>>> model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased')
>>> EARRegularizer = EARModel(
...      model = model,
...      ear_reg_strength = 0.01
... )
>>>
>>> trainer = Trainer(
...     model=EARRegularizer,
...     args=training_args,
...     train_dataset=train_dataset,
...     eval_dataset=val_dataset,
...     optimizers=(
...         AdamW(EARRegularizer.parameters(), lr=1e-5, weight_decay=0.1),
...         None
...         )
... )
>>> trainer.train()
>>> results = trainer.evaluate()
>>> print(results)
__init__(model: torch.nn.Module, ear_reg_strength: float = 0.01)[source]

Constructor for the EARModel class

Parameters:

  • model (nn.Module) – A language model.

  • ear_reg_strength (float) – Hyper-parameter containing the strength of the regularization term.

4.1.2.4. Selective unfreezing

Selective unfreezing (Gira et al., 2024) aims to circumvent catastrophic forgetting during fine-tuning by freezing a big amount of the model parameters, which also helps lessening computational expenses.

FairLangProc.algorithms.inprocessors.selective_updating.selective_unfreezing(model: torch.nn.Module, substrings: list[str]) → None[source]

Freeze all model’s parameters and selectively unfreeze those specified in parameters.

Parameters:

  • model (nn.Module) – The model whose parameters will be adjusted.

  • substrings (list[str]) – List of substrings to search for in parameter names.

Example

>>> from FairLangProc.algorithms.inprocessors import selective_unfreezing
>>>
>>> FrozenBert = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased')
>>> selective_unfreezing(FrozenBert, ["attention.self", "attention.output"])
>>>
>>> trainer = Trainer(
...     model=FrozenBert,
...     args=training_args,
...     train_dataset=train_CDA,
...     eval_dataset=val_dataset,
...     optimizers=(
...         AdamW(FrozenBert.parameters(), lr=1e-5, weight_decay=0.1),
...         None
...         )
... )
>>> trainer.train()
>>> results = trainer.evaluate()
>>> print(results)