# Chapter 7. Evaluations and Loss components

## Evaluations

This part evolves the main evaluation metrics for various tasks. These metrics derive from the save base class and have the same interface for easy use.

1. For classification problems, we implement: precision, recall, F1 and accuracy metrics

    import torch
    from graph4nlp.pytorch.modules.evaluation.accuracy import Accuracy
    ground_truth = torch.Tensor([0, 1, 3, 1, 1]).long()
    predict = torch.Tensor([0, 1, 3, 3, 1]).long()

    metric = Accuracy(metrics=["precision", "recall", "F1", "accuracy"])

    precision, recall, f1, accuracy = metric.calculate_scores(
            ground_truth=ground_truth, predict=predict, average="micro")

2. For generation tasks, we implement 6 metrics including: 1) BLEU (BLEU@1-4), 2), BLEUTranslation(SacreBLEU), 3)CIDEr, 4)METEOR, 5) ROUGE (ROUGE-L), 6)SummarizationRouge (implemented by pyrouge).

    from graph4nlp.pytorch.modules.evaluation import BLEU

    bleu_metrics = BLEU(n_grams=[1, 2, 3, 4])

    prediction = ["I am a PHD student.", "I am interested in Graph Neural Network."]
    ground_truth = ["I am a student.", "She is interested in Math."]

    scores = bleu_metrics.calculate_scores(ground_truth=ground_truth, predict=prediction)

## Loss components

We have implemented several specific loss functions for various tasks.

### Sequence generation loss.

We have wrapped the cross-entropy loss and the coverage loss(optional) to calculate the final loss for various sequence generation tasks (e.g., graph2seq, seq2seq).

    from graph4nlp.pytorch.modules.loss.seq_generation_loss import SeqGenerationLoss
    from graph4nlp.pytorch.models.graph2seq import Graph2Seq

    loss_function = SeqGenerationLoss(ignore_index=0, use_coverage=True)

    logits, enc_attn_weights, coverage_vectors = Graph2Seq(graph, tgt)
    graph2seq_loss = SeqGenerationLoss(logits, tgt, enc_attn_weights=enc_attn_weights, coverage_vectors=coverage_vectors)

### Knowledge Graph Loss

In the state-of-the-art KGE models, loss functions were designed according to various pointwise, pairwise and multi-class approaches. Refers to Loss Functions in Knowledge Graph Embedding Models

Pointwise Loss Function

1. MSELoss creates a criterion that measures the mean squared error (squared L2 norm) between each element in the input \(x\) and target \(y\). It is the wrapper of `nn.MSELoss` in pytorch.

2. SOFTMARGINLOSS Creates a criterion that optimizes a two-class classification logistic loss between input tensor \(x\) and target tensor \(y\) (containing 1 or -1). It is the wrapper of `nn.SoftMarginLoss` in pytorch.

The number of positive and negative samples should be about the same, otherwise it’s easy to overfit.

\[\text{loss}(x, y) = \sum_i \frac{\log(1 + \exp(-y[i]*x[i]))}{\text{x.nelement}()}\]

Pairwise Loss Function

1. SoftplusLoss refers to the paper OpenKE: An Open Toolkit for Knowledge Embedding

    class SoftplusLoss(nn.Module):
        def __init__(self, adv_temperature=None):
            super(SoftplusLoss, self).__init__()
            self.criterion = nn.Softplus()
            if adv_temperature != None:
                self.adv_temperature = nn.Parameter(torch.Tensor([adv_temperature]))
                self.adv_temperature.requires_grad = False
                self.adv_flag = True
            else:
                self.adv_flag = False

        def get_weights(self, n_score):
            return torch.softmax(n_score * self.adv_temperature, dim=-1).detach()

        def forward(self, p_score, n_score):
            if self.adv_flag:
                return (self.criterion(-p_score).mean() + (self.get_weights(n_score) * self.criterion(n_score)).sum(
                    dim=-1).mean()) / 2
            else:
                return (self.criterion(-p_score).mean() + self.criterion(n_score).mean()) / 2

2. SigmoidLoss refers to the paper OpenKE: An Open Toolkit for Knowledge Embedding

    class SigmoidLoss(nn.Module):
        def __init__(self, adv_temperature = None):
            super(SigmoidLoss, self).__init__()
            self.criterion = nn.LogSigmoid()
            if adv_temperature != None:
                self.adv_temperature = nn.Parameter(torch.Tensor([adv_temperature]))
                self.adv_temperature.requires_grad = False
                self.adv_flag = True
            else:
                self.adv_flag = False

        def get_weights(self, n_score):
            return torch.softmax(n_score * self.adv_temperature, dim = -1).detach()

        def forward(self, p_score, n_score):
            if self.adv_flag:
                return -(self.criterion(p_score).mean() + (self.get_weights(n_score) * self.criterion(-n_score)).sum(dim = -1).mean()) / 2
            else:
                return -(self.criterion(p_score).mean() + self.criterion(-n_score).mean()) / 2

The implementations of `SoftplusLoss` and `SigmoidLoss` refer to OpenKE.

Multi-Class Loss Function

1. Binary Cross Entropy Loss Creates a criterion that measures the Binary Cross Entropy between the target and the output. Note that the targets \(y\) should be numbers between 0 and 1. It is the wrapper of `nn.BCELoss` in pytorch.

Next it is a simple how to use code:

    import torch
    from graph4nlp.pytorch.modules.loss.kg_loss import KGLoss

    loss_function = KGLoss(loss_type="BCELoss")
    m = nn.Sigmoid()
    input = torch.randn(3, requires_grad=True)
    target = torch.empty(3).random_(2)
    output = loss_function(m(input), target)

### General Loss

It includes the most used loss functions containing:

1. `NLL` loss. It is the wrapper of `nn.NLLLoss` in pytorch.

2. `BCE` loss. It is the wrapper of `nn.BCELoss` in pytorch.

3. `BCEWithLogits` loss. It is the wrapper of `nn.BCEWithLogitsLoss` in pytorch.

4. `MultiLabelMargin` loss. It is the wrapper of `nn.MultiLabelMarginLoss` in pytorch.

5. `SoftMargin` loss. It is the wrapper of `nn.SoftMargin` in pytorch.

6. `CrossEntropy` loss. It is the wrapper of `nn.CrossEntropy` in pytorch.

Next it is a simple how to use code:

    import torch
    from graph4nlp.pytorch.modules.loss.general_loss import GeneralLoss

    loss_function = GeneralLoss(loss_type="CrossEntropy")
    input = torch.randn(3, 5)
    target = torch.empty(3, dtype=torch.long).random_(5)
    output = loss_function(input, target)