
.. _scikit-learn's implementation of SMAPE: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html
.. _scikit-learn's implementation of MAPE: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html
.. _Mean Average Precision: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Mean_average_precision
.. _Fall-out: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Fall-out
.. _Normalized Discounted Cumulative Gain: https://en.wikipedia.org/wiki/Discounted_cumulative_gain
.. _IR Precision: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Precision
.. _IR R-Precision: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#R-precision
.. _IR Recall: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Recall
.. _Accuracy: https://en.wikipedia.org/wiki/Accuracy_and_precision
.. _SMAPE: https://en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error
.. _SNR: https://en.wikipedia.org/wiki/Signal-to-noise_ratio
.. _ROC AUC: https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Further_interpretations
.. _Cohen's kappa score: https://en.wikipedia.org/wiki/Cohen%27s_kappa
.. _scikit-learn's implementation of confusion matrix: https://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix
.. _confusion matrix gets calculated per label: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html
.. _F-score: https://en.wikipedia.org/wiki/F-score
.. _Hamming distance: https://en.wikipedia.org/wiki/Hamming_distance
.. _Hinge loss: https://en.wikipedia.org/wiki/Hinge_loss
.. _Jaccard index: https://en.wikipedia.org/wiki/Jaccard_index
.. _KL divergence: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
.. _Matthews correlation coefficient: https://en.wikipedia.org/wiki/Matthews_correlation_coefficient
.. _Precision: https://en.wikipedia.org/wiki/Precision_and_recall
.. _Recall: https://en.wikipedia.org/wiki/Precision_and_recall
.. _Dice: https://en.wikipedia.org/wiki/Sørensen–Dice_coefficient
.. _Specificity: https://en.wikipedia.org/wiki/Sensitivity_and_specificity
.. _Type I and Type II errors: https://en.wikipedia.org/wiki/Type_I_and_type_II_errors
.. _confusion matrix: https://en.wikipedia.org/wiki/Confusion_matrix#Table_of_confusion
.. _sklearn averaging methods: https://scikit-learn.org/stable/modules/model_evaluation.html#multiclass-and-multilabel-classification
.. _Cosine Similarity: https://en.wikipedia.org/wiki/Cosine_similarity
.. _spearmans rank correlation coefficient: https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient
.. _WordErrorRate: https://en.wikipedia.org/wiki/Word_error_rate
.. _FID: https://en.wikipedia.org/wiki/Fr%C3%A9chet_inception_distance
.. _MIFID: https://arxiv.org/abs/2106.03062
.. _mean-squared-error: https://en.wikipedia.org/wiki/Mean_squared_error
.. _SSIM: https://en.wikipedia.org/wiki/Structural_similarity
.. _explained variance: https://en.wikipedia.org/wiki/Explained_variation
.. _IR Average precision: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Average_precision
.. _IR Fall-out: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Fall-out
.. _MAPE implementation returns: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_percentage_error.html
.. _mean squared logarithmic error: https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-log-error
.. _LPIPS: https://arxiv.org/abs/1801.03924
.. _Mean-Average-Precision (mAP) and Mean-Average-Recall (mAR): https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173
.. _Tweedie Deviance Score: https://en.wikipedia.org/wiki/Tweedie_distribution#The_Tweedie_deviance
.. _Permutation Invariant Training of Deep Models: https://ieeexplore.ieee.org/abstract/document/7952154
.. _Top-label Calibration Error: https://arxiv.org/abs/1909.10155
.. _Gradient Computation of Image: https://en.wikipedia.org/wiki/Image_gradient
.. _R2 Score_Coefficient Determination: https://en.wikipedia.org/wiki/Coefficient_of_determination
.. _Rank of element tensor: https://github.com/scipy/scipy/blob/v1.6.2/scipy/stats/stats.py#L4140-L4303
.. _Mean Reciprocal Rank: https://en.wikipedia.org/wiki/Mean_reciprocal_rank
.. _BERT_score: https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py
.. _Bert_score Evaluating Text Generation: https://arxiv.org/abs/1904.09675
.. _BLEU score: https://en.wikipedia.org/wiki/BLEU
.. _BLEU: http://www.aclweb.org/anthology/P02-1040
.. _SacreBLEU: https://github.com/mjpost/sacrebleu
.. _SacreBleu_ter: https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/ter.py
.. _Machine Translation Evolution: https://aclanthology.org/P04-1077
.. _Rouge score_Text Normalizition: https://github.com/google-research/google-research/blob/master/rouge/tokenize.py
.. _Calculate Rouge Score: https://en.wikipedia.org/wiki/ROUGE_(metric)
.. _Rouge Detail: https://aclanthology.org/W04-1013/
.. _Square Root of a Positive Definite Matrix: https://github.com/steveli/pytorch-sqrtm/blob/master/sqrtm.py
.. _Fid Score: https://github.com/photosynthesis-team/piq/blob/master/piq/fid.py
.. _Rethinking the Inception Architecture for ComputerVision: https://arxiv.org/abs/1512.00567
.. _GANs Trained by a Two Time-Scale: https://arxiv.org/abs/1706.08500
.. _Improved Techniques for Training GANs: https://arxiv.org/abs/1606.03498
.. _KID Score: https://github.com/toshas/torch-fidelity/blob/v0.3.0/torch_fidelity/metric_kid.py
.. _Demystifying MMD GANs: https://arxiv.org/abs/1801.01401
.. _Compute Peak Signal-to-Noise Ratio: https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio
.. _Turn a Metric into a Bootstrapped: https://en.wikipedia.org/wiki/Bootstrapping_(statistics)
.. _Metric Test for Reset: https://github.com/Lightning-AI/pytorch-lightning/pull/7055
.. _Compute Mean Absolute Error: https://en.wikipedia.org/wiki/Mean_absolute_error
.. _Mean Absolute Percentage Error: https://en.wikipedia.org/wiki/Mean_absolute_percentage_error
.. _mean squared error: https://en.wikipedia.org/wiki/Mean_squared_error
.. _Aggregate the statistics from multiple devices: https://stackoverflow.com/questions/68395368/estimate-running-correlation-on-multiple-nodes
.. _Pearson Correlation Coefficient: https://en.wikipedia.org/wiki/Pearson_correlation_coefficient
.. _Python ROUGE Implementation: https://pypi.org/project/rouge-score/
.. _Scikit_Learn-Ranking.py: https: //github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/_ranking.py
.. _Verified Uncertainty Calibration: https://arxiv.org/abs/1909.10155
.. _SQuAD Metric: https://arxiv.org/abs/1606.05250
.. _chrF score: https://aclanthology.org/W15-3049
.. _chrF++ score: https://aclanthology.org/W17-4770
.. _TER: https://aclanthology.org/2006.amta-papers.25
.. _ExtendedEditDistance: https://aclanthology.org/W19-5359
.. _MultiScaleSSIM: https://ece.uwaterloo.ca/~z70wang/publications/msssim
.. _UniversalImageQualityIndex: https://ieeexplore.ieee.org/abstract/document/995823
.. _SpectralDistortionIndex: https://www.semanticscholar.org/paper/Multispectral-and-panchromatic-data-fusion-without-Alparone-Aiazzi/b6db12e3785326577cb95fd743fecbf5bc66c7c9
.. _SpatialDistortionIndex: https://www.semanticscholar.org/paper/Multispectral-and-panchromatic-data-fusion-without-Alparone-Aiazzi/b6db12e3785326577cb95fd743fecbf5bc66c7c9
.. _QualityWithNoReference: https://www.semanticscholar.org/paper/Multispectral-and-panchromatic-data-fusion-without-Alparone-Aiazzi/b6db12e3785326577cb95fd743fecbf5bc66c7c9
.. _RelativeAverageSpectralError: https://www.semanticscholar.org/paper/Data-Fusion.-Definitions-and-Architectures-Fusion-Wald/51b2b81e5124b3bb7ec53517a5dd64d8e348cadf
.. _WMAPE: https://en.wikipedia.org/wiki/WMAPE
.. _CER: https://rechtsprechung-im-ostseeraum.archiv.uni-greifswald.de/word-error-rate-character-error-rate-how-to-evaluate-a-model
.. _MER: https://www.isca-speech.org/archive/interspeech_2004/morris04_interspeech.html
.. _WIL: https://www.isca-speech.org/archive/interspeech_2004/morris04_interspeech.html
.. _WIP: https://www.isca-archive.org/interspeech_2004/morris04_interspeech.pdf
.. _TV: https://en.wikipedia.org/wiki/Total_variation_denoising
.. _InfoLM: https://arxiv.org/abs/2112.01589
.. _alpha divergence: https://static.renyi.hu/renyi_cikkek/1961_on_measures_of_entropy_and_information.pdf
.. _beta divergence: https://www.sciencedirect.com/science/article/pii/S0047259X08000456
.. _AB divergence: https://pdfs.semanticscholar.org/744b/1166de34cb099100f151f3b1459f141ae25b.pdf
.. _Rényi divergence: https://static.renyi.hu/renyi_cikkek/1961_on_measures_of_entropy_and_information.pdf
.. _Fisher-Rao distance: https://arxiv.org/abs/1711.01530
.. _Cramer's V: https://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V
.. _Kendall Rank Correlation Coefficient: https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient
.. _The Treatment of Ties in Ranking Problems: https://www.jstor.org/stable/2332303
.. _LogCosh Error: https://arxiv.org/abs/2101.10427
.. _Tschuprow's T: https://en.wikipedia.org/wiki/Tschuprow%27s_T
.. _Pearson's Contingency Coefficient: https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/pearcont.htm
.. _CLIP score: https://arxiv.org/abs/2104.08718
.. _Huggingface OpenAI: https://huggingface.co/openai
.. _Theils Uncertainty coefficient: https://en.wikipedia.org/wiki/Uncertainty_coefficient
.. _Perceptual Evaluation of Speech Quality: https://en.wikipedia.org/wiki/Perceptual_Evaluation_of_Speech_Quality
.. _pesq package: https://github.com/ludlows/python-pesq
.. _Deep Noise Suppression performance evaluation based on Mean Opinion Score: https://arxiv.org/abs/2010.15258
.. _Non-Intrusive Speech Quality Assessment: https://arxiv.org/abs/2104.09494
.. _Cees Taal's website: http://www.ceestaal.nl/code/
.. _pystoi package: https://github.com/mpariente/pystoi
.. _stoi ref1: https://ieeexplore.ieee.org/abstract/document/5495701
.. _stoi ref2: https://ieeexplore.ieee.org/abstract/document/5713237
.. _stoi ref3: https://ieeexplore.ieee.org/abstract/document/7539284
.. _sdr ref1: https://ieeexplore.ieee.org/abstract/document/1643671
.. _sdr ref2: https://arxiv.org/abs/2110.06440
.. _Scale-invariant signal-to-distortion ratio: https://arxiv.org/abs/1811.02508
.. _Scale-invariant signal-to-noise ratio: https://arxiv.org/abs/1711.00541
.. _Source-aggregated signal-to-distortion ratio: https://arxiv.org/abs/2110.15581
.. _Complex scale-invariant signal-to-noise ratio: https://arxiv.org/abs/2011.09162
.. _Signal-to-noise ratio: https://arxiv.org/abs/1811.02508
.. _Speech-to-Reverberation Modulation Energy Ratio: https://ieeexplore.ieee.org/abstract/document/5547575
.. _SRMRToolbox: https://github.com/MuSAELab/SRMRToolbox
.. _SRMRpy: https://github.com/jfsantos/SRMRpy
.. _Permutation invariant training: https://arxiv.org/abs/1607.00325
.. _ranking ref1: https://link.springer.com/chapter/10.1007/978-0-387-09823-4_34
.. _Spectral Distortion Index: https://www.semanticscholar.org/paper/Multispectral-and-panchromatic-data-fusion-without-Alparone-Aiazzi/b6db12e3785326577cb95fd743fecbf5bc66c7c9
.. _Error relative global dimensionless synthesis: https://minesparis-psl.hal.science/hal-00464703/document
.. _fid ref1: https://arxiv.org/abs/1512.00567
.. _fid ref2: https://arxiv.org/abs/1706.08500
.. _inception ref1: https://arxiv.org/abs/1606.03498
.. _inception ref2: https://arxiv.org/abs/1706.08500
.. _kid ref1: https://arxiv.org/abs/1801.01401
.. _kid ref2: https://arxiv.org/abs/1706.08500
.. _Spectral Angle Mapper: https://ntrs.nasa.gov/citations/19940012238
.. _Multilabel coverage error: https://link.springer.com/chapter/10.1007/978-0-387-09823-4_34
.. _Panoptic Quality: https://arxiv.org/abs/1801.00868
.. _torchmetrics mAP example: https://github.com/Lightning-AI/torchmetrics/blob/master/_samples/detection_map.py
.. _Peak Signal to Noise Ratio With Blocked Effect: https://ieeexplore.ieee.org/abstract/document/5535179
.. _Minkowski Distance: https://en.wikipedia.org/wiki/Minkowski_distance
.. _Demographic parity: http://www.fairmlbook.org/
.. _Equal opportunity: https://proceedings.neurips.cc/paper/2016/hash/9d2682367c3935defcb1f9e247a97c0d-Abstract.html
.. _Seamless Scene Segmentation paper: https://arxiv.org/abs/1905.01220
.. _Fleiss kappa: https://en.wikipedia.org/wiki/Fleiss%27_kappa
.. _VIF: https://ieeexplore.ieee.org/abstract/document/1576816
.. _CLIP-IQA: https://arxiv.org/abs/2207.12396
.. _CLIP: https://arxiv.org/abs/2103.00020
.. _PPL : https://arxiv.org/abs/1812.04948
.. _CIOU: https://arxiv.org/abs/2005.03572
.. _DIOU: https://arxiv.org/abs/1911.08287v1
.. _GIOU: https://arxiv.org/abs/1902.09630
.. _Mutual Information Score: https://en.wikipedia.org/wiki/Mutual_information
.. _Normalized Mutual Information Score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.normalized_mutual_info_score.html
.. _Adjusted Mutual Information Score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_mutual_info_score.html#sklearn.metrics.adjusted_mutual_info_score
.. _pycocotools: https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools
.. _Rand Score: https://link.springer.com/article/10.1007/BF01908075
.. _faster-coco-eval: https://github.com/MiXaiLL76/faster_coco_eval
.. _fork of pycocotools: https://github.com/ppwwyyxx/cocoapi
.. _Adjusted Rand Score: https://en.wikipedia.org/wiki/Rand_index#Adjusted_Rand_index
.. _Dunn Index: https://en.wikipedia.org/wiki/Dunn_index
.. _V-Measure Score: https://www.aclweb.org/anthology/D07-1043
.. _Homogeneity Score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.homogeneity_score.html
.. _Completeness Score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.completeness_score.html
.. _Davies-Bouldin Score: https://en.wikipedia.org/wiki/Davies%E2%80%93Bouldin_index
.. _Fowlkes-Mallows Index: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.fowlkes_mallows_score.html#sklearn.metrics.fowlkes_mallows_score
.. _FLORES-101: https://arxiv.org/abs/2106.03193
.. _FLORES-200: https://arxiv.org/abs/2207.04672
.. _averaging curve objects: https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html
.. _SCC: https://www.ingentaconnect.com/content/tandf/tres/1998/00000019/00000004/art00013
.. _Normalized Root Mean Squared Error: https://onlinelibrary.wiley.com/doi/abs/10.1111/1365-2478.12109
.. _Generalized Dice Score: https://arxiv.org/abs/1707.03237
.. _Hausdorff Distance: https://en.wikipedia.org/wiki/Hausdorff_distance
.. _averaging curve objects: https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html
.. _Procrustes Disparity: https://en.wikipedia.org/wiki/Procrustes_analysis
.. _Log AUC: https://pubmed.ncbi.nlm.nih.gov/20735049/
.. _Negative Predictive Value: https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values
