Home

Awesome

<div align="center"> <img src="https://github.com/open-mmlab/mmeval/raw/main/docs/zh_cn/_static/image/mmeval-logo.png" height="100"/> <div>&nbsp;</div> <div align="center"> <b><font size="5">OpenMMLab website</font></b> <sup> <a href="https://openmmlab.com"> <i><font size="4">HOT</font></i> </a> </sup> &nbsp;&nbsp;&nbsp;&nbsp; <b><font size="5">OpenMMLab platform</font></b> <sup> <a href="https://platform.openmmlab.com"> <i><font size="4">TRY IT OUT</font></i> </a> </sup> </div> <div>&nbsp;</div>

PyPI - Python Version PyPI license open issues issue resolution

๐Ÿค”Reporting Issues

</div> <div align="center">

English | ็ฎ€ไฝ“ไธญๆ–‡

</div>

Introduction

MMEval is a machine learning evaluation library that supports efficient and accurate distributed evaluation on a variety of machine learning frameworks.

Major features:

<div align="center"> <img src="docs/zh_cn/_static/image/mmeval-arch.png" width="600"/> </div> <details> <summary> Supported distributed communication backends </summary>
MPI4Pytorch.distributedHorovodpaddle.distributedoneflow.comm
MPI4PyDistTorchCPUDist <br> TorchCUDADistTFHorovodDistPaddleDistOneFlowDist
</details> <details> <summary> Supported metrics and ML frameworks </summary>

NOTE: MMEval tested with PyTorch 1.6+, TensorFlow 2.4+, Paddle 2.2+ and OneFlow 0.8+.

Metricnumpy.ndarraytorch.Tensortensorflow.Tensorpaddle.Tensoroneflow.Tensor
Accuracyโœ”โœ”โœ”โœ”โœ”
SingleLabelMetricโœ”โœ”โœ”
MultiLabelMetricโœ”โœ”โœ”
AveragePrecisionโœ”โœ”โœ”
MeanIoUโœ”โœ”โœ”โœ”โœ”
VOCMeanAPโœ”
OIDMeanAPโœ”
COCODetectionโœ”
ProposalRecallโœ”
F1Scoreโœ”โœ”โœ”
HmeanIoUโœ”
PCKAccuracyโœ”
MpiiPCKAccuracyโœ”
JhmdbPCKAccuracyโœ”
EndPointErrorโœ”โœ”โœ”
AVAMeanAPโœ”
StructuralSimilarityโœ”
SignalNoiseRatioโœ”
PeakSignalNoiseRatioโœ”
MeanAbsoluteErrorโœ”
MeanSquaredErrorโœ”
</details>

Installation

MMEval requires Python 3.6+ and can be installed via pip.

pip install mmeval

To install the dependencies required for all the metrics provided in MMEval, you can install them with the following command.

pip install 'mmeval[all]'

Get Started

There are two ways to use MMEval's metrics, using Accuracy as an example:

from mmeval import Accuracy
import numpy as np

accuracy = Accuracy()

The first way is to directly call the instantiated Accuracy object to calculate the metric.

labels = np.asarray([0, 1, 2, 3])
preds = np.asarray([0, 2, 1, 3])
accuracy(preds, labels)
# {'top1': 0.5}

The second way is to calculate the metric after accumulating data from multiple batches.

for i in range(10):
    labels = np.random.randint(0, 4, size=(100, ))
    predicts = np.random.randint(0, 4, size=(100, ))
    accuracy.add(predicts, labels)

accuracy.compute()
# {'top1': ...}

Learn More

<details> <summary>Tutorials</summary> </details> <details> <summary>Examples</summary> </details> <details> <summary>Design</summary> </details>

In the works

Contributing

We appreciate all contributions to improve MMEval. Please refer to CONTRIBUTING.md for the contributing guideline.

License

This project is released under the Apache 2.0 license.

Projects in OpenMMLab