Auto Update Docs (#118)

Summary:
Pull Request resolved: #118

This diff adds a python script `update_docs.py` which automatically generates `torcheval.metrics.rst` and `torcheval.metrics.functional.rst`, ensuring the API docs will always be in sync.

I also fixed several errors with the docstrings and missing imports which I found while writing this code.

Reviewed By: ananthsub

Differential Revision: D42404082

fbshipit-source-id: 7c3fabffdb016d6c7528c2c3f3bfacc5e083fd26

.pre-commit-config.yaml

@@ -11,6 +11,7 @@ repos:
     -   id: check-added-large-files
         args: ['--maxkb=500']
     -   id: end-of-file-fixer
+        exclude: '.*\.rst'
 
 -   repo: https://github.com/Lucas-C/pre-commit-hooks
     rev: v1.1.7

docs/Makefile

@@ -17,4 +17,5 @@ help:
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
+	python update_docs.py
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/build_docs.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+# This script shows how to build the docs.
+# 1. First ensure you have the installed requirements for torcheval in `requirements.txt`
+# 2. Then make sure you have installed the requirements inside `docs/requirements.txt`
+# 3. Finally cd into docs/ and source this script. Sphinx reads through the installed module
+# pull docstrings, so this script just installs the current version of torcheval on your
+# system before it builds the docs with `make html`
+cd .. || exit
+pip install --no-build-isolation .
+cd docs || exit
+make html

docs/source/index.rst

@@ -8,15 +8,6 @@ A library with simple and straightforward tooling for model evaluations and a de
 3. Is designed with extensibility in mind: you have the freedom to easily create your own metrics and leverage our toolkit.
 4. Provides tools for profiling memory and compute requirements for PyTorch based models.
 
-TorchEval Tutorials
--------------------
-.. toctree::
-   :maxdepth: 2
-   :caption: Examples:
-
-   QuickStart Notebook <https://github.com/pytorch/torcheval/blob/main/examples/Introducing_TorchEval.ipynb>
-   metric_example.rst
-
 QuickStart
 ===========================================
 
@@ -78,6 +69,22 @@ Further Reading
 * Check out the distributed example
 * Check out how to make your own metric
 
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
+TorchEval Tutorials
+-------------------
+.. toctree::
+   :maxdepth: 2
+   :caption: Examples:
+
+   QuickStart Notebook <https://github.com/pytorch/torcheval/blob/main/examples/Introducing_TorchEval.ipynb>
+   metric_example.rst
+
 TorchEval API
 -----------------
 
@@ -89,10 +96,3 @@ TorchEval API
    torcheval.metrics.functional.rst
    torcheval.metrics.toolkit.rst
    torcheval.tools.rst
-
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`

docs/source/torcheval.metrics.functional.rst

@@ -1,51 +1,57 @@
-.. currentmodule:: torcheval.metrics.functional
-
 Functional Metrics
 ==================
 
 .. automodule:: torcheval.metrics.functional
 
 Aggregation Metrics
--------------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
    :nosignatures:
 
+   auc
    mean
    sum
    throughput
 
 Classification Metrics
-----------------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
    :nosignatures:
 
    binary_accuracy
+   binary_auprc
    binary_auroc
+   binary_binned_auroc
    binary_binned_precision_recall_curve
    binary_confusion_matrix
    binary_f1_score
    binary_normalized_entropy
    binary_precision
    binary_precision_recall_curve
    binary_recall
+   binary_recall_at_fixed_precision
    multiclass_accuracy
+   multiclass_auprc
    multiclass_auroc
+   multiclass_binned_auroc
    multiclass_binned_precision_recall_curve
    multiclass_confusion_matrix
    multiclass_f1_score
    multiclass_precision
    multiclass_precision_recall_curve
    multiclass_recall
    multilabel_accuracy
+   multilabel_auprc
+   multilabel_precision_recall_curve
+   multilabel_recall_at_fixed_precision
    topk_multilabel_accuracy
 
-
 Ranking Metrics
----------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
@@ -59,11 +65,25 @@ Ranking Metrics
    weighted_calibration
 
 Regression Metrics
-------------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
    :nosignatures:
 
    mean_squared_error
    r2_score
+
+Text Metrics
+-------------------------------------------------------------------
+
+.. autosummary::
+   :toctree: generated
+   :nosignatures:
+
+   bleu_score
+   perplexity
+   word_error_rate
+   word_information_preserved
+   word_information_lost
+

docs/source/torcheval.metrics.rst

@@ -1,18 +1,17 @@
-.. currentmodule:: torcheval.metrics
-
 Metrics
 =============
 
 .. automodule:: torcheval.metrics
 
 
 Aggregation Metrics
-----------------------------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
    :nosignatures:
 
+   AUC
    Cat
    Max
    Mean
@@ -21,13 +20,14 @@ Aggregation Metrics
    Throughput
 
 Classification Metrics
-----------------------------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
    :nosignatures:
 
    BinaryAccuracy
+   BinaryAUPRC
    BinaryAUROC
    BinaryBinnedAUROC
    BinaryBinnedPrecisionRecallCurve
@@ -37,7 +37,9 @@ Classification Metrics
    BinaryPrecision
    BinaryPrecisionRecallCurve
    BinaryRecall
+   BinaryRecallAtFixedPrecision
    MulticlassAccuracy
+   MulticlassAUPRC
    MulticlassAUROC
    MulticlassBinnedAUROC
    MulticlassBinnedPrecisionRecallCurve
@@ -47,10 +49,13 @@ Classification Metrics
    MulticlassPrecisionRecallCurve
    MulticlassRecall
    MultilabelAccuracy
+   MultilabelAUPRC
+   MultilabelPrecisionRecallCurve
+   MultilabelRecallAtFixedPrecision
    TopKMultilabelAccuracy
 
 Ranking Metrics
-----------------------------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
@@ -62,7 +67,7 @@ Ranking Metrics
    WeightedCalibration
 
 Regression Metrics
-----------------------------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
@@ -71,8 +76,21 @@ Regression Metrics
    MeanSquaredError
    R2Score
 
+Text Metrics
+-------------------------------------------------------------------
+
+.. autosummary::
+   :toctree: generated
+   :nosignatures:
+
+   BLEUScore
+   Perplexity
+   WordErrorRate
+   WordInformationLost
+   WordInformationPreserved
+
 Windowed Metrics
-----------------------------------
+-------------------------------------------------------------------
 
 .. autosummary::
    :toctree: generated
@@ -81,4 +99,6 @@ Windowed Metrics
    WindowedBinaryAUROC
    WindowedBinaryNormalizedEntropy
    WindowedClickThroughRate
+   WindowedMeanSquaredError
    WindowedWeightedCalibration
+