Update README and iterator docs (#889)

- adds more hints on how to build DALI on bare metal - how to set paths to some libs - adds info on how to set DALI build type - adds a more verbose description of `size` in Pytorch and MXNet iterators - adds missing python_operator example to the docs - fixes sphinx wranings Signed-off-by: Janusz Lisiecki <[email protected]>
msr-fiddle · May 23, 2019 · 1158435 · 1158435
1 parent e98cfb3
commit 1158435
Show file tree

Hide file tree

Showing 8 changed files with 45 additions and 15 deletions.
diff --git a/README.rst b/README.rst
@@ -24,10 +24,10 @@ Highlights of DALI are:
 
 .. overview-end-marker-do-not-remove
 
-.. installation-begin-marker-do-not-remove
-
 ----
 
+.. installation-begin-marker-do-not-remove
+
 DALI and NGC
 ------------
 
@@ -178,6 +178,7 @@ Change directory (``cd``) into Docker directory and run ``./build.sh``. If neede
 * CREATE_WHL - Create a standalone wheel. Default is ``YES``.
 * CREATE_RUNNER - Create Docker image with cuDNN, CUDA and DALI installed inside. It will create the ``Docker_run_cuda`` image, which needs to be run using ``nvidia-docker`` and DALI wheel in the ``wheelhouse`` directory under$
 * DALI_BUILD_FLAVOR - adds a suffix to DALI package name and put a note about it in the whl package description, i.e. `nightly` will result in the `nvidia-dali-nightly`
+* BUILD_TYPE - build type, available options: Debug, DevDebug, Release, RelWithDebInfo
 
 ----
 
@@ -353,6 +354,18 @@ Building DALI using Clang (experimental):
    We ensure that DALI compiles with all of those options turned OFF, but there may exist
    cross-dependencies between some of those features.
 
+Following CMake parameters could be helpful in setting the right paths:
+
+.. |libjpeg-turbo_cmake link| replace:: **libjpeg CMake docs page**
+.. _libjpeg-turbo_cmake link: https://cmake.org/cmake/help/v3.11/module/FindJPEG.html
+.. |protobuf_cmake link| replace:: **protobuf CMake docs page**
+.. _protobuf_cmake link: https://cmake.org/cmake/help/v3.11/module/FindProtobuf.html
+
+* FFMPEG_ROOT_DIR - path to installed FFmpeg
+* NVJPEG_ROOT_DIR - where nvJPEG can be found (from CUDA 10.0 it is shipped with the CUDA toolkit so this option is not needed there)
+* libjpeg-turbo options can be obtained from |libjpeg-turbo_cmake link|_
+* protobuf options can be obtained from |protobuf_cmake link|_
+
 Install Python bindings
 ^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/dali/pipeline/operators/reader/sequence_reader_op.cc b/dali/pipeline/operators/reader/sequence_reader_op.cc
@@ -32,23 +32,41 @@ stream. Extracted video stream is represented by one file for each frame, sortin
 frames lexicographically should give the original order of frames.
 Sequences do not cross stream boundary and only full sequences are considered - there is no padding.
 Example:
+
 > file_root
+
   > 0
+
     > 00001.png
+
     > 00002.png
+
     > 00003.png
+
     > 00004.png
+
     > 00005.png
+
     > 00006.png
+
     ....
+
   > 1
+
     > 00001.png
+
     > 00002.png
+
     > 00003.png
+
     > 00004.png
+
     > 00005.png
+
     > 00006.png
-    ....)code")
+
+    ....
+)code")
     .NumInput(0)
     .NumOutput(1)  // ([Frames])
     .AddArg("file_root",

diff --git a/dali/python/nvidia/dali/plugin/mxnet.py b/dali/python/nvidia/dali/plugin/mxnet.py
@@ -69,7 +69,7 @@ class DALIGenericIterator(object):
                  for data or label correspondingly.
                  output_names should be distinct.
     size : int
-           Epoch size.
+          Number of samples in the epoch (Usually the size of the dataset).
     data_layout : str, optional, default = 'NCHW'
                   Either 'NHWC' or 'NCHW' - layout of the pipeline outputs.
     fill_last_batch : bool, optional, default = True
@@ -280,7 +280,7 @@ class DALIClassificationIterator(DALIGenericIterator):
     pipelines : list of nvidia.dali.pipeline.Pipeline
                 List of pipelines to use
     size : int
-           Epoch size.
+           Number of samples in the epoch (Usually the size of the dataset).
     data_name : str, optional, default = 'data'
                 Data name for provided symbols.
     label_name : str, optional, default = 'softmax_label'

diff --git a/dali/python/nvidia/dali/plugin/pytorch.py b/dali/python/nvidia/dali/plugin/pytorch.py
@@ -68,7 +68,7 @@ class DALIGenericIterator(object):
                  of those names.
                  Each name should be distinct
     size : int
-           Epoch size.
+           Number of samples in the epoch (Usually the size of the dataset).
     auto_reset : bool, optional, default = False
                  Whether the iterator resets itself for the next epoch
                  or it requires reset() to be called separately.
@@ -247,7 +247,7 @@ class DALIClassificationIterator(DALIGenericIterator):
     pipelines : list of nvidia.dali.pipeline.Pipeline
                 List of pipelines to use
     size : int
-           Epoch size.
+           Number of samples in the epoch (Usually the size of the dataset).
     auto_reset : bool, optional, default = False
                  Whether the iterator resets itself for the next epoch
                  or it requires reset() to be called separately.

diff --git a/docs/examples/index.rst b/docs/examples/index.rst
@@ -13,3 +13,4 @@ Tutorials
    video/superres_pytorch/README.rst
    optical_flow/optical_flow_example.ipynb
    extend/create_a_custom_operator.ipynb
+   python_operator.ipynb
diff --git a/docs/examples/pytorch/single_stage_detector/pytorch_ssd.rst b/docs/examples/pytorch/single_stage_detector/pytorch_ssd.rst
@@ -1,5 +1,5 @@
 Single Shot MultiBox Detector training in PyTorch
-============================
+=================================================
 
 This example shows how DALI can be used in detection networks, specifically Single Shot Multibox Detector originally published by Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, 
 Scott Reed, Cheng-Yang Fu, Alexander C. Berg as `SSD: Single Shot MultiBox Detector <https://arxiv.org/abs/1512.02325>`_.

diff --git a/docs/examples/video/superres_pytorch/README.rst b/docs/examples/video/superres_pytorch/README.rst
@@ -1,4 +1,3 @@
-
 Example VideoReader usage: Video Super-Resolution
 =================================================
 
@@ -85,11 +84,10 @@ We make use of the FlowNet2-SD PyTorch implementation available `here <https://g
 In order to use the pre-trained FlowNet2-SD network run the following from the
 root directory of this repo:
 
-.. code-block::
+.. code-block:: bash
 
    git clone https://github.com/NVIDIA/flownet2-pytorch
 
-
 Training the VSRNet implemented here requires the use of pre-trained weights from the FlowNet2-SD network.  We provided a converted Caffe pre-trained model below.  Should you use these weights, please adhere to the `license agreement <https://drive.google.com/file/d/1TVv0BnNFh3rpHZvD-easMb9jYrPE2Eqd/view?usp=sharing>`_\ :
 
 `FlowNet2-SD <https://drive.google.com/file/d/1QW03eyYG_vD-dT-Mx4wopYvtPu_msTKn/view?usp=sharing>`_\ [173MB]
@@ -128,11 +126,10 @@ Training
 
 Training can be run by running the following command available in :fileref:`docs/examples/video/superres_pytorch/run.sh`:
 
-.. code-block::
+.. code-block:: bash
 
    ./run.sh
 
-
 This file allows configuration of a variety of
 training options - it is expected that you will modify data paths appropriately
 for your system.
@@ -181,7 +178,7 @@ Reference
 
 If you find this implementation useful in your work, please acknowledge it appropriately and cite the following papers:
 
-.. code-block::
+.. code-block:: BibTeX
 
    @InProceedings{IB17,
      author       = "O. Makansi and E. Ilg and and Thomas Brox",
@@ -192,7 +189,7 @@ If you find this implementation useful in your work, please acknowledge it appro
      url          = "http://lmb.informatik.uni-freiburg.de/Publications/2017/IB17"
    }
 
-.. code-block::
+.. code-block:: BibTeX
 
    @InProceedings{IMKDB17,
      author       = "E. Ilg and N. Mayer and T. Saikia and M. Keuper and A. Dosovitskiy and T. Brox",

diff --git a/docs/supported_op_devices.py b/docs/supported_op_devices.py
@@ -15,6 +15,7 @@ def main(argv):
     doc_table = ''
     doc_table += 'Below table lists all available operators and devices they can operate on.\n\n'
     doc_table += '.. |v| image:: images/tick.gif\n'
+    doc_table += '\n'
     doc_table += formater.format('', '', '', '', '', '', op_name_max_len = op_name_max_len, c='=')
     doc_table += formater.format('Operator name', 'CPU', 'GPU', 'Mixed', 'Support', 'Sequences', op_name_max_len = op_name_max_len, c=' ')
     doc_table += formater.format('', '', '', '', '', '', op_name_max_len = op_name_max_len, c='=')