[Doc] Fix profiling.rst for the profiling (ray-project#43615)

1. Currently the profiling docs for CPU is not correct, i.e., we need to add "export RAY_RAYLET_PERFTOOLS_PROFILER=1" so that we can profile on google-pprof.
2. Split clearly the CPU profiling and memory profiling methods.
3. To correct install the jemalloc, make install is also needed to correctly install jemalloc.
4. To visualized the plot, the graphviz is also needed otherwise there is a issue for jemalloc which doesn't show this problem clearly.
5. Add how to get the .svg plot for memory profile.

Signed-off-by: Bye-legumes <[email protected]>
Signed-off-by: Jiajun Yao <[email protected]>
Co-authored-by: Jiajun Yao <[email protected]>
Co-authored-by: angelinalg <[email protected]>

doc/source/ray-contribute/profiling.rst

@@ -29,24 +29,60 @@ symbolize on Mac OS have failed.
 
   sudo apt-get install google-perftools libgoogle-perftools-dev
 
-Launching the to-profile binary
--------------------------------
+You may need to install ``graphviz`` for ``pprof`` to generate flame graphs.
 
-If you want to launch Ray in profiling mode, define the following variables:
+.. code-block:: bash
+  
+  sudo apt-get install graphviz
+
+CPU profiling
+-------------
+
+To launch Ray in profiling mode and profile Raylet, define the following variables:
 
 .. code-block:: bash
 
   export PERFTOOLS_PATH=/usr/lib/x86_64-linux-gnu/libprofiler.so
   export PERFTOOLS_LOGFILE=/tmp/pprof.out
+  export RAY_RAYLET_PERFTOOLS_PROFILER=1
 
 
-The file ``/tmp/pprof.out`` will be empty until you let the binary run the
+The file ``/tmp/pprof.out`` is empty until you let the binary run the
 target workload for a while and then ``kill`` it via ``ray stop`` or by
-letting the driver exit.
+letting the driver exit. 
+
+Note: Enabling `RAY_RAYLET_PERFTOOLS_PROFILER` allows profiling of the Raylet component. 
+To profile other modules, use `RAY_{MODULE}_PERFTOOLS_PROFILER`,
+where `MODULE` represents the uppercase form of the process type, such as `GCS_SERVER`.
+
 
-Memory Profiling
+Visualizing the CPU profile
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can visualize the output of ``pprof`` in different ways. Below, the output is a
+zoomable ``.svg`` image displaying the call graph annotated with hot paths.
+
+.. code-block:: bash
+
+  # Use the appropriate path.
+  RAYLET=ray/python/ray/core/src/ray/raylet/raylet
+
+  google-pprof -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
+  # Then open the .svg file with Chrome.
+
+  # If you realize the call graph is too large, use -focus=<some function> to zoom
+  # into subtrees.
+  google-pprof -focus=epoll_wait -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
+
+Below is a snapshot of an example SVG output, from the official
+documentation:
+
+.. image:: http://goog-perftools.sourceforge.net/doc/pprof-test-big.gif
+
+Memory profiling
 ----------------
-If you want to run memory profiling on Ray core components, you can use Jemalloc (https://github.com/jemalloc/jemalloc).
+
+To run memory profiling on Ray core components, use Jemalloc (https://github.com/jemalloc/jemalloc).
 Ray supports environment variables to override LD_PRELOAD on core components.
 
 You can find the component name from `ray_constants.py`. For example, if you'd like to profile gcs_server, 
@@ -66,43 +102,37 @@ Users are supposed to provide 3 env vars for memory profiling.
   cd jemalloc-5.2.1 
   ./configure --enable-prof --enable-prof-libunwind 
   make
+  make install
 
-  # set jemalloc configs through MALLOC_CONF env variable
-  # read http://jemalloc.net/jemalloc.3.html#opt.lg_prof_interval
+
+  # Set jemalloc configs through MALLOC_CONF env variable.
+  # Read http://jemalloc.net/jemalloc.3.html#opt.lg_prof_interval.
   # for all jemalloc configs
   # Ray start will profile the GCS server component.
   RAY_JEMALLOC_CONF=prof:true,lg_prof_interval:33,lg_prof_sample:17,prof_final:true,prof_leak:true \
   RAY_JEMALLOC_LIB_PATH=~/jemalloc-5.2.1/lib/libjemalloc.so \
   RAY_JEMALLOC_PROFILE=gcs_server \
   ray start --head
 
-  # You should be able to see the following logs.
+  # You should see the following logs.
   2021-10-20 19:45:08,175	INFO services.py:622 -- Jemalloc profiling will be used for gcs_server. env vars: {'LD_PRELOAD': '/Users/sangbincho/jemalloc-5.2.1/lib/libjemalloc.so', 'MALLOC_CONF': 'prof:true,lg_prof_interval:33,lg_prof_sample:17,prof_final:true,prof_leak:true'}
 
-Visualizing the CPU profile
----------------------------
-
-The output of ``pprof`` can be visualized in many ways. Here we output it as a
-zoomable ``.svg`` image displaying the call graph annotated with hot paths.
+Visualizing the memory profile
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The output files are at the path where you call "ray start".
+For the example of profile file "jeprof.15786.0.f.heap", use following commands to generate the .svg plot.
 
 .. code-block:: bash
 
   # Use the appropriate path.
   RAYLET=ray/python/ray/core/src/ray/raylet/raylet
 
-  google-pprof -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
+  sudo jeprof $RAYLET jeprof.15786.0.f.heap --svg > /tmp/prof.svg
   # Then open the .svg file with Chrome.
 
-  # If you realize the call graph is too large, use -focus=<some function> to zoom
-  # into subtrees.
-  google-pprof -focus=epoll_wait -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
-
-Here's a snapshot of an example svg output, taken from the official
-documentation:
 
-.. image:: http://goog-perftools.sourceforge.net/doc/pprof-test-big.gif
 
-Running Microbenchmarks
+Running microbenchmarks
 -----------------------
 
 To run a set of single-node Ray microbenchmarks, use:
@@ -118,4 +148,4 @@ References
 
 - The `pprof documentation <http://goog-perftools.sourceforge.net/doc/cpu_profiler.html>`_.
 - A `Go version of pprof <https://github.com/google/pprof>`_.
-- The `gperftools <https://github.com/gperftools/gperftools>`_, including libprofiler, tcmalloc, and other goodies.
+- The `gperftools <https://github.com/gperftools/gperftools>`_, including libprofiler, tcmalloc, and other useful tools.