Update SuperPMI readme (dotnet#723)

* Update SuperPMI readme

Convert it to Markdown and increase the amount of detail describing
the collection process.

* Fix relative link

* Update links, add statement about publishing resultant files.

* More updates

src/coreclr/scripts/superpmi.md

@@ -1,19 +1,19 @@
-### An overview of using superpmi.py
--------------------------
+# An overview of using superpmi.py
 
-General information on [SuperPMI](https://github.com/dotnet/runtime/blob/master/src/coreclr/ToolBox/superpmi/readme.txt)
+General information on [SuperPMI](../src/ToolBox/superpmi/readme.md)
 
-------------------------
+## Overview
 
-#### Overview
-
-Although SuperPMI has many uses, setup and use of SuperPMI is not always trivial. superpmi.py is a tool to help automate the use of SuperPMI, augmenting its usefulness. The tool has three different modes, collect, replay, and asmdiffs. Below you will find more specific information on each of the different modes.
+Although SuperPMI has many uses, setup and use of SuperPMI is not always trivial.
+superpmi.py is a tool to help automate the use of SuperPMI, augmenting its usefulness.
+The tool has three different modes: collect, replay, and asmdiffs.
+Below you will find more specific information on each of the different modes.
 
 **SuperPMI has a large limitation, the replay/asmdiff functionality is sensitive to jitinterface changes. Therefore if there has been an interface change, the new JIT will not load and SuperPMI will fail.**
 
 **At the time of writing collections are done manually on all platforms that support libcoredistools. See before for a full list of supported platforms and where the .mch collection exists.**
 
-#### Supported Platforms
+## Supported Platforms
 
 | OS | Arch | Replay | AsmDiffs | mch location |
 | --- | --- | --- |--- | --- |
@@ -26,7 +26,7 @@ Although SuperPMI has many uses, setup and use of SuperPMI is not always trivial
 | Ubuntu | arm32 |  <ul><li>- [ ] </li></ul> |  <ul><li>- [ ] </li></ul> | N/A |
 | Ubuntu | arm64 |  <ul><li>- [ ] </li></ul> |  <ul><li>- [ ] </li></ul> | N/A |
 
-### Default Collections
+## Default Collections
 
 See the table above for locations of default collections that exist. If there is an mch file that exists, then SuperPMI will automatically download and setup the mch using that location. Please note that, it is possible that the collection is out of date, or there is a jitinterface change which makes the collection invalid. If this is the case, then in order to use the tool a collection will have to be done manually. In order to reproduce the default collections, please see below for what command the default collections are done with.
 
@@ -36,7 +36,7 @@ See the table above for locations of default collections that exist. If there is
 
 `/Users/jashoo/runtime/src/coreclr/scripts/superpmi.py collect bash "/Users/jashoo/runtime/src/coreclr/tests/runtest.sh x64 checked" --skip-cleanup`
 
-**Collect**
+## Collect
 
 Given a specific command collect over all of the managed code called by the child process. Note that this allows many different invocations of any managed code. Although it does specifically require that any managed code run by the child process to handle the complus variables set by SuperPMI and defer them to the later. These are below:
 
@@ -55,11 +55,11 @@ Note that collection will try to run as much managed code as possible. In order
 
 Also note that collection generates gigabytes of data, most of this data will be removed when the collection is finished and de-dupped into a single mch file. That being said, it is worth mentioning that this process will use 3x the size of the unclean mch file, which to give an example of the size, a collection of the coreclr `priority=1` tests uses roughly `200gb` of disk space. Most of this space will be used in a temp directory, which on Windows will default to `C:\Users\blah\AppData\Temp\...`. It is recommended to set the temp variable to a different location before running collect to avoid running out of disk space. This can be done by simply running `set TEMP=D:\TEMP`.
 
-**Replay**
+## Replay
 
 SuperPMI replay supports faster assertion checking over a collection than running the tests individually. This is useful if the collection includes a larger corpus of data that can reasonably be run against by executing the actual code. Note that this is similar to the PMI tool, with the same limitation, that runtime issues will not be caught by SuperPMI replay only assertions.
 
-**AsmDiffs**
+## Asm Diffs
 
 SuperPMI will take two different JITs, a baseline and diff JIT and run the compiler accross all the methods in the mch file. It uses coredistools to do a binary difference of the two different outputs. Note that sometimes the binary will differ, and SuperPMI will be run once again dumping the asm that was output in text format. Then the text will be diffed, if there are differences, you should look for text differences. If there are some then it is worth investigating the asm differences.
 

src/coreclr/src/ToolBox/superpmi/readme.md

@@ -0,0 +1,288 @@
+# Overview
+
+This directory (`src/coreclr/src/ToolBox/superpmi` in the GitHub
+https://github.com/dotnet/runtime repository) contains the SuperPMI
+tool used for testing the .NET just-in-time (JIT) compiler.
+
+## Purpose
+
+SuperPMI has two primary uses:
+1. Verification that a JIT code change doesn't cause any asserts.
+2. Finding test code where two JIT compilers generate different code, or
+verifying that the two compilers generate the same code.
+
+Case #1 is useful for doing quick regression checking when making a source
+code change to the JIT compiler. The process is: (a) make a JIT source code
+change, (b) run that newly built JIT through a SuperPMI run to verify no
+asserts have been introduced.
+
+Case #2 is useful for generating assembly language diffs, to help analyze the
+impact of a JIT code change.
+
+## SuperPMI architecture
+
+SuperPMI works in two phases: collection and playback.
+
+In the collection phase, the system is configured to collect SuperPMI data.
+Then, run any set of .NET managed programs. When these managed programs invoke the JIT
+compiler, SuperPMI gathers and captures all information passed between the
+JIT and its .NET host. This data is post-processed to remove essentially
+duplicate function information, and is collected into one or just a few
+files.
+
+In the playback phase, SuperPMI loads the JIT directly, and causes it to
+compile all the functions that were previously compiled in the collection
+phase, but using the collected data to provide answers to various questions
+that the JIT needs to ask. The .NET execution engine (EE) is not invoked at all.
+When doing playback for assertion checking, only a single JIT is loaded and
+used for compilation. When doing playback to check for assembly diffs, both
+a "baseline" and a "diff" compiler are loaded. Each JIT is asked to compile
+each recorded function. The generated results are compared with a built-in
+SuperPMI "near differ", which depends on an external disassembler component
+called `coredistools`. Typically, scripting has SuperPMI generate a list of
+functions with differences, then the script re-invokes each JIT to generate
+disassembly output (using `COMPlus_JitDisasm`) for each differing function.
+These are then compared either visually, or with the jitutils tool
+`jit-analyze`, or both.
+
+
+# Tools
+
+There are two native executable tools: `superpmi` and `mcs`. To do the collection,
+there is the `superpmi-shim-collector` binary (`.dll` or `.so` or `.dylib`).
+
+To harness collection, there is a .NET Core C# program that is built as
+part of the coreclr tests build called superpmicollect.exe
+(source: src/coreclr/tests/src/JIT/superpmi in https://github.com/dotnet/runtime repository).
+This tool also functions as a SuperPMI collection and playback unit test.
+
+The superpmicollect tool is also being moved to the jitutils repository
+(https://github.com/dotnet/jitutils).
+
+Each tool will show a help screen if passed `-?`.
+
+Finally, there is a Python script that harnesses SuperPMI collection,
+playback, and other functions, such as download of existing SuperPMI
+collections from well-known locations. This tool is `superpmi.py`,
+found in `src/corclr/scripts` in the dotnet/runtime repository. See
+[superpmi.md](../../../scripts/superpmi.md) for more details.
+
+
+# Collection
+
+To manually do a collection (not using the `superpmi.py` script or
+`superpmicollect.exe` tool), follow the following steps.
+
+## Overall collection process
+
+First, build the `dotnet/runtime` repo, which builds the `superpmi`, `mcs`,
+and `superpmi-shim-collector` programs, along with the rest of coreclr,
+and places them in the same native code directory as the JIT and the rest
+of coreclr, e.g., `f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\superpmi.exe`
+for a `dotnet/runtime` repo rooted at the `f:\gh\runtime` directory, and
+built on Windows for the x64 Checked architecture / build flavor combination.
+
+The SuperPMI collection process requires a lot of disk space. How much will
+depend on the number, size, and complexity of the functions the JIT needs to
+compile. Make sure that the disk where the collected data will be collected
+is sufficiently large. It is also best if the data is written on an SSD, to
+speed up the disk operations.
+
+These are the general steps that are followed when doing a SuperPMI collection:
+
+1. Collect .MC files. Set up for collection, then cause the JIT to be invoked
+by either running a scenario, running tests, crossgen compiling assemblies,
+or using PMI to force the JIT to compile functions in an assembly.
+During collection, the data for each JIT compiled function is stored
+in a uniquely named file with a ".MC" filename extension (for "method context").
+2. Merge .MC files to .MCH file. We want all the generated data to be merged to a
+single file (or some smaller set of files), to both be more manageable, and
+to collect the data for multiple compiled functions into one place for easier
+replay. (MCH stands for "method context hive".)
+3. Remove duplicates in the .MCH file. Many compiled functions are essentially
+equivalent, such as trivial class constructors, or if some functions are
+compiled multiple times in different scenarios or tests. We filter out the
+duplicates, which makes playback much faster, and the resultant MCH file much
+smaller.
+4. Create a "clean" .MCH with no SuperPMI failures. The original collected MCH
+file might not replay cleanly. This is generally due to existing, un-investigated
+SuperPMI bugs or limitations. We don't want to see these during normal playback,
+so filter out the failing replays at this point so the "baseline" replay is clean.
+5. Create a table of contents (TOC) file. This creates an index for the generated
+MCH file that greatly speeds up certain operations.
+6. Test final .MCH file is clean. This is purely to check that the resultant
+MCH file is in a good state for future use.
+
+
+## Collect .MC files
+
+Set the following environment variables:
+
+```
+SuperPMIShimLogPath=<full path to an existing, empty temporary directory>
+SuperPMIShimPath=<full path to clrjit.dll, the "standalone" JIT>
+COMPlus_AltJit=*
+COMPlus_AltJitNgen=*
+COMPlus_AltJitName=superpmi-shim-collector.dll
+```
+
+for example, on Windows:
+
+```
+mkdir f:\spmi\temp
+set SuperPMIShimLogPath=f:\spmi\temp
+set SuperPMIShimPath=f:\gh\runtime\artifacts\tests\coreclr\Windows_NT.x64.Checked\Tests\Core_Root\clrjit.dll
+set COMPlus_AltJit=*
+set COMPlus_AltJitNgen=*
+set COMPlus_AltJitName=superpmi-shim-collector.dll
+```
+
+(On Linux, use `libclrjit.so` and `libsuperpmi-shim-collector.so`.
+On Mac, use `libclrjit.dylib` and `libsuperpmi-shim-collector.dylib`.)
+
+Note that the `superpmi-shim-collector.dll` must live in the same directory as the `coreclr.dll`
+(or libcoreclr.so on Linux) that will be invoked when running .NET Core. This will normally be
+a `Core_Root` directory, since you must create such a directory to be able to run
+.NET Core applications (such as by using the `corerun` tool).
+
+If you want to collect using an official .NET Core build, you will need to build a matching set
+of superpmi binaries, and copy `superpmi-shim-collector.dll` to the correct directory. This
+option has not been tested (as far as I know).
+
+Then, cause the JIT to compile some code. Do one or more of:
+1. Run a managed scenario.
+2. Run managed code tests.
+3. Crossgen some assemblies.
+4. Run PMI over some assemblies (see https://github.com/dotnet/jitutils for details on PMI)
+
+When done running programs, un-set these environment variables.
+
+Now, you will have a large number of .MC files in the specified temporary
+directory (specified by `SuperPMIShimLogPath`).
+
+
+## Merge .MC files to .MCH file
+
+Merge the generated .MC files using the `mcs` tool:
+
+```
+mcs -merge base.mch *.mc -recursive
+```
+
+This assumes the current directory is the root directory where the .MC files
+were placed, namely the directory specified as `SuperPMIShimLogPath` above.
+You can also specify a directory prefix to the file system regular expression.
+The `-recursive` flag is only necessary if .MC files also exist in subdirectories
+of this, and you want those also added to the resultant, collected `base.mch`
+file. So, for the example above, you might use:
+
+```
+f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\mcs.exe -merge f:\spmi\base.mch f:\spmi\temp\*.mc -recursive
+```
+
+Note that `mcs -merge` is literally just a file concatenation of many files into
+one, so this step will double the required disk space.
+
+After this step, you can remove all the individual .MC files unless you want
+to keep them to debug the SuperPMI collection process itself.
+
+
+## Remove duplicates in the .MCH file
+
+One benefit of SuperPMI is the ability to remove duplicated compilations, so
+on replay only unique functions are compiled. Use the following to create a
+"unique" set of functions:
+
+```
+mcs -removeDup -thin base.mch unique.mch
+```
+
+Note that `-thin` is not required. However, it will delete all the compilation
+results collected during the collection phase, which makes the resulting MCH
+file smaller. Those compilation results are not required for playback for
+the ways in which we normally use SuperPMI.
+
+For the continuing example, you might use:
+
+```
+f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\mcs.exe -removeDup -thin f:\spmi\base.mch f:\spmi\unique.mch
+```
+
+After this step, you can remove the base.mch file (unless you want to debug
+the SuperPMI collection process itself).
+
+
+## Create a "clean" .MCH with no SuperPMI failures
+
+As stated above, due to various bugs or otherwise uninvestigated issues, a SuperPMI
+replay of the unique.mch file might contain errors. We don't want that, so we filter
+out those errors in a "baseline" run, as follows.
+
+```
+superpmi -p -f basefail.mcl unique.mch clrjit.dll
+mcs.exe -strip basefail.mcl unique.mch final.mch
+```
+
+Or, continuing the example above, giving full paths, we have:
+
+```
+f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\superpmi.exe -p -f f:\spmi\basefail.mcl f:\spmi\unique.mch f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\clrjit.dll
+f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\mcs.exe -strip f:\spmi\basefail.mcl f:\spmi\unique.mch f:\spmi\final.mch
+```
+
+
+## Create a table of contents (TOC) file
+
+This is
+
+```
+mcs -toc final.mch
+```
+
+or, using the full paths from above:
+
+```
+f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\mcs.exe -toc f:\spmi\final.mch
+```
+
+
+## Test final .MCH file is clean
+
+This is done using a replay just like for the "create clean .MCH file" step:
+
+```
+superpmi -p -f finalfail.mcl final.mch clrjit.dll
+```
+
+Or, continuing the example above, giving full paths, we have:
+
+```
+f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\superpmi.exe -p -f f:\spmi\finalfail.mcl f:\spmi\final.mch f:\gh\runtime\artifacts\bin\coreclr\Windows_NT.x64.Checked\clrjit.dll
+```
+
+In this case, if `finalfail.mcl` is not empty, there was a failure in the final "check" replay.
+This will lead to the same failure in all future replays, if the resultant final.mch file is
+published for general use. This is annoying, but not necessarily a fatal problem.
+It's possible to cycle the process, stripping this failure from the final MCH,
+testing for "clean-ness" again, and repeating. It should not be necessary, however.
+
+
+## Publishing the SuperPMI collection
+
+The files that should be published for consumption either by yourself or by a group
+are `final.mch` and `final.mch.mct` (the TOC). All other files described above are
+intermediate files in the collection process, and are not needed afterwards.
+
+
+# Playback
+
+Once you have a merged, de-duplicated MCH collection, you can play it back
+using:
+
+    superpmi -p final.mch clrjit.dll
+
+The `-p` switch says to utilize all the processors on your machine,
+and replay in parallel. You can omit this if you wish to replay
+without using parallelism (this is usually done when replaying
+a single compilation, whereas `-p` is usually used when replaying
+a full MCH file set of compilations).