Merge branch 'master' of /home/vihofer/consolidation/artifacts/coreclr

docs/coreclr/Profiling/IL Rewriting Basics.md

@@ -0,0 +1,23 @@
+# IL Rewriting Basics
+
+## Intro
+One of the common use cases of the `ICorProfiler*` interfaces is to perform IL rewriting. Some possible reasons a profiler would want to rewrite IL:
+- Inspecting interesting process state
+  - Capturing exception state
+  - Inspecting managed objects
+  - Inspecting function arguments/return values
+- Injecting method hooks at the start/end of the method that call in to another managed library
+
+There are two ways to rewrite IL
+
+1. At Module load time with `ICorProfilerInfo::SetILFunctionBody`
+    This approach has the benefit that it is 'set it and forget it'. You can replace the IL at module load, and the runtime will treat this new IL as if the module contained that IL - you don't have to worry about any of the quirks of ReJIT. The downside is that is is unrevertable - once it is set, you cannot change your mind.
+
+2. At any point during the process lifetime with `ICorProfilerInfo4::RequestReJIT` or `ICorProfilerInfo10::RequestReJITWithInliners`.
+   This approach means that you can modify functions in response to changing conditions, and you can revert the modified code if you decide you are done with it. See the other entries about ReJIT in this folder for more information.
+
+## How to rewrite IL
+Hopefully this section will be fleshed out in the future. Right now we have some documentation in the archives at [Creating an IL-rewriting profiler](<./davbr-blog-archive/Creating an IL-rewriting profiler.md>), but there is no start to finish tutorial on IL rewriting.
+
+## What if multiple profilers want to rewrite IL in a given process?
+The `ICorProfiler*` interfaces do not provide a way to multiplex different profilers, only one profiler can be loaded at a time. The [CLR Instrumentation Engine](https://github.com/microsoft/CLRInstrumentationEngine) project was created to address this limitation. If you are concerned about profiler multiplexing, head over and check out the project. A short summary is that it provides a higher level interface than the `ICorProfiler*` interfaces, and also provides way for multiple profilers to interact in a well defined manner.

docs/coreclr/Profiling/Profiler Attach on CoreCLR.md

@@ -0,0 +1,28 @@
+
+# Profiler Attach on CoreCLR
+
+Starting with .Net Core 3 preview6 there is a new profiler attach mechanism for CoreCLR. The desktop .Net Framework has had profiler attach since v4, but we have not had profiler attach on .Net Core. The desktop implementation was very Windows-centric and we did not have a good way to offer a cross platform profiler attach mechanism. The recent diagnostics port work means there is now a cross platform communication channel for external processes to communicate with a running CoreCLR process, and this allowed us to finally offer profiler attach for CoreCLR.
+
+## How do you attach a profiler to a running CoreCLR process?
+
+***Disclaimer: the code in the dotnet/diagnostics repo referred to below is in prelease and is under active development. You should expect things to change before the official release.***
+
+Attaching a profiler to a running CoreCLR process involves sending a message from an external process (the trigger process) on the diagnostics port telling the runtime which profiler to attach. We have a premade managed implementation over at the [Diagnostics repo](https://github.com/dotnet/diagnostics). The attach method is `DiagnosticHelpers.AttachProfiler` in the `Microsoft.Diagnostics.Tools.RuntimeClient` library, which will be shipped on NuGet once it is released. It takes five arguments:
+
+1) `int processId`          - (Required) The process ID to attach to.
+2) `uint attachTimeout`     - (Required) A timeout that informs the runtime how long to wait while attempting to attach. This does not impact the timeout of trying to send the attach message.
+3) `Guid profilerGuid`      - (Required) The profiler's GUID to use when initializing.
+4) `string profilerPath`    - (Required) The path to the profiler on disk.
+5) `byte[] additionalData`  - (Optional) A data blob that will be passed to `ICorProfilerCallback3::InitializeForAttach` as `pvClientData`. 
+
+This method returns a status HR following the usual convention, 0 (S_OK) means a profiler was successfully attached and any other value is an error indicating what went wrong.
+
+## What if you can't run managed code in your trigger process?
+
+If you are unable to run managed code as part of your trigger process, it is still possible to request a profiler attach. The spec for the diagnostics port is located [here](https://github.com/dotnet/diagnostics/blob/master/documentation/design-docs/ipc-protocol.md).
+
+You will have to do the following (according to the above spec):
+1) Open the appropriate channel - domain socket on Linux and a named pipe on Windows
+2) Construct the payload with the appropriate command (Profiler) and command ID (AttachProfiler), plus all of the arguments listed above
+3) Send the payload over the channel
+4) Parse the response

docs/coreclr/Profiling/Profiler Breaking Changes.md

@@ -0,0 +1,7 @@
+# Profiler Breaking Changes #
+
+Over time we will need to modify the Profiler APIs, this document will serve as a record of any breaking changes.
+
+1. Code Versioning introduced changes documented [here](../design-docs/code-versioning-profiler-breaking-changes.md)
+2. The work to allow adding new types and methods after module load means ICorProfilerInfo7::ApplyMetadata will now potentially trigger a GC, and will not be callable in situations where a GC can not happen (for example  ICorProfilerCallback::RootReferences).
+3. As part of the work to allow ReJIT on attach ReJITted methods will no longer be inlined (ever). Since the inlining is blocked there won't be a `ICorProfilerCallback::JITInlining` callback.

docs/coreclr/Profiling/Profiler Loading.md

@@ -0,0 +1,23 @@
+
+To enable profiling set the following environment variables:
+- `CORECLR_ENABLE_PROFILING=1`
+- `CORECLR_PROFILER={_CLSID of profiler_}`
+
+# Finding the profiler library
+Once profiling is enabled there are two ways we load your profiler, with enviroment variables (cross-plat) or through the registry (Windows only)
+
+## Environment Variable (cross-plat)
+Set one of the following (if all are set, the bitness-specific variables take precedence). The 32/64 ones specify which bitness of profiler is loaded
+- `CORECLR_PROFILER_PATH=full path to your profiler's DLL`
+- `CORECLR_PROFILER_PATH_32=full path to your profiler's DLL`
+- `CORECLR_PROFILER_PATH_64=full path to your profiler's DLL`
+
+If any of these environment variable are present, we skip the registry look up altogether, and just use the path from `CORECLR_PROFILER_PATH` to load your DLL.
+
+A couple things to note about this:
+- If you specify `CORECLR_PROFILER_PATH` _and_ register your profiler, then `CORECLR_PROFILER_PATH` always wins.  Even if `CORECLR_PROFILER_PATH` points to an invalid path, we will still use `CORECLR_PROFILER_PATH`, and just fail to load your profiler.
+- `CORECLR_PROFILER` is _always required_.  If you specify `CORECLR_PROFILER_PATH`, we skip the registry look up. We still need to know your profiler's CLSID, so we can pass it to your class factory's CreateInstance call.
+
+
+## Through the registry (Windows Only)
+If the `CORECLR_PROFILER_PATH*` environment variables above are not set (and you're running on Windows) then coreclr will look up the CLSID from `CORECLR_PROFILER` in the registry to find the full path to your profiler's DLL. Just like with any COM server DLL, we look for your profiler's CLSID under HKEY_CLASSES_ROOT, which merges the classes from HKLM and HKCU.

docs/coreclr/Profiling/ReJIT on Attach.md

@@ -0,0 +1,71 @@
+# ReJIT on Attach
+
+A longstanding feature request we've had from profiler authors is the ability to ReJIT a method after attach. There are non-trivial technical reasons this was never an option in desktop .Net, but CoreCLR has had some changes that made it more feasible to attain. As of .Net Core 3 preview5 profiler authors now have the ability to ReJIT methods after attach.
+
+## The new API
+
+To enable ReJIT on attach there is a new API `ICorProfilerInfo10::RequestReJITWithInliners`. Here is the signature:
+
+```cpp
+    HRESULT RequestReJITWithInliners(
+                [in]                       DWORD       dwRejitFlags,
+                [in]                       ULONG       cFunctions,
+                [in, size_is(cFunctions)]  ModuleID    moduleIds[],
+                [in, size_is(cFunctions)]  mdMethodDef methodIds[]);
+```
+
+Conceptually this works the same as `ICorProfilerInfo4::RequestReJIT` except it will automatically ReJIT any methods that have inlined the target method(s) in the past. The arguments are the same except for the additon of `dwRejitFlags` as the first parameter. The valid values for this argument come from this enum:
+
+```cpp
+typedef enum
+{
+    // ReJITted methods will be prevented from being inlined
+    COR_PRF_REJIT_BLOCK_INLINING = 0x1,
+
+    // This flag controls whether the runtime will call GetReJITParameters
+    // on methods that are ReJITted because they inline a method that was requested
+    // for ReJIT
+    COR_PRF_REJIT_INLINING_CALLBACKS    = 0x2
+} COR_PRF_REJIT_FLAGS;
+```
+
+Any callers of this API must set `COR_PRF_REJIT_BLOCK_INLINING`. Although it is possible that in the future this restriction will be lifted, the current implementation blocks ReJITted methods from being inlined (ever).
+
+The other value `COR_PRF_REJIT_INLINING_CALLBACKS` controls whether you get a `ICorProfilerCallback4::GetReJITParameters` callback for any methods that are ReJITted as inliners of the requested method. The default is to not receive callbacks for these methods. You will always receive a `GetReJITParameters` callback for any methods that are explictly requested.
+
+
+## Inner workings/Limitations
+
+With this API you are no longer required to monitor JIT callbacks to manually block inlining from occurring. To achieve that the runtime now globally blocks a ReJITted method from being inlined (even if it was ReJITted with `ICorProfilerInfo4::RequestReJIT` and not the new API). Once a method is reverted with `ICorProfilerInfo4::RequestRevert` inlining will occur again for any future jittings.
+
+It is important to mention here how `RequestRevert` works. When you revert a ReJITted method, the original native code is activated. This means there are potential pitfalls for calling `RequestRevert`. Consider an app where method A inlines method B and the profiler wants to ReJIT both A and B. Once A and B are both ReJITted, the application will behave as expected. However, if later on the profiler decides to revert method A but intends to leave method B ReJITted, it might be surprising to find that once the original native code for A is activated this includes the inlined non-ReJIT IL for method B. Effectively any calls to B through A will be calling the original, unmodified IL.
+
+To revert a method without having to reason about the inline sequence, we suggest calling RequestReJIT again on the method but providing the original IL in GetReJITParameters.
+
+The limitation of collectible and dynamic methods has not been lifted. It is not currently possible to ReJIT these types of methods, although we would like to lift that restriction in the future. Even if you never intend to call RequestReJIT directly on a collectible or dynamic method, this may still affect you when doing ReJIT on attach if the method you would like to ReJIT has been inlined in a collectible or dynamic method. I.e. if you would like to ReJIT method A which has been inlined in collectible method B, there is currently no way to make method B call the updated method A.
+
+## Metadata Changes on Attach
+
+Usually profiler authors do not want to trivially change the IL for ReJITted methods, but rather inject new types/methods and call those new types/methods. Previously our guidance has been to do any metadata rewriting during the `ICorProfilerCallback::ModuleLoadFinished` callback. This advice presents a challenge if the profiler is not attached during module load and still wants to modify metadata.
+
+To work around this a set of metadata changes is now legal to make at any point as long as you call `ICorProfilerInfo7::ApplyMetadata` afterwards:
+* `DefineUserString`
+* `DefineTypeRefByName`
+* `DefineMemberRef`
+* `DefineTypeDef`
+* `DefineMethod`            - methods on new types, or non-virtual methods on existing types
+* `DefineNestedType`        - only on new types
+* `DefineCustomAttribute`
+* `DefinePinvokeMap`
+* `DefineModuleRef`
+* `DefineField`             - only on new types
+* `DefineEvent`
+* `DefineMethodImpl`        - only on new types
+* `DefineMethodSpec`
+
+
+There are still some metadata changes that are definitely illegal and will almost certainly never become legal due to restrictions/assumptions existing in the runtime:
+* Adding a virtual method to an existing type
+* Adding a field to an existing type
+
+Anything not listed as either legal or illegal is untested and should not be assumed to work.