Merge pull request worstcase#8 from newbrough/feature/more-docs

Feature/more docs

README.md

@@ -5,38 +5,52 @@ Gumshoe Load Investigator
 Overview
 --------
 
-This tool profiles I/O operations and presents statistics over time per calling stack as a flame graph or root tree.
+Monitor application performance statistics associated with individual calling stacks,
+interactively filter and view as a flame graph or root graph.
 
 Gumshoe was first created initially for internal use in the Dell Cloud Manager application but
 source code has since been released for public use under [these terms](LICENSE.rst).  
 
-Packages
+[Features](docs/features.md)
 --------
 
-* gumshoe-hooks
+* Analyze resource usage: measure TCP, UDP, filesystem or processor utilization
+* Pinpoint lines of code: all statistics are associated with a call stack and individual stack frames
+* Capture, filter and visualize as statistics are generated
+* Intuitive views: flame graph and root graph
+* Filter stack frames at capture and/or during visualization, modify on the fly. 
 
-    A very small set of classes that must be loaded as part of the JVM bootclasspath to capture raw I/O.
+Documentation
+-------------
 
-* gumshoe-probe
+* Short intro and demo on youtube: [latest](https://www.youtube.com/watch?v=GGJFZfwXJ44) or the original [boring version](https://www.youtube.com/watch?v=1M9GX4ENMeI).
+* [Quick start guide](QUICK-START.md) walks through using with a sample application.
+* Full [user guide](docs/index.md)
 
-    A queue and filter system to queue, filter and summarize I/O events and pass results to listeners.
+Don't Just Measure and Report:  Understand
+------------------------------------------
 
-* gumshoe-tools
+Looking at stack traces scaled by metrics gives a fast, intuitive way to understand application resource usage.  Consider
+network I/O in a hypothetical application...
 
-    A swing GUI to configure the probe and display and manipulate results. 
+Flame graph example:
 
-Features
---------
+![image](docs/flame-graph.png)
 
-Capture and visualize live socket I/O statistics and identify what is causing it.
-View flame graph or root graph representation.
-Filter stack frames at capture and/or during visualization, modify on the fly. 
+Looking at callers into an application, Invoker.invoke is responsible for >90% of the read operations
+in this application.  It always calls RestRequestHandler.handle which in turn calls three different REST
+operations.  One -- showItemDetails -- is responsible for >80% of read operations as it makes different calls
+into the DAO layer.
 
-Documentation
--------------
+Root graph example:
 
-* Short intro and demo on youtube: [latest](https://www.youtube.com/watch?v=GGJFZfwXJ44) or the original [boring version](https://www.youtube.com/watch?v=1M9GX4ENMeI).
+![image](docs/root-graph.png)
 
-* [Quick start guide](QUICK-START.md) walks through using with a sample application.
+Grouping now by the last frame where the application calls out to a resource,
+the couchdb Database.getDocument is responsible for over half of the read operations,
+while the JDBC Statement.execute about 25%, the vast majority of those from getItemDetails.
 
-* [User guide](docs/index.md)
+These two examples are completely contrived, but not overly simplified.  The original call stacks in
+your application are generally huge, with hundreds of frames,
+but gumshoe uses [stack filters](docs/filters.md) to find
+just the relevant portions and focus the view on just those parts.

docs/features.md

@@ -1,26 +1,27 @@
 Gumshoe Features
 ================
 
-Measure CPU Usage
------------------
+Data Collection
+---------------
 
-Measure TCP Socket I/O
-----------------------
+These kinds of resource usage are monitored and reported:
+- [socket I/O (TCP)](types/socket-io.md)
+- [file I/O](types/file-io.md)
+- [CPU utilization](types/cpu-stats.md)
+- [datagram I/O (UDP)](types/datagram-io.md)
+- [unclosed socket detection](types/socket-unclosed.md)
 
-Measure File I/O
-----------------
-
-Measure UDP Datagram I/O
-------------------------
+Visualization
+-------------
 
-Detect Unclosed Sockets
------------------------
-
-Flame Graph
------------
-
-Root Graph
-----------
+All data collection is associated with a specific call stack, so information is not "system-wide"
+but tied to individual threads and method calls.  If multiple threads include the same method call,
+resource usage can be combined across threads for a total for that method call over all similar threads.
+This can be presented by combining identical frames starting at the bottom of the call stack, 
+which results in a flame graph; or starting at the bottom, which results in a root graph.
 
 Live capture and view 
 ---------------------
+
+Often data is collected from the target application in a text file and analyzed later with the gumshoe viewer.
+However, the viewer can also be launched from within the same JVM and data viewed as it is collected.  

docs/filters.md

@@ -69,16 +69,6 @@ There are four levels of simplification that can be performed:
     NO_METHOD           drop line number and method name
     NO_INNER_CLASSES    drop line number, method and inner classes
     NO_CLASSES          keep only the package name
-    NONE                do not simplify frame
-
-Recursion
----------
-
-Recursive code can lead to especially long stack traces that repeat the same frame or sequence of frames
-over and over.  The recursion filter will remove repeated sequences up to a configurable length.
-There may be a lot of overhead comparing frames for each stack generated, and the overhead is directly
-related to the length chosen.  Use the recursion filter sparingly and with as low a length value as
-reasonable.
 
 Empty Stacks
 ------------

docs/hooks.md

@@ -0,0 +1,19 @@
+Gumshoe Hooks
+=============
+
+Hooks are techniques used by gumshoe to collect information from the JVM that is filtered, accumulated and
+finally reported by probes.  Some probes may share a hook, and some probes may work without a hook.
+
+List of hooks:
+
+[IoTrace](hooks/io-trace.md) is packaged in its own jar and must be loaded into the
+bootclasspath of the JVM to support socket I/O and file I/O probes.
+
+[SelectorProviderImpl](hooks/selector-provider.md) requires a System property
+during startup to override the default factory to support NIO monitoring in
+the socket I/O and datagram I/O probes.
+
+[SocketFactoryImpl](hooks/socket-factory.md) is installed by the unclosed socket probe.
+
+[DatagramFactoryImpl](hooks/datagram-socket-factory.md) is installed by the datagram I/O probe.
+

docs/hooks/datagram-socket-factory.md

@@ -0,0 +1,9 @@
+Datagram I/O Hook
+=================
+
+Summary
+-------
+
+The DatagramSocketFactoryImpl mechanism was used to define wrapper classes for regular (non-NIO) 
+datagram sockets that report I/O statistics to IoTraceAdapter.  The wrappers invoke the actual datagram operations
+on the original default factory implementation while tracking operations, size and elapsed time.

docs/hook.md → docs/hooks/io-trace.md

@@ -1,4 +1,4 @@
-Gumshoe Hook
+IoTrace Hook
 ============
 
 Summary
@@ -12,7 +12,8 @@ To collect socket or file I/O statistics, the target JVM must include the option
 Details
 -------
 
-At its core, gumshoe collects I/O data using a callback mechanism built into the JRE.
+At its core, gumshoe collects socket and file I/O data using a callback 
+mechanism built into the JRE.
 Socket and file operations make a call to an empty class IoTrace before and after
 each read or write operation.  The gumshoe hook replaces this empty class with one
 that can report the details of those operations and the call stack where it occurred.
@@ -26,7 +27,11 @@ Other kinds of statistics can still be collected.
 Having the hook in place will allow applications to start and stop monitoring
 as needed, and it will introduce almost no overhead when not monitoring.
 Specifically, instead of the default empty method in IoTrace, the replacement
-[IoTrace](../gumshoe-hooks/src/main/java/sun/misc/IoTrace.java) 
+[IoTrace](../../gumshoe-hooks/src/main/java/sun/misc/IoTrace.java) 
 methods make one call into a 
 [null object](https://en.wikipedia.org/wiki/Null_Object_pattern) IoTraceAdapter
-with an empty method.  
+
+The IoTraceAdapter used provides additional methods to collect datagram I/O
+using similar methods.  The datagram hooks report directly to this adapter
+and not sun.misc.IoTrace, however, so the original class signature of
+IoTrace is unchanged.

docs/hooks/selector-provider.md

@@ -0,0 +1,13 @@
+NIO Monitoring Hook
+===================
+
+The NIO SelectorProviderImpl mechanism is used to define wrapper classes that add missing IoTrace callsback
+to NIO socket non-blocking reads and IoTraceAdapter callsbacks to datagram NIO operations.  IO is passed
+to the original default implementation and the wrapper only provides event reporting.
+
+Without the hook, all NIO socket writes, and blocking NIO socket reads and all file I/O are still reported.
+The hook is needed for non-blocking socket reads and datagram I/O only.  
+
+To use the hook, the JVM must start with system property:
+
+    java.nio.channels.spi.SelectorProvider=com.dell.gumshoe.network.IoTraceSelectorProvider

docs/hooks/socket-factory.md

@@ -0,0 +1,9 @@
+Unclosed Socket Hook
+====================
+
+Summary
+-------
+
+The SocketFactoryImpl mechanism was used to define wrapper classes for regular (non-NIO) 
+sockets that track closure and report sockets left open.  I/O operations are passed to
+the original default factory generated sockets.

docs/index.md

@@ -27,7 +27,7 @@ Getting Started
 Collecting Samples
 ------------------
 
-- [About the hook](hook.md)
+- [About the hooks](hooks.md)
 - [Configuring the probe](probe.md)
 - [Using filters](filters.md)
 - [Running your program](run.md)

docs/probe.md

@@ -17,7 +17,7 @@ The Probe is used to initialize and begin monitoring:
 
 It can stop monitoring or update configuration:
 - Explicitly invoke Probe methods from your application
-- [Invoke JMX operations](jmx.md)
+- Invoke JMX operations
 
 Handling Data
 -------------
@@ -34,9 +34,9 @@ The values will install JVM hooks, manage filters and data collection,
 and determine when and where results are reported.
 
 For details:
-- [Properties for socket I/O reporting](properties-socket-io.md)
-- [Properties for CPU usage reporting](properties-cpu-stats.md)
-- [Properties for unclosed socket reporting](properties-unclosed-socket.md)
+- [Properties for socket I/O reporting](probe/properties-socket-io.md)
+- [Properties for CPU usage reporting](probe/properties-cpu-stats.md)
+- [Properties for unclosed socket reporting](probe/properties-unclosed-socket.md)
 
 Managing Configuration with JMX
 -------------------------------
@@ -59,6 +59,6 @@ and connect to your JVM using a JMX client such as jconsole.
 By default, the MBeans installed will have names beginning with "com.dell.gumshoe".  
 
 For details:
-- [Socket I/O MBean](jmx-socket-io.md)
-- [CPU Stats MBean](jmx-cpu-stats.md)
-- [Unclosed Socket MBean](jmx-unclosed-socket.md) 
+- [Socket I/O MBean](probe/jmx-socket-io.md)
+- [CPU Stats MBean](probe/jmx-cpu-stats.md)
+- [Unclosed Socket MBean](probe/jmx-unclosed-socket.md) 

docs/probe/event-handling.md

@@ -0,0 +1,100 @@
+I/O Probe Event Handler
+=======================
+
+The socket, datagram and file I/O probes each create entries in a (separate) bounded queue 
+for each read and write operation to offload statistics tabulation from the thread 
+performing I/O.  If the consumer of the thread is not able to keep up with the events
+being generated, the queue will reach capacity and events may be lost.
+
+Monitoring the Queue
+--------------------  
+
+If an event queue fills, a message is displayed on STDOUT such as:
+
+    GUMSHOE: IO event queue for SocketIOMonitor is full
+
+If this message is not found in STDOUT, then no further investigation or configuration is needed.
+If it is seen, use JMX MBeans to determine how many events are being lost.
+
+SocketIOProbe, DatagramIOProbe and FileIOProbe each have corresponding MBeans with attributes and operations:
+
+    QueueStatisticsEnabled  set true to collect number of events dropped and average queue size
+    QueueStats              text description of average and max queue size, number of events and % dropped
+    resetQueueStats()       operation to reset counters to zero
+
+Also to enable collecting queue statistics during startup, System properties can be used:
+
+    gumshoe.socket-io.handler.stats-enabled=true    
+    gumshoe.datagram-io.handler.stats-enabled=true    
+    gumshoe.file-io.handler.stats-enabled=true    
+
+Using these controls, enable statistics collection and monitor the target application for some time.
+Then look at QueueStats to see what portion of events are dropped.
+
+Live with It?
+-------------
+
+In most cases, dropping even a significant fraction of the events is not a problem.  The events 
+received should still be a representative sample of all I/O and although the total counts and number of bytes
+may not be accurate, the relative weight of each stack and frame should still be usable.
+
+Deal with It!
+-------------
+
+If you decide you are dropping more events than you are comfortable with, there are a number of ways
+to improve reporting.
+
+  - if there are only intermittent periods of dropped events, increasing queue size may be enough to
+    let the consumer handle events in between peaks.  Use properties to set the queue size (default is 500):
+
+    gumshoe.socket-io.handler.queue-size=1000    
+    gumshoe.datagram-io.handler.queue-size=1000    
+    gumshoe.file-io.handler.queue-size=1000    
+
+
+
+  The queue will fill if events are produced (individual I/O operations) faster than they are consumed.
+  Some possible reasons:
+
+  - The target application is performing a lot of small network operations
+
+    This could be an area to improve the target application.  
+    Lots of small operations are less efficient than fewer large operations.
+
+    Or this could just be the nature of the expected application load,
+    so increase the gumshoe event queue size and the handler thread priority to accommodate. 
+
+  - The JVM is CPU bound
+
+    The event queue may back up if the target application is CPU bound.  This could be
+    an issue in the target application itself, and you may want to look at
+    [processor utilization statistics](../types/cpu-stats.md) before socket I/O.
+
+    Or it could be due to gumshoe stack filters.  Each stack filter configured has to
+    modify the event call stack on the same event handling thread.  Complex filters
+    (such as the recursion filter) or deep call stacks can result in more load than the
+    thread can handle.  Relax [filters](../filters.md) (at the expense of more memory use) or increase the
+    event handler thread priority.      
+
+  If the event queue is full:
+
+  - Ignore it (_really!, it isn't so bad..._)
+
+    If the problem is intermittent, it may not affect all samples, 
+    and data reported in those affected is still likely a representative subset of all I/O.
+    Total I/O values will not be accurate but the relative I/O comparisons between threads
+    should still provide insight into what the target application is doing to generate the I/O load.
+
+  - Increase queue size
+
+    If the problem is intermittent, then a larger queue can let the handler thread
+    catch up after load spikes.  However, if load is consistently over the handler capacity,
+    this will just delay and not fix the problem.  (Requires restart)
+
+  - Increase handler thread priority
+
+    Socket and file I/O events perform all filtering and accumulation functions on the
+    handler thread.  The default is to run at Thread.MIN_PRIORITY, reflecting the decision to
+    risk dropping data rather than impact the target application.  This can be changed to a
+    higher value to reduce dropping events even if it means taking some CPU time away from
+    the target application.

docs/filter-properties.md → docs/probe/filter-properties.md

@@ -16,23 +16,23 @@ In this section we will define the various properties that can be used
 Configuration Properties
 ------------------------
 
-Stacks should generally be [filtered](filters.md) reduce overhead and simplify later analysis:
+Stacks should generally be [filtered](../filters.md) reduce overhead and simplify later analysis:
 
-    ...filter.exclude-jdk    Exclude frames from java built-in packages and gumshoe 
-    ...filter.include        Include only these packages or classes (comma-separated)
-    ...filter.exclude        Exclude these packages or classes 
+    ...filter.simplify       Retain only portions of the stack frames.  Allowed values are
+                               NO_LINE_NUMBERS, NO_METHOD, NO_INNER_CLASSES, NO_CLASSES, NONE
+                             Default is "NONE" (do not simplify frames)
     ...filter.recursion.threshold  The recursion filter is only applied to stacks with
                              more than this number of frames (after other filters).
                              There is no default value, recursion is not applied unless set.
     ...filter.recursion.depth  Find and exclude frames performing recursion.
                              The value must be an integer length indicating the length
                              of the longest sequence of repeated frames the filter will find.
                              The default is "1", but is unused unless the threshold is set.
+    ...filter.exclude-jdk    Exclude frames from java built-in packages and gumshoe 
+    ...filter.include        Include only these packages or classes (comma-separated)
+    ...filter.exclude        Exclude these packages or classes 
     ...filter.top            Number of frames at top of stack to retain
     ...filter.bottom         Number of frames at bottom of stack to retain
-    ...filter.simplify       Retain only portions of the stack frames.  Allowed values are
-                               NO_LINE_NUMBERS, NO_METHOD, NO_INNER_CLASSES, NO_CLASSES, NONE
-                             Default is "NONE" (do not simplify frames)
     ...filter.allow-empty-stack    If filters excluded all frames from a stack,
                                                   the full unfiltered stack can be restored (if false),
                                                   or the empty stack will be used and collect stats