CORDA-822 - JMX Jolokia instrumentation (corda#2197)

* JMX Jolokia instrumentation WIP (driverDSL, webserver, cordformation, hibernate statistics, access policy config file hardening)

* Cordformation changes to support jolokia agent instrumentation at JVM startup.

* Minor updates to reflect usage of Jolokia 1.3.7 (which uses slightly different .war naming)

* Use relative path reference in -javaagent to prevent problem with long path names with spaces.

* Fixed incorrect regex pattern and added assertion to test.

* Enable JMX monitoring.

* Reporting of Hibernate JMX statistics is configurable (by default, only switched on in devMode)

* Make Artemis JMX enablement configurable.

* Re-instate banning of java serialization.

* Improve JUnit.

* Fixes following rebase from master.

* Re-instated correct regex for picking up Jolokia agent jar.

* Fixed broken integration test.

* Updated documentation

* Updated following PR review feedback.

* Fixed compilation error caused by change in DriverDSL argument type.

* Fixed compilation error caused by change in DriverDSL argument type.

* Fail fast if jolokia-agent-jvm.jar is not located.

* Applied changes in cordformation following review feedback from CA.

build.gradle

@@ -24,7 +24,7 @@ buildscript {
     ext.jackson_version = '2.9.2'
     ext.jetty_version = '9.4.7.v20170914'
     ext.jersey_version = '2.25'
-    ext.jolokia_version = '2.0.0-M3'
+    ext.jolokia_version = '1.3.7'
     ext.assertj_version = '3.8.0'
     ext.slf4j_version = '1.7.25'
     ext.log4j_version = '2.9.1'

config/dev/jolokia-access.xml

@@ -1,5 +1,5 @@
 <?xml version="1.0" encoding="utf-8"?>
-
+<!-- an all powerful policy file -->
 <restrict>
     <http>
         <method>post</method>
@@ -8,23 +8,10 @@
 
     <commands>
         <command>read</command>
+        <command>write</command>
+        <command>exec</command>
         <command>list</command>
+        <command>search</command>
+        <command>version</command>
     </commands>
-
-    <!-- allow anyone to force a garbage collection -->
-    <allow>
-        <mbean>
-            <name>java.lang:type=Memory</name>
-            <operation>gc</operation>
-        </mbean>
-    </allow>
-
-    <!-- in case we ever end up using c3pio connection pooling, this example from the docs prevents the password being exported -->
-    <deny>
-        <mbean>
-            <name>com.mchange.v2.c3p0:type=PooledDataSource,*</name>
-            <attribute>properties</attribute>
-        </mbean>
-    </deny>
-
 </restrict>

config/prod/jolokia-access.xml

@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Jolokia agent and MBean access policy based security -->
+<!-- TODO: review these settings before production deployment -->
+<restrict>
+    <!-- IP based restrictions -->
+    <remote>
+        <!-- IP address, a host name, or a netmask given in CIDR format (e.g. "10.0.0.0/16" for all clients coming from the 10.0 network). -->
+        <host>127.0.0.1</host>
+        <host>localhost</host>
+    </remote>
+    <!-- commands for which access is granted: read, write, exec, list, search, version -->
+    <commands>
+        <command>version</command>
+        <command>read</command>
+    </commands>
+    <!-- MBean access and deny restrictions -->
+    <!-- HTTP method restrictions: get, post -->
+    <http>
+        <method>get</method>
+    </http>
+    <!-- Cross-Origin Resource Sharing (CORS) restrictions
+         (by default, allow cross origin access from any host)
+     -->
+</restrict>

docs/source/changelog.rst

@@ -6,6 +6,9 @@ from the previous milestone release.
 
 UNRELEASED
 ----------
+* Exporting additional JMX metrics (artemis, hibernate statistics) and loading Jolokia agent at JVM startup when using
+  DriverDSL and/or cordformation node runner.
+
 * Removed confusing property database.initDatabase, enabling its guarded behaviour with the dev-mode.
   In devMode Hibernate will try to create or update database schemas, otherwise it will expect relevant schemas to be present
   in the database (pre configured via DDL scripts or equivalent), and validate these are correct.

docs/source/corda-configuration-file.rst

@@ -73,6 +73,7 @@ path to the node's base directory.
         :serverNameTablePrefix: Prefix string to apply to all the database tables. The default is no prefix.
         :transactionIsolationLevel: Transaction isolation level as defined by the ``TRANSACTION_`` constants in
             ``java.sql.Connection``, but without the "TRANSACTION_" prefix. Defaults to REPEATABLE_READ.
+        :exportHibernateJMXStatistics: Whether to export Hibernate JMX statistics (caution: expensive run-time overhead)
 
 :dataSourceProperties: This section is used to configure the jdbc connection and database driver used for the nodes persistence.
     Currently the defaults in ``/node/src/main/resources/reference.conf`` are as shown in the first example. This is currently
@@ -163,7 +164,9 @@ path to the node's base directory.
     Each should be a string.  Only the JARs in the directories are added, not the directories themselves.  This is useful
     for including JDBC drivers and the like. e.g. ``jarDirs = [ 'lib' ]``
 
-:sshd: If provided, node will start internal SSH server which will provide a management shell. It uses the same credentials
-        and permissions as RPC subsystem. It has one required parameter.
+:sshd: If provided, node will start internal SSH server which will provide a management shell. It uses the same credentials and permissions as RPC subsystem. It has one required parameter.
 
     :port: The port to start SSH server on
+
+:exportJMXTo: If set to ``http``, will enable JMX metrics reporting via the Jolokia HTTP/JSON agent.
+    Default Jolokia access url is http://127.0.0.1:7005/jolokia/

docs/source/node-administration.rst

@@ -93,6 +93,8 @@ formats for accessing MBeans, and provides client libraries to work with that pr
 
 Here are a few ways to build dashboards and extract monitoring data for a node:
 
+* `hawtio <https://hawt.io>`_ is a web based console that connects directly to JVM's that have been instrumented with a
+  jolokia agent. This tool provides a nice JMX dashboard very similar to the traditional JVisualVM / JConsole MBbeans original.
 * `JMX2Graphite <https://github.com/logzio/jmx2graphite>`_ is a tool that can be pointed to /monitoring/json and will
   scrape the statistics found there, then insert them into the Graphite monitoring tool on a regular basis. It runs
   in Docker and can be started with a single command.
@@ -105,6 +107,29 @@ Here are a few ways to build dashboards and extract monitoring data for a node:
   It can bridge any data input to any output using their plugin system, for example, Telegraf can
   be configured to collect data from Jolokia and write to DataDog web api.
 
+The Node configuration parameter `exportJMXTo` should be set to ``http`` to ensure a Jolokia agent is instrumented with
+the JVM run-time.
+
+The following JMX statistics are exported:
+
+* Corda specific metrics: flow information (total started, finished, in-flight; flow duration by flow type), attachments (count)
+* Apache Artemis metrics: queue information for P2P and RPC services
+* JVM statistics: classloading, garbage collection, memory, runtime, threading, operating system
+* Hibernate statistics (only when node is started-up in `devMode` due to to expensive run-time costs)
+
+When starting Corda nodes using Cordformation runner (see :doc:`running-a-node`), you should see a startup message similar to the following:
+**Jolokia: Agent started with URL http://127.0.0.1:7005/jolokia/**
+
+When starting Corda nodes using the `DriverDSL`, you should see a startup message in the logs similar to the following:
+**Starting out-of-process Node USA Bank Corp, debug port is not enabled, jolokia monitoring port is 7005 {}**
+
+Several Jolokia policy based security configuration files (``jolokia-access.xml``) are available for dev, test, and prod
+environments under ``/config/<env>``.
+
+The following diagram illustrates Corda flow metrics visualized using `hawtio <https://hawt.io>`_ :
+
+.. image:: resources/hawtio-jmx.png
+
 Memory usage and tuning
 -----------------------
 

gradle-plugins/cordapp/src/main/kotlin/net/corda/plugins/Utils.kt

@@ -23,6 +23,13 @@ class Utils {
                 project.configurations.single { it.name == "compile" }.extendsFrom(configuration)
             }
         }
+        fun createRuntimeConfiguration(name: String, project: Project) {
+            if(!project.configurations.any { it.name == name }) {
+                val configuration = project.configurations.create(name)
+                configuration.isTransitive = false
+                project.configurations.single { it.name == "runtime" }.extendsFrom(configuration)
+            }
+        }
     }
 
 }

gradle-plugins/cordformation/src/main/kotlin/net/corda/plugins/Cordformation.kt

@@ -10,6 +10,8 @@ import java.io.File
  */
 class Cordformation : Plugin<Project> {
     internal companion object {
+        const val CORDFORMATION_TYPE = "cordformationInternal"
+
         /**
          * Gets a resource file from this plugin's JAR file.
          *
@@ -31,5 +33,8 @@ class Cordformation : Plugin<Project> {
 
     override fun apply(project: Project) {
         Utils.createCompileConfiguration("cordapp", project)
+        Utils.createRuntimeConfiguration(CORDFORMATION_TYPE, project)
+        val jolokiaVersion = project.rootProject.ext<String>("jolokia_version")
+        project.dependencies.add(CORDFORMATION_TYPE, "org.jolokia:jolokia-jvm:$jolokiaVersion:agent")
     }
 }

gradle-plugins/cordformation/src/main/kotlin/net/corda/plugins/Node.kt

@@ -1,6 +1,8 @@
 package net.corda.plugins
 
-import com.typesafe.config.*
+import com.typesafe.config.ConfigFactory
+import com.typesafe.config.ConfigRenderOptions
+import com.typesafe.config.ConfigValueFactory
 import net.corda.cordform.CordformNode
 import org.bouncycastle.asn1.x500.X500Name
 import org.bouncycastle.asn1.x500.style.BCStyle
@@ -90,6 +92,7 @@ class Node(private val project: Project) : CordformNode() {
         if (config.hasPath("webAddress")) {
             installWebserverJar()
         }
+        installAgentJar()
         installBuiltCordapp()
         installCordapps()
         installConfig()
@@ -177,6 +180,29 @@ class Node(private val project: Project) : CordformNode() {
         }
     }
 
+    /**
+     * Installs the jolokia monitoring agent JAR to the node/drivers directory
+     */
+    private fun installAgentJar() {
+        val jolokiaVersion = project.rootProject.ext<String>("jolokia_version")
+        val agentJar = project.configuration("runtime").files {
+            (it.group == "org.jolokia") &&
+            (it.name == "jolokia-jvm") &&
+            (it.version == jolokiaVersion)
+            // TODO: revisit when classifier attribute is added. eg && (it.classifier = "agent")
+        }.first()  // should always be the jolokia agent fat jar: eg. jolokia-jvm-1.3.7-agent.jar
+        project.logger.info("Jolokia agent jar: $agentJar")
+        if (agentJar.isFile) {
+            val driversDir = File(nodeDir, "drivers")
+            project.copy {
+                it.apply {
+                    from(agentJar)
+                    into(driversDir)
+                }
+            }
+        }
+    }
+
     /**
      * Installs the configuration file to this node's directory and detokenises it.
      */

gradle-plugins/cordformation/src/noderunner/kotlin/net/corda/plugins/NodeRunner.kt

@@ -22,6 +22,11 @@ private object debugPortAlloc {
     internal fun next() = basePort++
 }
 
+private object monitoringPortAlloc {
+    private var basePort = 7005
+    internal fun next() = basePort++
+}
+
 fun main(args: Array<String>) {
     val startedProcesses = mutableListOf<Process>()
     val headless = GraphicsEnvironment.isHeadless() || args.contains(HEADLESS_FLAG)
@@ -49,8 +54,9 @@ private abstract class JarType(private val jarName: String) {
             return null
         }
         val debugPort = debugPortAlloc.next()
+        val monitoringPort = monitoringPortAlloc.next()
         println("Starting $jarName in $dir on debug port $debugPort")
-        val process = (if (headless) ::HeadlessJavaCommand else ::TerminalWindowJavaCommand)(jarName, dir, debugPort, javaArgs, jvmArgs).start()
+        val process = (if (headless) ::HeadlessJavaCommand else ::TerminalWindowJavaCommand)(jarName, dir, debugPort, monitoringPort, javaArgs, jvmArgs).start()
         if (os == OS.MACOS) Thread.sleep(1000)
         return process
     }
@@ -69,15 +75,23 @@ private abstract class JavaCommand(
         jarName: String,
         internal val dir: File,
         debugPort: Int?,
+        monitoringPort: Int?,
         internal val nodeName: String,
         init: MutableList<String>.() -> Unit, args: List<String>,
         jvmArgs: List<String>
 ) {
+    private val jolokiaJar by lazy {
+        File("$dir/drivers").listFiles { _, filename ->
+            filename.matches("jolokia-jvm-.*-agent\\.jar$".toRegex())
+        }.first().name
+    }
+
     internal val command: List<String> = mutableListOf<String>().apply {
         add(getJavaPath())
         addAll(jvmArgs)
         add("-Dname=$nodeName")
         null != debugPort && add("-Dcapsule.jvm.args=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=$debugPort")
+        null != monitoringPort && add("-Dcapsule.jvm.args=-javaagent:drivers/$jolokiaJar=port=$monitoringPort")
         add("-jar")
         add(jarName)
         init()
@@ -89,14 +103,14 @@ private abstract class JavaCommand(
     internal abstract fun getJavaPath(): String
 }
 
-private class HeadlessJavaCommand(jarName: String, dir: File, debugPort: Int?, args: List<String>, jvmArgs: List<String>)
-    : JavaCommand(jarName, dir, debugPort, dir.name, { add("--no-local-shell") }, args, jvmArgs) {
+private class HeadlessJavaCommand(jarName: String, dir: File, debugPort: Int?, monitoringPort: Int?, args: List<String>, jvmArgs: List<String>)
+    : JavaCommand(jarName, dir, debugPort, monitoringPort, dir.name, { add("--no-local-shell") }, args, jvmArgs) {
     override fun processBuilder() = ProcessBuilder(command).redirectError(File("error.$nodeName.log")).inheritIO()
     override fun getJavaPath() = File(File(System.getProperty("java.home"), "bin"), "java").path
 }
 
-private class TerminalWindowJavaCommand(jarName: String, dir: File, debugPort: Int?, args: List<String>, jvmArgs: List<String>)
-    : JavaCommand(jarName, dir, debugPort, "${dir.name}-$jarName", {}, args, jvmArgs) {
+private class TerminalWindowJavaCommand(jarName: String, dir: File, debugPort: Int?, monitoringPort: Int?, args: List<String>, jvmArgs: List<String>)
+    : JavaCommand(jarName, dir, debugPort, monitoringPort, "${dir.name}-$jarName", {}, args, jvmArgs) {
     override fun processBuilder() = ProcessBuilder(when (os) {
         OS.MACOS -> {
             listOf("osascript", "-e", """tell app "Terminal"

node-api/src/main/kotlin/net/corda/nodeapi/internal/persistence/CordaPersistence.kt

@@ -20,7 +20,8 @@ const val NODE_DATABASE_PREFIX = "node_"
 data class DatabaseConfig(
         val initialiseSchema: Boolean = true,
         val serverNameTablePrefix: String = "",
-        val transactionIsolationLevel: TransactionIsolationLevel = TransactionIsolationLevel.REPEATABLE_READ
+        val transactionIsolationLevel: TransactionIsolationLevel = TransactionIsolationLevel.REPEATABLE_READ,
+        val exportHibernateJMXStatistics: Boolean = false
 )
 
 // This class forms part of the node config and so any changes to it must be handled with care

node-api/src/main/kotlin/net/corda/nodeapi/internal/persistence/HibernateConfiguration.kt

@@ -17,8 +17,10 @@ import org.hibernate.type.AbstractSingleColumnStandardBasicType
 import org.hibernate.type.descriptor.java.PrimitiveByteArrayTypeDescriptor
 import org.hibernate.type.descriptor.sql.BlobTypeDescriptor
 import org.hibernate.type.descriptor.sql.VarbinaryTypeDescriptor
+import java.lang.management.ManagementFactory
 import java.sql.Connection
 import java.util.concurrent.ConcurrentHashMap
+import javax.management.ObjectName
 import javax.persistence.AttributeConverter
 
 class HibernateConfiguration(
@@ -60,9 +62,31 @@ class HibernateConfiguration(
 
         val sessionFactory = buildSessionFactory(config, metadataSources, databaseConfig.serverNameTablePrefix)
         logger.info("Created session factory for schemas: $schemas")
+
+        // export Hibernate JMX statistics
+        if (databaseConfig.exportHibernateJMXStatistics)
+            initStatistics(sessionFactory)
+
         return sessionFactory
     }
 
+    // NOTE: workaround suggested to overcome deprecation of StatisticsService (since Hibernate v4.0)
+    // https://stackoverflow.com/questions/23606092/hibernate-upgrade-statisticsservice
+    fun initStatistics(sessionFactory: SessionFactory) {
+        val statsName = ObjectName("org.hibernate:type=statistics")
+        val mbeanServer = ManagementFactory.getPlatformMBeanServer()
+
+        val statisticsMBean = DelegatingStatisticsService(sessionFactory.statistics)
+        statisticsMBean.isStatisticsEnabled = true
+
+        try {
+            mbeanServer.registerMBean(statisticsMBean, statsName)
+        }
+        catch (e: Exception) {
+            logger.warn(e.message)
+        }
+    }
+
     private fun buildSessionFactory(config: Configuration, metadataSources: MetadataSources, tablePrefix: String): SessionFactory {
         config.standardServiceRegistryBuilder.applySettings(config.properties)
         val metadata = metadataSources.getMetadataBuilder(config.standardServiceRegistryBuilder.build()).run {