SAMZA-976 - Samza REST Documentation

.gitignore

@@ -23,6 +23,7 @@ build
 **/bin
 samza-test/state
 docs/learn/documentation/*/api/javadocs
+docs/learn/documentation/*/rest/javadocs
 .DS_Store
 out/
 *.patch

bin/generate-javadocs.sh

@@ -18,11 +18,19 @@
 
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 BASE_DIR=$DIR/..
-JAVADOC_DIR=$BASE_DIR/docs/learn/documentation/versioned/api/javadocs
+API_JAVADOC_DIR=$BASE_DIR/docs/learn/documentation/versioned/api/javadocs
+REST_JAVADOC_DIR=$BASE_DIR/docs/learn/documentation/versioned/rest/javadocs
 
 cd $BASE_DIR
 ./gradlew javadoc
-rm -rf $JAVADOC_DIR
-mkdir -p $JAVADOC_DIR
-cp -r $BASE_DIR/samza-api/build/docs/javadoc/* $JAVADOC_DIR
+
+# API Docs
+rm -rf $API_JAVADOC_DIR
+mkdir -p $API_JAVADOC_DIR
+cp -r $BASE_DIR/samza-api/build/docs/javadoc/* $API_JAVADOC_DIR
+
+# REST Docs
+rm -rf $REST_JAVADOC_DIR
+mkdir -p $REST_JAVADOC_DIR
+cp -r $BASE_DIR/samza-rest/build/docs/javadoc/* $REST_JAVADOC_DIR
 cd -

docs/learn/documentation/versioned/index.html

@@ -94,4 +94,13 @@ <h4>Operations</h4>
   <li><a href="operations/security.html">Security</a></li>
   <li><a href="operations/kafka.html">Kafka</a></li>
 </ul>
+
+<h4>REST Service</h4>
+
+<ul class="documentation-list">
+  <li><a href="rest/overview.html">Overview</a></li>
+  <li><a href="rest/resources.html">Resources</a></li>
+  <li><a href="rest/monitors.html">Monitors</a></li>
+  <li><a href="rest/resource-directory.html">Resource Directory</a></li>
+</ul>
 </div>

docs/learn/documentation/versioned/rest/monitors.md

@@ -0,0 +1,35 @@
+---
+layout: page
+title: Monitors
+---
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+
+Samza REST supports the ability to add Monitors to the service. The initial implementation is very basic. Monitors are essentially tasks that can be scheduled to run periodically. They do not read the config and they are all scheduled at the same global interval. More capabilities will be added later, but the initial implementation supports simple cases like monitoring the YARN NodeManager and restarting it if it dies.
+
+## Implementing a New Monitor
+Implement the [Monitor](javadocs/org/apache/samza/monitor/Monitor.html) interface with some behavior that should be executed periodically. The Monitor is Java code that invokes some method on the SAMZA Rest Service, runs a bash script to restart a failed NodeManager, or cleans old RocksDB sst files left by Host Affinity, for example.
+
+## Adding a New Monitor to the Samza REST Service
+Add the fully-qualified class name of the Monitor implementation to the `monitor.classes` property in the service config.
+
+Set the `monitor.run.interval.ms` property to the appropriate interval. The `monitor()` method will be invoked at this interval.
+
+For more information on these properties, see the config table in the [Overview page.](overview.html)
+
+## [Resource Reference &raquo;](resource-directory.html)

docs/learn/documentation/versioned/rest/overview.md

@@ -0,0 +1,94 @@
+---
+layout: page
+title: Overview
+---
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+Samza provides a REST service that is deployable on any node in the cluster and has pluggable interfaces to add custom [Resources](resources.html) and [Monitors](monitors.html). It is intended to be a node-local delegate for common actions such as starting jobs, sampling the local state, measuring disk usage, taking heap dumps, verifying liveness, and so on.
+
+The Samza REST Service does not yet have SSL enabled or authentication, so it is initially geared for more backend and operations use cases. It would not be wise to expose it as a user-facing API in environments where users are not allowed to stop each other's jobs, for example.
+
+Samza REST is packaged and configured very similarly to Samza jobs. A notable difference is Samza REST must be deployed and executed on each host you want it to run on, whereas a Samza Job is typically started on a master node in the cluster manager and the master deploys the job to the other nodes.
+
+### Deployment
+Samza REST is intended to be a proxy for all operations which need to be executed from the nodes of the Samza cluster. It can be deployed to all the hosts in the cluster and may serve different purposes on different hosts. In such cases it may be useful to deploy the same release tarball with different configs to customize the functionality for the role of the hosts. For example, Samza REST may be deployed on a YARN cluster with one config for the ResourceManager (RM) hosts and another config for the NodeManager (NM) hosts.
+
+Deploying the service is very similar to running a Samza job. First build the tarball using:
+{% highlight bash %}
+./gradlew samza-rest:clean releaseRestServiceTar
+{% endhighlight %}
+
+
+Then from the extracted location, run the service using:
+{% highlight bash %}
+samza-example/target/bin/run-samza-rest-service.sh  \
+  --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory \
+  --config-path=file://$PWD/config/samza-rest.properties
+{% endhighlight %}
+
+The two config parameters have the same purpose as they do for [run-job.sh](../jobs/job-runner.html).
+
+Follow the [getting started tutorial](../../../tutorials/{{site.version}}/samza-rest-getting-started.html) to quickly deploy and test the Samza REST Service for the first time.
+
+### Configuration
+The Samza REST Service relies on the same configuration system as Samza Jobs. However, the Samza REST Service config file itself is completely separate and unrelated to the config files for your Samza jobs.
+
+The configuration may provide values for the core configs as well as any additional configs needed for Resources or Monitors that you may have added to the service.  A basic configuration file which includes configs for the core service as well as the [JobsResource](resources/jobs.html#configuration) looks like this:
+
+{% highlight jproperties %}
+# Service port. Set to 0 for a dynamic port.
+services.rest.port=9139
+
+# JobProxy
+job.proxy.factory.class=org.apache.samza.rest.proxy.job.SimpleYarnJobProxyFactory
+# Installation path for hello-samza project. Your root may vary.
+job.installations.path=/hello-samza-ROOT/deploy/samza
+{% endhighlight %}
+
+###### Core Configuration
+<table class="table table-condensed table-bordered table-striped">
+  <thead>
+    <tr>
+      <th>Name</th>
+      <th>Default</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>services.rest.port</td><td></td><td><b>Required:</b> The port to use on the local host for the Samza REST Service. If 0, an available port will be dynamically chosen.</td>
+    </tr>
+    <tr>
+      <td>rest.resource.factory.classes</td><td></td><td>A comma-delimited list of class names that implement ResourceFactory. These factories will be used to create specific instances of resources and can pull whatever properties they need from the provided server config. The instance returned will be used for the lifetime of the server. If no value is provided for this property or <pre>rest.resource.classes</pre> then <pre>org.apache.samza.rest.resources.DefaultResourceFactory</pre> will be used as a default.</td>
+    </tr>
+    <tr>
+      <td>rest.resource.classes</td><td></td><td>A comma-delimited list of class names of resources to register with the server. These classes can be instantiated as often as each request, the life cycle is not guaranteed to match the server. Also, the instances do not receive any config. Note that the lifecycle and ability to receive config are the primary differences between resources added via this property versus rest.resource.factory.classes</td>
+    </tr>
+    <tr>
+      <td>monitor.classes</td><td></td><td>A comma-delimited list of monitor classes to use. These should be fully-qualified (org.apache.samza...) and must implement the Monitor interface.</td>
+    </tr>
+    <tr>
+      <td>monitor.run.interval.ms</td><td>60000</td><td>The interval at which to call the run() method of each monitor. This one value applies to all monitors. They are not individually configurable.</td>
+    </tr>
+  </tbody>
+</table>
+
+### Logging
+Samza REST uses SLF4J for logging. The `run-samza-rest-service.sh` script mentioned above by default expects a log4j.xml in the package's bin directory and writes the logs to a logs directory in the package root. However, since the script invokes the same `run-class.sh` script used to run Samza jobs, it can be reconfigured very similarly to [logging for Samza jobs](../jobs/logging.html).
+
+## [Resources &raquo;](resources.html)

docs/learn/documentation/versioned/rest/resource-directory.md

@@ -0,0 +1,27 @@
+---
+layout: page
+title: Resource Directory
+---
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+The Samza REST Service ships with the JAX-RS Resources listed below.
+
+- [JobsResource](resources/jobs.html)
+- (Second resource coming soon)
+
+## [Jobs Resource &raquo;](resources/jobs.html)

docs/learn/documentation/versioned/rest/resources.md

@@ -0,0 +1,61 @@
+---
+layout: page
+title: Resources
+---
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+Samza REST can expose any JAX-RS Resource. By default, it ships with a [JobsResource](./resources/jobs.html) which is discussed below. You can implement your own Resources and specify them in the config.
+
+## Implementing a New Resource
+Samza REST uses the [Jersey](https://jersey.java.net/) implementation of the [JAX-RS specification](https://jax-rs-spec.java.net/). The Jersey documentation and examples are helpful for creating a new Resource.
+
+In addition, Samza REST imposes the following conventions, which should be followed for all Resources.
+
+### Versioned Paths
+By convention, all resources prefix their path with a version number to enable supporting legacy APIs in the future. For example, the base URL for the JobsResource is
+
+	/v1/jobs
+If a future version of JobsResource implements a different API, it will use a different version number in the base path.
+
+	/v2/jobs
+All resources should be exposed on the latest version so clients can use a common version base for all requests.
+
+### Error Messages
+Every error response will include a JSON message body containing a single `message` field describing the problem. For example:
+
+	{
+	    "message": "Unrecognized status: null"
+	}
+
+### Configuration
+There are a few extra steps for Resources that require configuration values.
+
+1. Implement a configuration class that extends [MapConfig](../api/javadocs/org/apache/samza/config/MapConfig.html) with all the constants and accessors for the Resource's configs. The [SamzaRestConfig](javadocs/org/apache/samza/rest/SamzaRestConfig.html) should not be cluttered with Resource configs. See the [JobsResourceConfig](javadocs/org/apache/samza/rest/resources/JobsResourceConfig.html) for an example.
+2. Implement or extend a [ResourceFactory](javadocs/org/apache/samza/rest/resources/ResourceFactory.html) which will use the global Samza REST config file to instantiate the MapConfig implementation from step 1 and use it to construct the Resource that requires the config. See the [DefaultResourceFactory](javadocs/org/apache/samza/rest/resources/DefaultResourceFactory.html) for an example.
+3. Add all the necessary resource properties to the Samza REST config file. The [SamzaRestApplication](javadocs/org/apache/samza/rest/SamzaRestApplication.html) passes the global config into the configured ResourceFactories to instantiate the Resources with configs.
+
+## Adding a New Resource to the Samza REST Service
+Resources are added to the Samza REST Service via config. There are two ways to add a new Resource depending on whether the Resource requires properties from the config file or not. In the former case, the Resource is instantiated once with the config and the instance is registered with the SamzaRestApplication. In the latter case, the Resource *class* is registered and it can be instantiated many times over the lifecycle of the application.
+
+- To add a configured Resource add the fully-qualified class name of the ResourceFactory implementation that instantiates the Resource to the `rest.resource.factory.classes` property in the service config.
+
+* To add a config-less Resource, add the fully-qualified class name of the Resource implementation to the `rest.resource.classes` property in the service config.
+
+For more information on these config properties, see the config table in the [Overview page.](overview.html#configuration)
+
+## [Monitors &raquo;](monitors.html)