A tool to record and visualize metrics captured from Cascading (Scalding) workflows at runtime.
Designed to target the pain points of analysts and end users of Cascading, Sahale provides insight into a workflow's runtime resource usage and makes job debugging and locating relevant Hadoop logs easy. The tool reveals optimization opportunities by exposing inefficient MapReduce jobs in a larger workflow, and enables users to track the execution history of their workflows.
Sahale has been verified to work with several Cascading DSLs, but the example projects and several features are tailored to Twitter's Scalding DSL, which is the DSL we use at Etsy. Pull requests for better support for other flavors of Cascading are welcome!
There are two components to Sahale, a flowtracker JAR that contains the code that tracks running jobs and a Node.JS dashboard for visualizing the tracked jobs.
The flowtracker JAR is published to Maven Central for easy inclusion in your projects:
<dependency>
<groupId>com.etsy.sahale</groupId>
<artifactId>flowtracker_2.11</artifactId>
<version>2.0.0</version>
</dependency>or, for the Google-Auth-enabled variant:
<dependency>
<groupId>com.etsy.sahale</groupId>
<artifactId>flowtracker-gcp_2.11</artifactId>
<version>2.0.0</version>
<exclusions>
<exclusion>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
</exclusion>
</exclusions>
</dependency>Note that we recommend excluding the transitive dependency google-api-client when running on Google Compute Engine / Dataproc, because the Dataproc clusters already have versions of google-api-client preinstalled. We have run into errors when including our own copy of google-api-client, despite our attempts to match the packaged version with the preinstalled version.
You will need add some configuration for the flow tracker. Create the file src/main/resources/flow-tracker.properties in your project with the following contents:
host=http://host.where.nodejs.server.is.running
port=5735
The host setting will be the hostname of the server where you install the dashboard (see below)
You can also add the setting sahale.flow.selected.configs in this file. This is a comma-delimited list of Hadoop configuration properties that will be tracked along with the other job information.
If this configuration step is not performed job tracking will fail.
-
A MySQL instance that Sahale can use.
-
A server with Node.JS and
npminstalled.
-
Set up the MySQL database. Follow the instructions in
create_db_tables.sqlto do this. -
Clone the Sahale repo onto the server where you would like to run the dashboard.
-
Modify
db-config.jsonto point to your database. -
Run
npm installfrom the root directory of the Sahale repo checkout.
Now you can run node app to start the Sahale dashboard. It will be running on port 5735, so go to <hostname>:5735 in your browser to see the dashboard.
Sahale FlowTracker can be configured to send Google-Auth bearer tokens along with its requests to the Sahale server. This is used when the Sahale server is running in Google Cloud Platform, for example under Google App Engine with authentication enabled. To use this feature, include flowtracker-gcp in your Cascading project, and use the com.etsy.sahale.GoogleAuthFlowTracker class instead of the ordinary com.etsy.sahale.FlowTracker. GoogleAuthFlowTracker works on Hadoop clusters running in Google Compute Engine or Dataproc without any configuration, generating tokens using the application default service account credentials. For clusters not running in GCE/Dataproc, it also accepts a service account JSON file as an optiona argument.
Sahale server does not yet support Google authentication directly; our approach has been to run Sahale server behind an authenticating proxy server, for which there are several open source options.
Sahale server (NodeJS app) and FlowTracker (Scala client jar) must always maintain parity between clients and server versions. An upgrade deployment must coordinate the distribution of the new client jar and restart of the updated server. In rare cases (tagged in the Git repo) Sahale will make breaking changes that will require addition steps. The two notable cases are listed below:
In addition to some minor fixes, we upgrade to scalding 0.17.4. We are alo publishing scala 2.11 and 2.12 artifacts for the first time. We will drop scala 2.11 support in a future version.
We added a new FlowTracker module, flowtracker-gcp, which adds a new FlowTracker subclass, GoogleAuthFlowTracker. This subclass adds Authorization headers to its requests to Sahale server, passing OAuth bearer tokens generated from Google service accounts. This class is meant to be used with a Sahale server running in an environment employing service-to-service authentication via, for example, Google App Engine.
There were no data model or other incompatibility changes in this release.
There were no data model or other incompatibility changes, but Sahale jar is now built with Scala 2.11 and published as flowtracker_2.11.
There were no data model or other incompatibility changes, but the flowtracker JAR is now being published to Maven Central. It is no longer required to manually build this JAR; see the above instructions for its use and configuration.
The data model has changed. Please recreate your backing MySQL tables using the script in src/main/sql before using the new client jar or restarting the NodeJS app. The old tables and data will remain, the new tables will be suffixed with _new unless you opt to alias them. No additional change is required.
Mark incompatible changes between older and newer versions of Scala/Scalding. If your org still uses older versions of Scala/Scalding, please see this commit. All other changes and feature upgrades in the 0.6 line will work as expected with this commit reverted and your own choice of versions applied to the pom.xml.
Sahale was handmade at Etsy.com and is named for Sahale Mountain, which is a wonderful vantage point from which to view the Cascades