Merge pull request #2 from eyeezzi/distributed-tracing

Distributed tracing

.vscode/launch.json

@@ -0,0 +1,32 @@
+{
+	// Use IntelliSense to learn about possible attributes.
+	// Hover to view descriptions of existing attributes.
+	// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+	"version": "0.2.0",
+	"configurations": [
+		{
+			"type": "node",
+			"request": "attach",
+			"name": "Attach to user-simulator container",
+			"address": "localhost",
+			"port": 9230,
+			"protocol": "inspector",
+			"localRoot": "${workspaceFolder}/user-simulator/src",
+			"remoteRoot": "/usr/src/app",
+			"restart": true
+		},
+		{
+			"type": "node",
+			"request": "attach",
+			"name": "Attach to api-server container",
+			// should match the exposed debug address specified in docker-compose
+			"address": "localhost",
+			"port": 9229,
+			"protocol": "inspector",
+			// you typically copy the src folder into a location in docker...specify this mapping.
+			"localRoot": "${workspaceFolder}/api-server/src",
+			"remoteRoot": "/usr/src/app",
+			"restart": true
+		}
+	]
+}

README.md

@@ -1,43 +1,94 @@
 # Horus
 
-Monitoring containerized microservices with a centralized logging architecture.
+A project for learning about microservices observability. 
 
-## Dependencies
+* Centralized Logging
+* Distributed Tracing
+
+## Development Requirements
 
 - Docker Compose v1.23.2
+- OSX or GNU/Linux
+
+## Development Setup
+
+1. Clone this repo and open the root folder in an IDE like *Visual Studio Code*.
+
+2. For each microservice, rename `example.env` to `.env` and supply the needed secrets.
+    > TODO: Eliminate this friction.
+
+3. Start all microservices in *development mode*.
+
+        docker-compose -f docker-compose.dev.yml \
+            up -d --build
+
+    > In Development Mode
+    >
+    > - You can attach a remote debugger to a running service Docker for seemless debugging like placing breakpoints and watching variables.
+    > - Changes to source files will automatically restart the corresponding docker service.
+
+4. Optionally, attach the IDE's debugger to a service as follows in Visual Studio Code: *shift+cmd+D > Select a debug configuration > F5*.
+    > All vscode debug configurations are stored in *.vscode/launch.json*. You can modify configs as you see fit.
+
+5. Visit http://localhost:16686 to view traces.
+
+### Useful dev commands
 
-## Setup
+    # list all running services
+    docker-compose -f docker-compose.dev.yml ps
 
-1. Signup with an ELK SaaS provider like [Logz.io](logz.io) to obtain an authentication token. Then for each microservice, rename `example.env` to `.env` and supply the needed secrets.
+    # stop all services
+    docker-compose -f docker-compose.dev.yml down
 
-2. Run the following commands.
+    # restart all [or specific] service
+    docker-compose -f docker-compose.dev.yml \
+        up -d --no-deps --build [service-name]
 
-		docker-compose build --pull
-		docker-compose up -d --force-recreate
-		
-3. Then log into your ELK SaaS and view your microservices logs.
+    # tail logs from all [or specific] service
+    docker-compose -f docker-compose.dev.yml \
+        logs -f [service-name]
+        
+    # see how an image was built
+    docker history <image-name>
 
-## Project Documentation
+## Project Architecture
 
-### System Architecture
+### Logging Infrastructure
 
 ![](docs/container-architecture.svg)
 
-I wrote an accompanying [article](https://hackernoon.com/monitoring-containerized-microservices-with-a-centralized-logging-architecture-ba6771c1971a) explaining this architecture.
+Read this [article](https://hackernoon.com/monitoring-containerized-microservices-with-a-centralized-logging-architecture-ba6771c1971a) for more details.
 
-## Notes
+### Tracing Infrastructure
 
-### Docker Networking
+![Tracing Backend Architecture](docs/distributed-tracing/tracing-backend.svg)
+
+Read this [article](#todo) for more details.
+
+## Miscellaneous Notes
+
+### TODO (Improvement Considerations)
+
+- Research **jaeger-operator**
 
-By default each containerized process runs in an isolated network namespace. For inter-container communication, place them in the same network namespace...as seen in *docker-compose.yml*.
+- Name Duplication: The value of the `API_SERVER_ADDRESS` variable in *user-simulator/.env* depends on the service name `api-server` specified in *docker-compose.yml*. If we rename the service, we must also change the variable. Is there a way to make this DRY?
+
+- In the log-shipper container, I had to install a logz.io-specific plugin. Can't this step be eliminated since fluentd is capable of connecting to https endpoints without plugins?
+
+- Use sub-second precision for fluentd timestamps (probably best to use nanoseconds.)
 
 ### Best practices
 
 1. You can pass secrets for a microservice using the `env_file` attribute in *docker-compose.yml*.
 2. Microservices can communicate using their service names if they are in the same docker network.
 
-### Improvement Considerations
+### Docker Networking
+
+By default each containerized process runs in an isolated network namespace. For inter-container communication, place them in the same network namespace.
 
-1. **Name Duplication:** The value of the `API_SERVER_ADDRESS` variable in *user-simulator/.env* depends on the service name `api-server` specified in *docker-compose.yml*. If we rename the service, we must also change the variable. Is there a way to make this DRY?
+### References
 
-2. In the log-shipper container, I had to install a logz.io-specific plugin. Can't this step be eliminated since fluentd is capable of connecting to https endpoints without plugins?
+- https://medium.com/lucjuggery/docker-in-development-with-nodemon-d500366e74df
+- https://blog.risingstack.com/how-to-debug-a-node-js-app-in-a-docker-container/
+- https://codefresh.io/docker-tutorial/debug_node_in_docker/
+- https://code.visualstudio.com/docs/editor/debugging

api-server/.dockerignore

@@ -1,4 +1,4 @@
 .env
-/node_modules
-npm-debug.log
+node_modules
+yarn-error.log
 .DS_Store

api-server/.gitignore

@@ -1,4 +1,4 @@
 .env
-/node_modules
+node_modules
 npm-debug.log
 .DS_Store

api-server/Dockerfile

@@ -2,12 +2,19 @@ FROM node:10.10.0-alpine
 
 WORKDIR /usr/src/app
 
-# optimization to only rebuild npm modules iff package[-lock].json changes
-COPY src/package*.json ./
+RUN apk add --no-cache bash curl \
+	&& yarn global add nodemon
 
-# For production `RUN npm install --only=production`
-RUN npm install
+COPY wait.sh /usr/local/bin/
+RUN chmod +x /usr/local/bin/wait.sh
+
+COPY src/package.json src/yarn.lock ./
+RUN yarn install
 
 COPY src .
 
-CMD ["npm", "start"]
+CMD ["node", "."]
+
+# ref: https://github.com/nodejs/docker-node/blob/master/docs/BestPractices.md
+# https://nodejs.org/en/docs/guides/nodejs-docker-webapp/
+# https://runnable.com/blog/9-common-dockerfile-mistakes

api-server/README.md

@@ -13,4 +13,16 @@
 		docker logs -f <container-name>
 
 		# From another terminal on the host machine, test the server
-		curl http://localhost:5000/api/v1/tokens
+		curl http://localhost:5000/api/v1/tokens
+
+## Dev Notes
+
+To quickly test a container
+
+    docker build -t <image-name> --no-cache . \
+        && docker run -it -v $(pwd)/src:/usr/src/app <imgage-name> sh
+
+    docker build -t api-server --no-cache . \
+        && docker run -it -v $(pwd)/src:/usr/src/app api-server sh
+
+> The `-v` flag maps host absolute path to container absolute path.

api-server/example.env

@@ -1 +1 @@
-PORT=
+# TODO

api-server/src/app.js

@@ -1,19 +1,84 @@
-// const dotenv = require('dotenv')
-// if (process.env.NODE_ENV !== 'production') {
-// 	const result = dotenv.config()
-// 	if (result.error) {
-// 		throw result.error
-// 	}
-// }
-
 const app = require('express')()
 const uuid = require('uuid/v1')
+const axios = require('axios')
+
+// For portability, we initialize tracer from envars instead of local options.
+// See: https://www.npmjs.com/package/jaeger-client#environment-variables
+var opentracing = require('opentracing')
+var initTracer = require('jaeger-client').initTracerFromEnv;
+var tracer = initTracer()
+
+app.use('/health', (req, res) => {
+	res.json(null)
+})
 
 app.use('/api/v1/tokens', (req, res) => {
+	const span = tracer.startSpan('token-request')
+
 	console.log(`Handling request for token`)
 	res.json({token: uuid()})
+
+	span.finish()
 })
 
-const server = app.listen(process.env.PORT, () => {
+app.use('/api/v1/whereami', async (req, res, next) => {
+	const parentSpan = createContinuationSpan(tracer, req, 'whereami-request')
+
+	try {
+		// get location of IP Address
+		const IP = '23.16.76.104'
+		const locSpan = tracer.startSpan('get-location', {childOf: parentSpan})
+		const location = await axios.get(`http://ip-api.com/json/${IP}`)
+		locSpan.finish()
+		const {lat, lon, city, country} = location.data
+
+		// do some other async task
+		const fakeSpan = tracer.startSpan('get-weather', {childOf: parentSpan})
+		const _ = await fakeFetch(1500, 0.7)
+		fakeSpan.finish()
+
+		// return results
+		const data = {lat: lat, lon: lon, city: city, country: country}
+		res.json(data)
+	} catch(err) {
+		parentSpan.setTag('ERROR', err)
+		next(err)
+	}
+
+	parentSpan.finish()
+})
+
+const server = app.listen(process.env.PORT || 3000, () => {
 	console.log(`Listening on ${ server.address().address }:${ server.address().port }`)
-})
+})
+
+function fakeFetch(msDelay, successRate) {
+	return new Promise((resolve, reject) => {
+		setTimeout(() => {
+			if (Math.random() <= successRate) {
+				resolve()
+			} else {
+				reject('Fake fetch failed randomly.')
+			}
+		}, msDelay)
+	})
+}
+
+function extractContext(tracer, req) {
+	return tracer.extract(opentracing.FORMAT_HTTP_HEADERS, req.headers)
+}
+
+// If the request is already being traced, continue the trace
+// else start a new trace.
+function createContinuationSpan(tracer, req, spanName) {
+	const incomingSpanContext = extractContext(tracer, req)
+
+	let newSpan = null
+	if (incomingSpanContext == null) {
+		newSpan = tracer.startSpan(spanName)
+	} else {
+		newSpan = tracer.startSpan(spanName, {childOf: incomingSpanContext})
+	}
+
+	return newSpan
+}