Skip to content

Commit

Permalink
Adding alert functionality to cloud-iot-rtdp tutorial (GoogleCloudPla…
Browse files Browse the repository at this point in the history
  • Loading branch information
kingman authored and jmdobry committed May 7, 2018
1 parent 12534bf commit 0a82005
Show file tree
Hide file tree
Showing 6 changed files with 122 additions and 43 deletions.
12 changes: 12 additions & 0 deletions tutorials/cloud-iot-rtdp/bin/create_temp_alert_store.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import os
from google.cloud import datastore

client = datastore.Client(os.environ['GCLOUD_PROJECT'])
for i in range(30):
incomplete_key = client.key('device')
device = datastore.Entity(key=incomplete_key)
device.update({
'name':u'device{}'.format(str(i+1)),
'tempAlertThredshold':i+1
})
client.put(device)
24 changes: 24 additions & 0 deletions tutorials/cloud-iot-rtdp/function/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,34 @@
# See the License for the specific language governing permissions and
# limitations under the License.
*/
const Datastore = require('@google-cloud/datastore');
// Instantiates a client
const datastore = Datastore();

exports.iot = function (event, callback) {
const pubsubMessage = event.data;
var attrs = Buffer.from(pubsubMessage.data, 'base64').toString().split(',');

const deviceProm = getDeviceBy(attrs[0]);
deviceProm.then(devices => {
const device = devices[0][0];
controlDeviceTemperature(device, attrs[2]);
});

console.log(attrs[0] + ', ' + attrs[1] + ', ' + attrs[2] + ', ' + attrs[3] +
', ' + attrs[4] + ', ' + attrs[5]);
callback();
};

function getDeviceBy (deviceName) {
const query = datastore
.createQuery('device')
.filter('name', '=', deviceName);
return datastore.runQuery(query);
}

function controlDeviceTemperature (device, tempMeasured) {
if (tempMeasured > device.tempAlertThredshold) {
console.error(new Error('Measured temperature of: ' + tempMeasured + ' exceeds alert thredshold: ' + device.tempAlertThredshold + ' for ' + device.name));
}
}
5 changes: 5 additions & 0 deletions tutorials/cloud-iot-rtdp/function/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"dependencies": {
"@google-cloud/datastore": "1.3.4"
}
}
124 changes: 81 additions & 43 deletions tutorials/cloud-iot-rtdp/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,28 +5,37 @@ author: teppeiy
tags: Cloud IoT Core, Cloud Dataflow, Cloud Functions, Java, Node.js
date_published: 2018-04-10
---
## Overview
## Application Background
The setup described in this tutorial addresses following scenario: At industrial facilities, sensors are installed to monitor the equipment on site. Sensor data is continuously streamed to the cloud. There it is handled by different components for various purposes, such as real-time monitoring and alerts, long-term data storage for analysis, performance improvement, and model training.

Feasible scenarios are:

- Geographically dispersed facilities with centralized monitoring system.
- Monitoring of remote unmanned sites—power transformer stations, mobile base stations, etc.

In this tutorial, the sensors are simulated by a Java application script that continuously generates random measurement points and sends them to the cloud.

This tutorial focuses on two aspect of the monitoring application setup:

- Using [Cloud IoT Core](https://cloud.google.com/iot-core/docs/), a cloud managed service for IoT, to enforce structured handling of sensor devices' security keys and metadata, and secured delivery of measurement data between sensors and cloud.
- In-stream data handling in the cloud, where two parallel processing pipelines separate real-time monitoring and alerting from the less critical need for data storage and analysis.

## Technical Overview

This tutorial demonstrates how to push updates from Message Queueing Telemetry
Transport (MQTT) devices to Google Cloud Platform (GCP) and process them in real
time.

It is important to detect warning signs quickly from manufacturing lines to
prevent failures, which eventually can result in opportunity loss. To this end,
the manufacturing industry has been investing in monitoring machinery and
equipment in factories. These monitoring devices and alerting help improve
production efficiency and business continuity.

This tutorial includes sample code to show two kinds of data processing
The tutorial includes sample code to show two kinds of data processing
approaches that use GCP products:

1. A function deployed in Cloud Functions transforms data and logs it to
Stackdriver Logging.
1. A streaming application deployed in Cloud Dataflow transforms data and
inserts it into BigQuery.
1. A function deployed in [Cloud Functions](https://cloud.google.com/functions/docs/) transforms data and logs it to
[Stackdriver Logging](https://cloud.google.com/logging/).
1. A streaming application deployed in [Cloud Dataflow](https://cloud.google.com/dataflow/docs) transforms data and
inserts it into [BigQuery](https://cloud.google.com/bigquery/docs/).

In both cases, sample temperature data is collected that is generated from
simulated devices, is transformed into other data formats, and is passed to
simulated devices. This data is transformed into other data formats, and is passed to
another GCP product for further data processing and analysis. Cloud Functions is
suitable for simple Extract/Transform/Load (ETL) processing, while Cloud
Dataflow can handle more sophisticated data pipelines that involve multiple
Expand All @@ -41,16 +50,16 @@ the surrounding environment.
The sample MQTT client simulates devices and generates sample data with the
following attributes:

- DeviceId: A unique identifier for individual devices.
- Timestamp: A timestamp for when a temperature is measured.
- Temperature: The measured temperature from the device.
- Coordinates: The longitude and latitude of the device.
- `DeviceId`: A unique identifier for individual devices.
- `Timestamp`: A timestamp for when a temperature is measured.
- `Temperature`: The measured temperature from the device.
- `Coordinates`: The longitude and latitude of the device.

## Architecture

The sample MQTT client simulates a device and sends sample data to Cloud IoT
Core, which transforms and redirects requests to a Cloud Pub/Sub topic. After
data is stored in Cloud Pub/Sub, it is retrieved by two subscribers: a function
Core, which transforms and redirects requests to a [Cloud Pub/Sub](https://cloud.google.com/pubsub/docs) topic. After
the data is stored in Cloud Pub/Sub, it is retrieved by two subscribers: a function
in Cloud Functions and a streaming job running in Cloud Dataflow.

This tutorial shows how data is transformed and processed in Cloud Functions and
Expand All @@ -67,7 +76,7 @@ This tutorial demonstrates how to:
- Deploy a streaming application to Cloud Dataflow that transforms temperature
data into BigQuery row format and inserts it into BigQuery.
- Run an MQTT client that generates simulated temperature and coordinates, and
submits the data to Cloud IoT Core.
then submits the data to Cloud IoT Core.

## Cost

Expand All @@ -77,8 +86,10 @@ This tutorial uses billable components of GCP, including:
- Cloud Dataflow
- Cloud Functions
- Cloud IoT Core
- Cloud Storage
- Compute Engine
- Cloud Pub/Sub
- [Cloud Storage](https://cloud.google.com/storage/docs/)
- [Compute Engine](https://cloud.google.com/compute/docs/)
- [Cloud Datastore](https://cloud.google.com/datastore/docs/)

You can use the [Pricing Calculator](https://cloud.google.com/products/calculator/)
to generate a cost estimate that is based on your projected usage.
Expand All @@ -93,7 +104,7 @@ Make sure you have the following software installed:
- [Java SE 8](http://www.oracle.com/technetwork/java/javase/downloads/index.html)
- [Apache Maven 3.3.x or later](https://maven.apache.org/install.html)

Clone the following repository and move into the directory for this tutorial's
Clone the following repository and change to into the directory for this tutorial's
code:

git clone https://github.com/GoogleCloudPlatform/community.git
Expand Down Expand Up @@ -122,10 +133,9 @@ code:

![Storage](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/create_storage.png)

The bucket name must be unique across Cloud Storage, so use a name unique to
you.
The bucket name must be unique across Cloud Storage.

1. Click **Create folder**, enter a temporary folder name and then click
1. Click **Create folder**, enter a temporary folder name, and then click
**Create**.

![folder](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/create_folder.png)
Expand All @@ -136,7 +146,7 @@ To make it easier to run commands, you can set environment variables so that you
don't have to supply options for some values that you’ll use repeatedly. You
will create the corresponding resources in later steps.

1. Open [Google Cloud Shell](https://cloud.google.com/shell/docs/quickstart)
1. Open [Cloud Shell](https://cloud.google.com/shell/docs/quickstart)
1. Set the following environment variables:

export PROJECT=[PROJECT_ID]
Expand All @@ -152,23 +162,23 @@ In this section, you create a topic in Cloud Pub/Sub and configure Cloud IoT
Core to receive data from MQTT clients.

1. Open the [Cloud Pub/Sub console](https://console.developers.google.com/cloudpubsub)
1. In the left navigation menu click the **Topics** menu.
1. In the left navigation menu, click the **Topics** menu.
1. Click **Create a topic**. In the **Name** box, enter the topic name that you
assigned earlier to the environment variable, and then click **Create**.

![topic](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/create_topic.png)

1. Open the [Cloud IoT Core console](https://console.developers.google.com/iot).
1. Click **Create device registry**.
1. In the **Registry ID** box, type **myregistry**. Select a Cloud region close
1. In the **Registry ID** box, type **myregistry**. Select a GCP region close
to you, and select the Pub/Sub topic that you just created.

![registry](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/create_registry.png)

1. When you're done, click **Create**.
1. In the **Grant permission to service account** dialog box, click **Continue**.
1. In Cloud Shell, generate a new public/private key pair, which will override
the checked in pair.
the checked in pair:

cd bin
./create_cert.sh
Expand All @@ -180,10 +190,29 @@ Core to receive data from MQTT clients.
Warning: This tutorial includes a public/private key pair for testing
purposes. Do not use this pair in a production environment.

## Create threshold values in Cloud Datastore
In this section, you insert threshold values for each of the devices, registered in the IoT Core Device Manager, in Cloud Datastore.

1. In Cloud Shell, run a Python script to insert the device objects into Cloud Datastore:

export GCLOUD_PROJECT=$PROJECT
virtualenv env && source env/bin/activate
pip install google-cloud-datastore
cd bin
python create_temp_alert_store.py
deactivate

1. Open the [Cloud Datastore console](https://console.developers.google.com/datastore).
1. Confirm that the device entities have been created with the corresponding threshold temperature value:

![data_store_confirm](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/view_ds.png)

## Deploy a Cloud Function

In this section, you set up a function that logs data that is sent to Cloud IoT
Core and retrieved through Cloud Pub/Sub.
Core and is retrieved through Cloud Pub/Sub. It also compares the temperature
received against the threshold value in Cloud Datastore. If the threshold is
exceeded, an error is logged.

1. In Cloud Shell, deploy a function to Cloud Functions:

Expand All @@ -201,7 +230,7 @@ Core and retrieved through Cloud Pub/Sub.
...

1. Open the [Cloud Functions console](https://console.developers.google.com/functions).
1. Confirm that you created a function.
1. Confirm that you created a function:

![function_confirm](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/create_cf.png)

Expand All @@ -224,10 +253,10 @@ data that is retrieved from Cloud Pub/Sub and loads it into a BigQuery table.
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ cloud-iot-rtdp ---
...

1. Open the [Cloud Dataflow console](https://console.developer.google.com/dataflow).
1. Confirm that a streaming job is running.
1. Confirm that a streaming job is running:

![stream_confirm](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/create_df.png)

Expand All @@ -253,7 +282,7 @@ coordinates and then submits it to Cloud IoT Core.
...

1. Open the [Cloud Functions console](https://console.developers.google.com/functions).
1. To confirm that a function is processing data, click the More options icon
1. To confirm that a function is processing data, click the **More options** icon
on the right side of your function, and then click **View logs**:

![view_logs](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/cf_console.png)
Expand All @@ -264,26 +293,35 @@ coordinates and then submits it to Cloud IoT Core.

1. Open the [Cloud Dataflow console](https://console.developers.google.com/dataflow).
1. To confirm that a streaming Cloud Dataflow job is processing data, click the
job ID.
job ID:

![view_df](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/view_df.png)

1. Open the [BigQuery](https://bigquery.cloud.google.com/).
1. Click **COMPOSE QUERY** button to open the query editor.
1. To confirm the temperature data is stored in a BigQuery table, run the
1. Open [BigQuery](https://bigquery.cloud.google.com/).
1. Click the **Compose Query** button to open the query editor.
1. To confirm that the temperature data is stored in a BigQuery table, run the
following query in the editor:

SELECT count(*) from [[PROJECT_ID]:iotds.temp_sensor]

![bq_editor](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/bq_console.png)

If everything is working, you a single row in the results that displays a
count of all the records that have been processed.
If everything is working, you should see a single row in the results that
displays a count of all the records that have been processed.

## Handling alerts

Temperature measurements that are above the configured threshold for each device
are logged as errors by Cloud Functions. You can view and analyse these in the
[Error console](https://console.developers.google.com/errors).

To active the error notifications, follow the documentation on [Error reporting notifications](https://cloud.google.com/error-reporting/docs/notifications).

![error_console](https://storage.googleapis.com/gcp-community/tutorials/cloud-iot-rtdp/view_error_report.png)

## Next steps

This tutorial demonstrates how to process data that is generated from IoT
devices. You can learn more about IoT, data processing, and visualization from
You can learn more about IoT, data processing, and visualization from
the following links:

- [Overview of IoT Solutions](https://cloud.google.com/solutions/iot/)
Expand Down
Binary file added tutorials/cloud-iot-rtdp/view_ds.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tutorials/cloud-iot-rtdp/view_error_report.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 0a82005

Please sign in to comment.