ingest-data-event-hub.md

title	description	services	author	ms.author	ms.reviewer	ms.service	ms.topic	ms.date
Quickstart: Ingest data from Event Hub into Azure Data Explorer	In this quickstart, you learn how to ingest (load) data into Azure Data Explorer from Event Hub.	data-explorer	orspod	v-orspod	mblythe	data-explorer	quickstart	09/24/2018

Quickstart: Ingest data from Event Hub into Azure Data Explorer

Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Azure Data Explorer offers ingestion (data loading) from Event Hubs, a big data streaming platform and event ingestion service. Event Hubs can process millions of events per second in near real-time. In this quickstart, you create an event hub, connect to it from Azure Data Explorer and see data flow through the system.

If you don't have an Azure subscription, create a free Azure account before you begin.

Prerequisites

In addition to an Azure subscription, you need the following to complete this quickstart:

• A test cluster and database

• A sample app that generates data and sends it to an event hub

• Visual studio 2017 Version 15.3.2 or greater to run the sample app

Sign in to the Azure portal

Sign in to the Azure portal.

Create an event hub

In this quickstart, you generate sample data and send it to an event hub. The first step is to create an event hub. You do this by using an Azure Resource Manager template in the Azure portal.

1. Use the following button to start the deployment. We recommend opening the link in another tab or window, so you can follow the rest of the steps in this article.

   

   The Deploy to Azure button takes you to the Azure portal to fill out a deployment form.

   

2. Select the subscription where you want to create the event hub, and create a resource group named test-hub-rg.

   

3. Fill out the form with the following information.

   

   Use defaults for any settings not listed in the following table.

   Setting	Suggested value	Field description
   Subscription	Your subscription	Select the Azure subscription that you want to use for your event hub.
   Resource group	test-hub-rg	Create a new resource group.
   Location	West US	Select West US for this quickstart. For a production system, select the region that best meets your needs.
   Namespace name	A unique namespace name	Choose a unique name that identifies your namespace. For example, mytestnamespace. The domain name servicebus.windows.net is appended to the name you provide. The name can contain only letters, numbers, and hyphens. The name must start with a letter, and it must end with a letter or number. The value must be between 6 and 50 characters long.
   Event hub name	test-hub	The event hub sits under the namespace, which provides a unique scoping container. The event hub name must be unique within the namespace.
   Consumer group name	test-group	Consumer groups enable multiple consuming applications to each have a separate view of the event stream.

4. Select Purchase, which acknowledges that you're creating resources in your subscription.

5. Select Notifications on the toolbar to monitor the provisioning process. It might take several minutes for the deployment to succeed, but you can move on to the next step now.

   

Create a target table in Azure Data Explorer

Now you create a table in Azure Data Explorer, to which Event Hubs will send data. You create the table in the cluster and database provisioned in Prerequisites.

1. In the Azure portal, navigate to your cluster then select Query.

   

2. Copy the following command into the window and select Run.

   .create table TestTable (TimeStamp: datetime, Name: string, Metric: int, Source:string)

   

3. Copy the following command into the window and select Run.

   .create table TestTable ingestion json mapping 'TestMapping' '[{"column":"TimeStamp","path":"$.timeStamp","datatype":"datetime"},{"column":"Name","path":"$.name","datatype":"string"},{"column":"Metric","path":"$.metric","datatype":"int"},{"column":"Source","path":"$.source","datatype":"string"}]'

   This command maps incoming JSON data to the column names and data types of the table (TestTable).

Connect to the event hub

Now you connect to the event hub from Azure Data Explorer. When this connection is in place, data that flows into the event hub streams to the test table you created earlier in this article.

1. Select Notifications on the toolbar to verify that the event hub deployment was successful.

2. Under the cluster you created, select Databases then TestDatabase.

   

3. Select Data ingestion then Add data connection.

   

4. Fill out the form with the following information, then select Create.

   

   Setting	Suggested value	Field description
   Data connection name	test-hub-connection	The name of the connection you want to create in Azure Data Explorer.
   Event hub namespace	A unique namespace name	The name you chose earlier that identifies your namespace.
   Event hub	test-hub	The event hub you created.
   Consumer group	test-group	The consumer group defined in the event hub you created.
   Target table	Leave My data includes routing info unselected.	There are two options for routing: static and dynamic. For this quickstart, you use static routing (the default), where you specify the table name, file format, and mapping. You can also use dynamic routing, where your data includes the necessary routing information.
   Table	TestTable	The table you created in TestDatabase.
   Data format	JSON	JSON and CSV formats are supported.
   Column mapping	TestMapping	The mapping you created in TestDatabase, which maps incoming JSON data to the column names and data types of TestTable.

Copy the connection string

When you run the sample app listed in Prerequisites, you need the connection string for the event hub namespace.

1. Under the event hub namespace you created, select Shared access policies, then RootManageSharedAccessKey.

   

2. Copy Connection string - primary key. You paste it in the next section.

   

Generate sample data

Now that Azure Data Explorer and the event hub are connected, you use the sample app you downloaded to generate data.

1. Open the sample app solution in Visual Studio.

2. In the program.cs file, update the connectionString constant to the connection string you copied from the event hub namespace.

   const string eventHubName = "test-hub";
   // Copy the connection string ("Connection string-primary key") from your Event Hub namespace.
   const string connectionString = @"<YourConnectionString>";

3. Build and run the app. The app sends messages to the event hub, and it prints out status every ten seconds.

4. After the app has sent a few messages, move on to the next step: reviewing the flow of data into your event hub and test table.

Review the data flow

With the app generating data, you can now see the flow of that data from the event hub to the table in your cluster.

1. In the Azure portal, under your event hub, you see the spike in activity while the app is running.

   

2. Go back to the sample app and stop it after it reaches message 99.

3. To check how many messages have made it to the database so far, run the following query in your test database.

   TestTable
   | count

4. To see the content of the messages, run the following query.

   TestTable

   The result set should look like the following.

   

Clean up resources

If you don't plan to use your event hub again, clean up test-hub-rg, to avoid incurring costs.

1. In the Azure portal, select Resource groups on the far left, and then select the resource group you created.

   If the left menu is collapsed, select to expand it.

   

2. Under test-resource-group, select Delete resource group.

3. In the new window, type the name of the resource group to delete (test-hub-rg), and then select Delete.

Next steps

[!div class="nextstepaction"] Quickstart: Query data in Azure Data Explorer