Skip to content

Commit 13852c6

Browse files
sethmanheimsethmanheim
committed
Adding redirects as articles moved to samples repo
1 parent e3e0803 commit 13852c6

3 files changed

+6
-243
lines changed

‎articles/event-hubs/event-hubs-pulling-public-data.md

+2-79
Original file line numberDiff line numberDiff line change
@@ -1,80 +1,3 @@
11
---
2-
title: Pulling public data into Azure Event Hubs | Microsoft Docs
3-
description: Overview of the Event Hubs import from web sample
4-
services: event-hubs
5-
documentationcenter: na
6-
author: spyrossak
7-
manager: timlt
8-
editor: ''
9-
10-
ms.assetid: 2a5d7f9e-efd3-4200-908c-a8d7418b1d43
11-
ms.service: event-hubs
12-
ms.devlang: na
13-
ms.topic: article
14-
ms.tgt_pltfrm: na
15-
ms.workload: na
16-
ms.date: 08/25/2016
17-
ms.author: spyros;sethm
18-
19-
---
20-
# Pulling public data into Azure Event Hubs
21-
In typical Internet of Things (IoT) scenarios, you have devices that you can program to push data to Azure,
22-
either to an Azure Event Hub or an IoT hub. Both of those hubs are entry points into Azure for storing,
23-
analyzing, and visualizing with a myriad of tools made available on Microsoft Azure. However, they both require that you push data to them, formatted as JSON and secured in specific ways.
24-
This brings up the following question. What do you do if you want to bring in data from either public or private
25-
sources where the data is exposed as a web service or feed of some sort, but you do not have the ability to
26-
change how the data is published? Consider the weather, or traffic, or stock quotes - you can't tell NOAA,
27-
or WSDOT, or NASDAQ to configure a push to your Event Hub. To solve this problem, we've written and open-sourced
28-
a small cloud sample that you can modify and deploy that will pull the data from some such source and push
29-
it to your Event Hub. From there, you can do whatever you want with it, subject, of course, to the license
30-
terms from the producer. You can find the application [here](https://azure.microsoft.com/documentation/samples/event-hubs-dotnet-importfromweb/).
31-
32-
Note that the code in this sample only shows how to pull data from typical web feeds, and how to write to an Azure Event Hub. This is NOT intended to be a production application, and no attempts have been made to make it suitable for use in such an environment - it is strictfly a DIY, developer-focused example only. Furthermore, the existence of this sample is NOT tantamount to a recommendation that you should **pull** data into Azure rather than **push** it. You should review security, performance, functionality, and cost factors before settling on an end-to-end architecture.
33-
34-
## Application structure
35-
The application is written in C#, and the [sample description](https://azure.microsoft.com/documentation/samples/event-hubs-dotnet-importfromweb/) contains all the information you need to
36-
modify, build, and publish the application. The following sections provide a high-level overview of what the
37-
application does.
38-
39-
We start with the assumption that you have access to a data feed. For example, you might want to pull in the
40-
traffic data from the Washington State Department of Transportation, or the weather data from NOAA, either to
41-
display custom reports or to combine that data with other data in your application. You'll also need to have set
42-
up an Azure Event Hub, and know the connection string needed to access it.
43-
44-
When the GenericWebToEH solution starts up, it reads a configuration file (App.config) to get a number of things:
45-
46-
1. The URL, or a list of URLs, for the site publishing the data. Ideally, this is a site that publishes data
47-
in JSON, such as those referenced by WSDOT [here](http://www.wsdot.wa.gov/Traffic/api/).
48-
2. Credentials for the URL, if needed. Many public sources do not need credentials, or you can put the credentials
49-
in the URL string. Others require that you supply separately. (Note that you can only specify one set of credentials
50-
in this application, so it will only work if you specify only one URL, not a list of URLs.)
51-
3. The connection string and the name of the Event Hub in that Event Hubs namespace, to which you will push the data. You can
52-
find this information in the Azure portal.
53-
4. A sleep interval, in milliseconds, for the interval between polling the public data site. Setting this requires
54-
some thought. If you poll too infrequently, you may miss data; on the other hand, if you poll too frequently, you may
55-
get a lot of repetitive data, or worse yet, you may be blocked as a nefarious bot. Consider how often the data
56-
source is updated - weather or traffic data may be refreshed every 15 minutes, but stock quotes maybe every few
57-
seconds, depending upon where you get them.
58-
5. A flag to tell the application whether the data is coming in as JSON or XML. Since you need to push the
59-
data to an Event Hub, the application has a module to convert XML into JSON before sending.
60-
61-
After reading the configuration file, the application goes into a loop - accessing the public web site, converting
62-
the data if necessary, writing it to your Event Hub, and then waiting for the sleep interval before doing it all over
63-
again. Specifically:
64-
65-
* Reading the public website. For receiving ready-to-send data the instance of RawXMLWithHeaderToJsonReader
66-
class is used from Azure/GenericWebToEH/ApiReaders/RawXMLWithHeaderToJsonReader.cs. It reads source stream
67-
in the GetData() method, and then splits it to smaller pieces (i.e. records) using GetXmlFromOriginalText.
68-
This method will read XML as well as well-formed JSON or JSON array. Then processing is started
69-
using MergeToXML configuration from App.config (default=empty).
70-
* The receiving and sending data is implemented as a loop in the Process() method in Program.cs.
71-
After receiving output results from GetData(), the method enqueues separated values to the Event Hub.
72-
73-
## Next Steps
74-
To deploy the solution, clone or download the [GenericWebToEH](https://azure.microsoft.com/documentation/samples/event-hubs-dotnet-importfromweb/)
75-
application, edit the App.config file, build it, and finally publish it. Once you have published the application,
76-
you can see it running in the Azure classic portal under Cloud Services, and you can change some of the configuration
77-
settings (such as the Event Hub target and the sleep interval) in the **Configure** tab.
78-
79-
See more Event Hubs samples in the [Azure samples gallery](https://azure.microsoft.com/documentation/samples/?service=event-hubs) and on [MSDN](https://code.msdn.microsoft.com/site/search?query=event%20hubs&f%5B0%5D.Value=event%20hubs&f%5B0%5D.Type=SearchText&ac=5).
80-
2+
redirect_url: https://github.com/Azure-Samples/event-hubs-dotnet-importfromweb
3+
---

‎articles/event-hubs/event-hubs-pulling-sql-data.md

+2-86
Original file line numberDiff line numberDiff line change
@@ -1,87 +1,3 @@
11
---
2-
title: Pulling SQL data into Azure Event Hubs | Microsoft Docs
3-
description: Overview of the Event Hubs import from SQL sample
4-
services: event-hubs
5-
documentationcenter: na
6-
author: spyrossak
7-
manager: timlt
8-
editor: ''
9-
10-
ms.assetid: 3dbc21f2-5e97-463f-85c8-78450369ca48
11-
ms.service: event-hubs
12-
ms.devlang: na
13-
ms.topic: article
14-
ms.tgt_pltfrm: na
15-
ms.workload: na
16-
ms.date: 08/25/2016
17-
ms.author: spyros;sethm
18-
19-
---
20-
# Pulling data from SQL into an Azure Event Hub
21-
A typical architecture for an application for processing real-time data involves
22-
first pushing it to an Azure Event Hub. It may be an IoT scenario, such as
23-
monitoring the traffic on different stretches of a highway, or a gaming scenario, monitoring
24-
the actions of a horde of frenzied contestants, or an enterprise scenario, such as
25-
monitoring building occupancy. In these cases, the data producers are generally external
26-
objects producing time-series data that you need to collect, analyse, store, and act upon,
27-
and you may have invested a lot of effort into building out the infrastructure for these
28-
processes. What do you do if you want to bring in data from something like a database rather
29-
than a source of streaming data, and use it in conjunction with your other streaming data?
30-
Consider the case where you want to use Azure Stream Analytics, Remote Data Explorer (RDX), or
31-
some other tool to analyze and act on slowly-changing data from Microsoft Dynamics AX or a
32-
custom facilities-management system? To solve this problem, we've written and open-sourced a small cloud
33-
sample that you can modify and deploy that will pull the data from a SQL table and push it
34-
to an Azure Event Hub to use as an input in your downstream analytical applications. Do realize that this is a
35-
rare scenario, and the opposite of what you normally do with an Event Hub. However, if you
36-
do find yourself in the situation where this is what you need to do, you can find the code in the Azure
37-
Samples gallery, [here](https://azure.microsoft.com/documentation/samples/event-hubs-dotnet-import-from-sql/).
38-
39-
Note that the code in this sample is just that, a sample. It is **not** intended to be a production application, and no attempts have been made to make it suitable for use
40-
in such an environment - it is stricly a DIY, developer-focused, example.
41-
You need to review all sorts of security, performance, functionality, and cost factors before settling on
42-
an end-to-end architecture.
43-
44-
## Application structure
45-
The application is written in C#, and the readme.md file in the sample contains all the information you need to
46-
modify, build, and publish the application. The following sections provide a high level overview of what the
47-
application does.
48-
49-
We start with the assumption that you have access to a SQL Azure table. You'll also need to have set
50-
up an Azure Event Hub, and know the connection string needed to access it.
51-
52-
When the SqlToEventHub solution starts up, it reads a configuration file (App.config)
53-
to get a number of things, as outlined in the readme.md file. These are pretty self-explanatory,
54-
such as the name of the data table, etc, and there is no need to rehash the explanations
55-
here.
56-
57-
Once the application has read the config file, it goes into a loop, reading the SQL table and
58-
pushing records to the event hub, then waiting for a user-defined sleep interval before doing it
59-
all over again. A few things are worth noting:
60-
61-
1. The application is based upon the assumption that the SQL table is being updated by some
62-
external process, and you want to send all and only the updates to an Event Hub.
63-
2. The SQL table needs to have a field that has a unique and increasing number, for example,
64-
a record numer. This can be as simple as a field called "Id", or anything else that is
65-
incremented as whatever updates that database adds records such as "Creation_time" or "Sequence_number". The application notes and stores
66-
the value of that field in each iteration. In each subsequent pass through the loop, the
67-
application essentially queries the table for all records where the value of that field exceeds
68-
the value it saw the last time through the loop. We are calling this last value the "offset".
69-
3. The application creates a table "TableOffsets" when it starts up, to store the offsets. The
70-
table is created with the query "CreateOffsetTableQuery" defined in the config file.
71-
4. There are several queries used for working with the offset table, defined in the config
72-
file as "OffsetQuery", "UpdateOffsetQuery", and "InsertOffsetQuery". You should not change these.
73-
5. Finally, the query "DataQuery" defined in the config file is the query that will be run to
74-
pull the records from your SQL table. It is currently limited to the top 1,000 records in each pass
75-
through the loop for optimization purposes - if, for example, 25,000 records have been added
76-
to the database since the last query, it could take a while to execute the query. By limiting
77-
the query to 1,000 records each time, the queries are much faster. Selecting
78-
the top 1,000 simple pushes successive batches of 1,000 records to the event hub.
79-
80-
## Next Steps
81-
To deploy the solution, clone or download the SqlToEventHub
82-
application, edit the App.config file, build it, and finally publish it. Once you have published the application,
83-
you can see it running in the Azure classic portal under Cloud Services, and monitor the events
84-
coming in to your event hub. Note that the frequency will be determined by two things: the
85-
frequency of the updates to the SQL table, and the sleep interval you have specified in the
86-
config file for the application.
87-
2+
redirect_url: https://github.com/Azure-Samples/event-hubs-dotnet-import-from-sql
3+
---

‎articles/event-hubs/event-hubs-sensors-notify-users.md

+2-78
Original file line numberDiff line numberDiff line change
@@ -1,79 +1,3 @@
1-
---
2-
title: Notify users of data received from sensors or other systems | Microsoft Docs
3-
description: Describes how to use Event Hubs to notify users of sensor data.
4-
services: event-hubs
5-
documentationcenter: na
6-
author: spyrossak
7-
manager: timlt
8-
editor: ''
9-
10-
ms.assetid: a4c0d679-2cac-4ac6-ad48-26cf9a6d8627
11-
ms.service: event-hubs
12-
ms.devlang: na
13-
ms.topic: article
14-
ms.tgt_pltfrm: na
15-
ms.workload: na
16-
ms.date: 08/25/2016
17-
ms.author: spyros;sethm
18-
191
---
20-
# Notify users of data received from sensors or other systems
21-
Suppose you have an application that monitors data in real time, or produces reports on a schedule. If you look at the website on which those real-time charts or reports are displayed, you might see something that requires action. What if you need to be alerted to those situations, rather than relying on remembering to check the website? Imagine that you have a grow light in a greenhouse, and you need to know immediately if the light goes out. One way to do that would be with a light sensor in the greenhouse, arranging to be sent an email if the light is off.
22-
23-
![][1]
24-
25-
In another scenario, imagine that you run a pet boarding facility, and you must be alerted to low inventory supply levels. For example, you might arrange to be sent a text message from your ERP system if your warehouse inventory of dog food has fallen to a critical level.
26-
27-
![][2]
28-
29-
The problem is how to get critical information when certain conditions are met, not when you get around to checking out a static report. If you are using an [Azure Event Hub][Azure Event Hub] or [Azure IoT Hub][Azure IoT Hub] to receive data from devices or enterprise applications such as [Dynamics AX][Dynamics AX], you have several options for how to process them. You can view them on a website, you can analyze them, you can store them, and you can use them to trigger commands to do something. To do this, you can use powerful tools such as [Azure Websites][Azure Websites], [SQL Azure][SQL Azure], [HDInsight][HDInsight], [Cortana Intelligence Suite][Cortana Intelligence Suite], [IoT Suite][IoT Suite], [Logic Apps][Logic Apps], or [Azure Notification Hubs][Azure Notification Hubs]. But sometimes all you want to do is to send that data to someone with a minimum of overhead. To show you how to do that with just a little bit of code, we’ve provided a new sample, [AppToNotifyUsers][AppToNotifyUsers]. Options included are email (SMTP), SMS, and phone.
30-
31-
## Application structure
32-
The application is written in C#, and the readme file in the sample contains all the info you need to modify, build, and publish the application. The following sections provide a high-level overview of what the application does.
33-
34-
We start with the assumption that you have critical events being pushed to an Azure Event Hub or IoT Hub. Any hub will do, as long as you have access to it and know the connection string.
35-
36-
If you do not already have an Event Hub or IoT hub, you can easily set up a test bed with an Arduino shield and a Raspberry Pi, following the instructions in the [Connect The Dots](https://github.com/Azure/connectthedots) project. The light sensor on the Arduino shield sends the light levels through the Pi to an [Azure Event Hub][Azure Event Hub] (**ehdevices**), and an [Azure Stream Analytics](https://azure.microsoft.com/services/stream-analytics/) job pushes alerts to a second event hub (**ehalerts**) if the light levels received fall below a certain level.
37-
38-
When **AppToNotify** starts, it reads a configuration file (App.config) to get the URL and credentials for the Event Hub receiving the alerts. It then spawns a process to continuously monitor that Event Hub for any message that comes through – as long as you have can access the URL for the Event Hub or IoT hub and valid credentials, this Event Hubs reader code will continuously read what's coming in. During startup, the application also reads the URL and credentials for the messaging service (email, SMS, phone) you want to use, and the name/address of the sender and a list of recipients.
39-
40-
Once the Event Hub monitor detects a message, it triggers a process that sends that message using the method specified in the configuration file. Note that it sends every message it detects. If you set the monitor to point to an Event Hub that receives ten messages per second, the sender will send ten messages per second – ten emails per second, ten SMS messages per second, ten phone calls per second. For that reason, make sure that you monitor an Event Hub that only receives the alerts that need to be sent out, not an Event Hub that receives all the raw data from your sensors or applications.
41-
42-
## Applicability
43-
The code in this sample only shows how to monitor Event Hubs and how to call external messaging services in the event that you want to add this functionality to your application. Note that this solution is a DIY, developer-focused example only. It does not address enterprise requirements such as redundancy, fail-over, restart upon failure, etc. For more comprehensive and production solutions, see the following:
44-
45-
* Using connectors or push notifications using the [Azure Logic Apps](../app-service-logic/app-service-logic-connectors-list.md) service.
46-
* Using [Azure Notification Hubs](https://msdn.microsoft.com/library/azure/jj927170.aspx), as described the blog [Broadcast push notifications to millions of mobile devices using Azure Notification Hubs](http://weblogs.asp.net/scottgu/broadcast-push-notifications-to-millions-of-mobile-devices-using-windows-azure-notification-hubs).
47-
48-
## Next steps
49-
It is straightforward to create a simple notification service that sends emails or text messages to recipients, or calls them, to relay data received by an Event Hub or IoT Hub. To deploy the solution to notify users based upon data received by these hubs, visit [AppToNotifyUsers][AppToNotifyUsers].
50-
51-
For more information about these hubs, see the following articles:
52-
53-
* [Azure Event Hubs]
54-
* [Azure IoT Hub]
55-
* Get started with an [Event Hubs tutorial].
56-
* A complete [sample application that uses Event Hubs].
57-
58-
To deploy the solution to notify users based on data received by these hubs, visit:
59-
60-
* [AppToNotifyUsers][AppToNotifyUsers]
61-
62-
[Event Hubs tutorial]: event-hubs-csharp-ephcs-getstarted.md
63-
[Azure IoT Hub]: https://azure.microsoft.com/services/iot-hub/
64-
[Azure Event Hubs]: https://azure.microsoft.com/services/event-hubs/
65-
[Azure Event Hub]: https://azure.microsoft.com/services/event-hubs/
66-
[sample application that uses Event Hubs]: https://code.msdn.microsoft.com/Service-Bus-Event-Hub-286fd097
67-
[AppToNotifyUsers]: https://github.com/Azure-Samples/event-hubs-dotnet-user-notifications
68-
[Dynamics AX]: http://www.microsoft.com/dynamics/erp-ax-overview.aspx
69-
[Azure Websites]: https://azure.microsoft.com/services/app-service/web/
70-
[SQL Azure]: https://azure.microsoft.com/services/sql-database/
71-
[HDInsight]: https://azure.microsoft.com/services/hdinsight/
72-
[Cortana Intelligence Suite]: http://www.microsoft.com/server-cloud/cortana-analytics-suite/Overview.aspx?WT.srch=1&WT.mc_ID=SEM_lLFwOJm3&bknode=BlueKai
73-
[IoT Suite]: https://azure.microsoft.com/solutions/iot-suite/
74-
[Logic Apps]: https://azure.microsoft.com/services/app-service/logic/
75-
[Azure Notification Hubs]: https://azure.microsoft.com/services/notification-hubs/
76-
[Azure Stream Analytics]: https://azure.microsoft.com/services/stream-analytics/
77-
78-
[1]: ./media/event-hubs-sensors-notify-users/event-hubs-sensor-alert.png
79-
[2]: ./media/event-hubs-sensors-notify-users/event-hubs-erp-alert.png
2+
redirect_url: https://github.com/Azure-Samples/event-hubs-dotnet-user-notifications
3+
---

0 commit comments

Comments
 (0)