Azure IoT Hub - Keep Telemetry Best Practices

Question

I am working on an IoT solution that will store weather data. I’ve been going to work on creating a backend for several days now. I am going to use the Azure IoT Hub to handle communications, but the next step is the problem.

I want to save telemetry to a database. I'm embarrassed here. Some examples say that I should use Azure Blob Storage or Azure Table Storage or Azure SQL.

After years of collecting data, I want to start creating data reports. Therefore, the repository should work well with big data.

The next problem I'm stuck with is the worker who will receive the D2C and save it in the database. All Azure IoT examples use the console application, and some use Azure Stream analytics to transfer the event to the database. What is the best practice? It should be able to scale and use best practices.

Thanks in advance!

Answer 1

If you choose an IoT Hub for communication processing, you have several options for data management (make sure that the IoT hub is the right choice for you, if you do not need bidirectional communication, maybe the Azure Event Hub will be the best choice, it is much cheaper when working with big data).

• Flow analytics. Allows you to output the incoming data to the SQL database, BLOB, event concentrator, table storage, queue of service queues and topics, document database, Power Bi storage and DataLake. In this option, you do not have to manage your own worker to process the data.
• EventProcessorHost - here you will need to write your own implementation for receiving data and storing it. This option will give you the flexibility to store data in every storage you need, but you will have to manage your EPH hosting. Azure Worker Rule is a good choice for hosting and scaling.
• Storm (HD Insights). You can use the Apache flurry to read data from the IoT hub, it will also give you real-time calculation options that are much wider than those provided by Stream Analytics. After reading the data from Storm, you also have the opportunity to save it in any storage you want. keep in mind that a storm on Azure is very expensive and may be redundant for your application.

As for reports - it really depends on your needs, I would avoid blob / storage tables for any complex reporting, those 2 are more optimized for storing a lot of data and less for creating complex queries.

If you want to create your own reports / queries, you can choose Sql / DocumentDb. but make sure that if you choose NoSql, you will get less architecture from the architecture.

To solve Paas, you can choose Power BI - https://powerbi.microsoft.com/en-us/blog/outputting-real-time-stream-analytics-data-to-a-power-bi-dashboard/

Disclaimer - I answered your question based on the assumption that you want to use the Azure stack.

Good luck.

Answer 2

@KristerJohansson. According to your description, based on my understanding, this is an IoT solution in which a data collector receives weather data from some devices with sensors and stores this data for analysis and reporting. I think there are some key fundamentals that need to be considered as variables that determine the amount of data, such as meteorological columns, data format, sample rate, number of devices, etc.

Therefore, for scaling and big data, in my experience, as reference information, I believe that the best practice is to use IoTHub to process communications and use Stream analytics to extract and store data from IoTHub to Blob storage. After years of collecting data, you can use Azure Machine Learning to read this data from the blob for analysis and reporting.

Any concern, please feel free to let me know.

Answer 3

Azure has added an interesting new feature to your problem.

You can now route IoT messages directly to Azure Storage. https://azure.microsoft.com/en-us/blog/route-iot-device-messages-to-azure-storage-with-azure-iot-hub/

I have not tested it yet, but the article looks promising.