Upload files from your device to the cloud with Azure IoT Hub (.NET)

  • Article

This article demonstrates how to file upload capabilities of IoT Hub upload a file to Azure blob storage, using an Azure IoT .NET device and service SDKs.

The Send cloud-to-device messages with IoT Hub article shows the basic cloud-to-device messaging functionality of IoT Hub. The Configure Message Routing with IoT Hub article shows a way to reliably store device-to-cloud messages in Azure blob storage. However, in some scenarios, you can't easily map the data your devices send into the relatively small device-to-cloud messages that IoT Hub accepts. For example:

  • Videos
  • Large files that contain images
  • Vibration data sampled at high frequency
  • Some form of preprocessed data

These files are typically batch processed in the cloud, using tools such as Azure Data Factory or the Hadoop stack. When you need to upload files from a device, you can still use the security and reliability of IoT Hub. This article shows you how.

At the end of this article, you run two .NET console apps:

  • FileUploadSample. This device app uploads a file to storage using a SAS URI provided by your IoT hub. This sample is from the Azure IoT C# SDK repository that you download in the prerequisites.

  • ReadFileUploadNotification. This service app receives file upload notifications from your IoT hub. You create this app.

Note

IoT Hub supports many device platforms and languages (including C, Java, Python, and JavaScript) through Azure IoT device SDKs. Refer to the Azure IoT Developer Center to learn how to connect your device to Azure IoT Hub.

Important

File upload functionality on devices that use X.509 certificate authority (CA) authentication is in public preview, and preview mode must be enabled. It is generally available on devices that use X.509 thumbprint authentication or X.509 certificate attestation with Azure Device Provisioning Service. To learn more about X.509 authentication with IoT Hub, see Supported X.509 certificates.

Prerequisites

  • An IoT hub in your Azure subscription. If you don't have a hub yet, you can follow the steps in Create an IoT hub.

  • A device registered in your IoT hub. If you don't have a device in your IoT hub, follow the steps in Register a device.

  • The sample applications you run in this article are written using C# with .NET Core.

    Download the .NET Core SDK for multiple platforms from .NET.

    Verify the current version of the .NET Core SDK on your development machine using the following command:

    dotnet --version

  • Download the Azure IoT C# SDK from Download sample and extract the ZIP archive.

  • Port 8883 should be open in your firewall. The sample in this article uses MQTT protocol, which communicates over port 8883. This port may be blocked in some corporate and educational network environments. For more information and ways to work around this issue, see Connecting to IoT Hub (MQTT).

Associate an Azure Storage account to IoT Hub

To upload files from a device, you must have an Azure Storage account and Azure Blob Storage container associated with your IoT hub. Once you associate the storage account and container with your IoT hub, your IoT hub can provide the elements of a SAS URI when requested by a device. The device can then use these elements to construct the SAS URI that it uses to authenticate with Azure Storage and upload files to the blob container.

To associate an Azure Storage account with your IoT hub:

  1. Under Hub settings, select File upload on the left-pane of your IoT hub.

    Screen capture showing select file upload settings from the portal.

  2. On the File upload pane, select Azure Storage Container. For this article, it's recommended that your storage account and IoT Hub be located in the same region.

    • If you already have a storage account you want to use, select it from the list.

    • To create a new storage account, select +Storage account. Provide a name for the storage account and make sure the Location is set to the same region as your IoT hub, then select OK. The new account is created in the same resource group as your IoT hub. When the deployment completes, select the storage account from the list.

    After you select the storage account, the Containers pane opens.

  3. On the Containers pane, select the blob container.

    • If you already have a blob container you want to use, select it from the list and click Select.

    • To create a new blob container, select + Container. Provide a name for the new container. For the purposes of this article, you can leave all other fields at their default. Select Create. When the deployment completes, select the container from the list and click Select.

  4. Back on the File upload pane, make sure that file notifications are set to On. You can leave all other settings at their defaults. Select Save and wait for the settings to complete before moving on to the next section.

    Screen capture showing confirm file upload settings in the portal.

For more detailed instructions on how to create an Azure Storage account, see Create a storage account. For more detailed instructions on how to associate a storage account and blob container with an IoT hub, see Configure file uploads using the Azure portal.

Upload file from a device app

In this article, you use a sample from the Azure IoT C# SDK repository you downloaded earlier as the device app. You can open the files below using Visual Studio, Visual Studio Code, or a text editor of your choice.

The sample is located at azure-iot-sdk-csharp/iothub/device/samples/getting started/FileUploadSample in the folder where you extracted the Azure IoT C# SDK.

Examine the code in FileUpLoadSample.cs. This file contains the main sample logic. After creating an IoT Hub device client, it follows the standard three-part procedure for uploading files from a device:

  1. The code calls the GetFileUploadSasUriAsync method on the device client to get a SAS URI from the IoT hub:

    var fileUploadSasUriRequest = new FileUploadSasUriRequest
{
    BlobName = fileName
};

// Lines removed for clarity

FileUploadSasUriResponse sasUri = await _deviceClient.GetFileUploadSasUriAsync(fileUploadSasUriRequest);
Uri uploadUri = sasUri.GetBlobUri();

  2. The code uses the SAS URI to upload the file to Azure storage. In this sample, it uses the SAS URI to create an Azure storage block blob client and uploads the file:

    var blockBlobClient = new BlockBlobClient(uploadUri);
await blockBlobClient.UploadAsync(fileStreamSource, new BlobUploadOptions());

  3. The code notifies the IoT hub that it has completed the upload. This tells the IoT hub that it can release resources associated with the upload (the SAS URI). If file upload notifications are enabled, the IoT hub sends a notification message to backend services.

    var successfulFileUploadCompletionNotification = new FileUploadCompletionNotification
{
    // Mandatory. Must be the same value as the correlation id returned in the sas uri response
    CorrelationId = sasUri.CorrelationId,

    // Mandatory. Will be present when service client receives this file upload notification
    IsSuccess = true,

    // Optional, user defined status code. Will be present when service client receives this file upload notification
    StatusCode = 200,

    // Optional, user-defined status description. Will be present when service client receives this file upload notification
    StatusDescription = "Success"
};

await _deviceClient.CompleteFileUploadAsync(successfulFileUploadCompletionNotification);

If you examine the parameter.cs file, you see that:

  • The sample requires you to pass a parameter, p, which takes a device connection string.

  • By default, the device sample uses the MQTT protocol to communicate with IoT Hub. You can use the parameter t to change this transport protocol. Regardless of this selection, the Azure blob client always uses HTTPS as the protocol to upload the file Azure storage.

Get the IoT hub connection string

In this article, you create a backend service to receive file upload notification messages from your IoT hub. To receive file upload notification messages, your service needs the service connect permission. By default, every IoT Hub is created with a shared access policy named service that grants this permission.

To get the IoT Hub connection string for the service policy, follow these steps:

  1. In the Azure portal, select Resource groups. Select the resource group where your hub is located, and then select your hub from the list of resources.

  2. On the left-side pane of your IoT hub, select Shared access policies.

  3. From the list of policies, select the service policy.

  4. Copy the Primary connection string and save the value.

Screenshot that shows how to retrieve the connection string from your IoT Hub in the Azure portal.

For more information about IoT Hub shared access policies and permissions, see Access control and permissions.

Receive a file upload notification

In this section, you create a C# console app that receives file upload notification messages from your IoT hub.

  1. Open a command window and go to the folder where you want to create the project. Create a folder named ReadFileUploadNotifications and change directories to that folder.

    mkdir ReadFileUploadNotification
cd ReadFileUploadNotification

  2. Run the following command to create a C# console project. After running the command, the folder will contain a Program.cs file and a ReadFileUploadNotification.csproj file.

    dotnet new console --language c#

  3. Run the following command to add the Microsoft.Azure.Devices package to the project file. This package is the Azure IoT .NET service SDK.

    dotnet add package Microsoft.Azure.Devices

  4. Open the Program.cs file and add the following statement at the top of the file:

    using Microsoft.Azure.Devices;

  5. Add the following fields to the Program class. Replace the {iot hub connection string} placeholder value with the IoT hub connection string that you copied previously in Get the IoT hub connection string:

    static ServiceClient serviceClient;
static string connectionString = "{iot hub connection string}";

  6. Add the following method to the Program class:

    private async static void ReceiveFileUploadNotificationAsync()
{
    var notificationReceiver = serviceClient.GetFileNotificationReceiver();
    Console.WriteLine("\nReceiving file upload notification from service");
    while (true)
    {
        var fileUploadNotification = await notificationReceiver.ReceiveAsync();
        if (fileUploadNotification == null) continue;
        Console.ForegroundColor = ConsoleColor.Yellow;
        Console.WriteLine("Received file upload notification: {0}", 
          string.Join(", ", fileUploadNotification.BlobName));
        Console.ResetColor();
        await notificationReceiver.CompleteAsync(fileUploadNotification);
    }
}

    Note this receive pattern is the same one used to receive cloud-to-device messages from the device app.

  7. Finally, replace the lines in the Main method with the following:

    Console.WriteLine("Receive file upload notifications\n");
serviceClient = ServiceClient.CreateFromConnectionString(connectionString);
ReceiveFileUploadNotificationAsync();
Console.WriteLine("Press Enter to exit\n");
Console.ReadLine();

Run the applications

Now you're ready to run the applications.

  1. First, run the service app to receive file upload notifications from the IoT hub. At your command prompt in the ReadFileUploadNotification folder, run the following commands:

    dotnet restore
dotnet run

    The app starts and waits for a file upload notification from your IoT hub:

    Receive file upload notifications


Receiving file upload notification from service
Press Enter to exit

  2. Next, run the device app to upload the file to Azure storage. Open a new command prompt and change folders to the azure-iot-sdk-csharp\iothub\device\samples\getting started\FileUploadSample under the folder where you expanded the Azure IoT C# SDK. Run the following commands. Replace the {Your device connection string} placeholder value in the second command with the device connection string you saw when you registered a device in the IoT hub.

    dotnet restore
dotnet run --p "{Your device connection string}"

    The following output is from the device app after the upload has completed:

      Uploading file TestPayload.txt
  Getting SAS URI from IoT Hub to use when uploading the file...
  Successfully got SAS URI (https://contosostorage.blob.core.chinacloudapi.cn/contosocontainer/MyDevice%2FTestPayload.txt?sv=2018-03-28&sr=b&sig=x0G1Baf%2BAjR%2BTg3nW34zDNKs07p6dLzkxvZ3ZSmjIhw%3D&se=2021-05-04T16%3A40%3A52Z&sp=rw) from IoT Hub
  Uploading file TestPayload.txt using the Azure Storage SDK and the retrieved SAS URI for authentication
  Successfully uploaded the file to Azure Storage
  Notified IoT Hub that the file upload succeeded and that the SAS URI can be freed.
  Time to upload file: 00:00:01.5077954.
  Done.

  3. Notice that the service app shows that it has received the file upload notification:

    Receive file upload notifications


Receiving file upload notification from service
Press Enter to exit

Received file upload notification: myDeviceId/TestPayload.txt

Verify the file upload

You can use the portal to view the uploaded file in the storage container you configured:

  1. Navigate to your storage account in Azure portal.

  2. On the left pane of your storage account, select Containers.

  3. Select the container you uploaded the file to.

  4. Select the folder named after your device.

  5. Select the blob that you uploaded your file to. In this article, it's the blob named TestPayload.txt.

    Screenshot of selecting the uploaded file in the Azure portal.

  6. View the blob properties on the page that opens. You can select Download to download the file and view its contents locally.

Next steps

In this article, you learned how to use the file upload feature of IoT Hub to simplify file uploads from devices. You can continue to explore this feature with the following articles: