Upload files from your device to the cloud with Azure IoT Hub (Python)

This article demonstrates how to file upload capabilities of IoT Hub upload a file to Azure blob storage, using Python.

The Send cloud-to-device messages with IoT Hub article shows the basic device-to-cloud and cloud-to-device messaging functionality of IoT Hub. The Configure Message Routing with IoT Hub tutorial shows a way to reliably store device-to-cloud messages in Azure blob storage. However, in some scenarios, you can't easily map the data your devices send into the relatively small device-to-cloud messages that IoT Hub accepts. For example:

• Videos
• Large files that contain images
• Vibration data sampled at high frequency
• Some form of pre-processed data.

These files are typically batch processed in the cloud, using tools such as Azure Data Factory or the Hadoop stack. When you need to upload files from a device, you can still use the security and reliability of IoT Hub. This article shows you how.

At the end of this article, you run the Python console app FileUpload.py, which uploads a file to storage using the Python Device SDK.

Prerequisites

• An active Azure account. (If you don't have an account, you can create a Trial in just a couple of minutes.)

• An IoT hub in your Azure subscription. If you don't have a hub yet, you can follow the steps in Create an IoT hub.

• A device registered in your IoT hub. If you don't have a device in your IoT hub, follow the steps in Register a device.

• Python version 3.7 or later is recommended. Make sure to use the 32-bit or 64-bit installation as required by your setup. When prompted during the installation, make sure to add Python to your platform-specific environment variable.

• Port 8883 should be open in your firewall. The device sample in this article uses MQTT protocol, which communicates over port 8883. This port may be blocked in some corporate and educational network environments. For more information and ways to work around this issue, see Connecting to IoT Hub (MQTT).

Associate an Azure Storage account to IoT Hub

To upload files from a device, you must have an Azure Storage account and Azure Blob Storage container associated with your IoT hub. Once you associate the storage account and container with your IoT hub, your IoT hub can provide the elements of a SAS URI when requested by a device. The device can then use these elements to construct the SAS URI that it uses to authenticate with Azure Storage and upload files to the blob container.

To associate an Azure Storage account with your IoT hub:

1. Under Hub settings, select File upload on the left-pane of your IoT hub.

   

2. On the File upload pane, select Azure Storage Container. For this article, it's recommended that your storage account and IoT Hub be located in the same region.

   • If you already have a storage account you want to use, select it from the list.

   • To create a new storage account, select +Storage account. Provide a name for the storage account and make sure the Location is set to the same region as your IoT hub, then select OK. The new account is created in the same resource group as your IoT hub. When the deployment completes, select the storage account from the list.

   After you select the storage account, the Containers pane opens.

3. On the Containers pane, select the blob container.

   • If you already have a blob container you want to use, select it from the list and click Select.

   • To create a new blob container, select + Container. Provide a name for the new container. For the purposes of this article, you can leave all other fields at their default. Select Create. When the deployment completes, select the container from the list and click Select.

4. Back on the File upload pane, make sure that file notifications are set to On. You can leave all other settings at their defaults. Select Save and wait for the settings to complete before moving on to the next section.

   

For more detailed instructions on how to create an Azure Storage account, see Create a storage account. For more detailed instructions on how to associate a storage account and blob container with an IoT hub, see Configure file uploads using the Azure portal.

Upload a file from a device app

In this section, you create the device app to upload a file to IoT hub.

1. At your command prompt, run the following command to install the azure-iot-device package. You use this package to coordinate the file upload with your IoT hub.

   pip install azure-iot-device
   

2. At your command prompt, run the following command to install the azure.storage.blob package. You use this package to perform the file upload.

   pip install azure.storage.blob
   

3. Create a test file that you'll upload to blob storage.

4. Using a text editor, create a FileUpload.py file in your working folder.

5. Add the following import statements and variables at the start of the FileUpload.py file.

   import os
   from azure.iot.device import IoTHubDeviceClient
   from azure.core.exceptions import AzureError
   from azure.storage.blob import BlobClient
   
   CONNECTION_STRING = "[Device Connection String]"
   PATH_TO_FILE = r"[Full path to local file]"
   

6. In your file, replace [Device Connection String] with the connection string of your IoT hub device. Replace [Full path to local file] with the path to the test file that you created or any file on your device that you want to upload.

7. Create a function to upload the file to blob storage:

   def store_blob(blob_info, file_name):
       try:
           sas_url = "https://{}/{}/{}{}".format(
               blob_info["hostName"],
               blob_info["containerName"],
               blob_info["blobName"],
               blob_info["sasToken"]
           )
   
           print("\nUploading file: {} to Azure Storage as blob: {} in container {}\n".format(file_name, blob_info["blobName"], blob_info["containerName"]))
   
           # Upload the specified file
           with BlobClient.from_blob_url(sas_url) as blob_client:
               with open(file_name, "rb") as f:
                   result = blob_client.upload_blob(f, overwrite=True)
                   return (True, result)
   
       except FileNotFoundError as ex:
           # catch file not found and add an HTTP status code to return in notification to IoT Hub
           ex.status_code = 404
           return (False, ex)
   
       except AzureError as ex:
           # catch Azure errors that might result from the upload operation
           return (False, ex)
   

   This function parses the blob_info structure passed into it to create a URL that it uses to initialize an azure.storage.blob.BlobClient. Then it uploads your file to Azure blob storage using this client.

8. Add the following code to connect the client and upload the file:

   def run_sample(device_client):
       # Connect the client
       device_client.connect()
   
       # Get the storage info for the blob
       blob_name = os.path.basename(PATH_TO_FILE)
       storage_info = device_client.get_storage_info_for_blob(blob_name)
   
       # Upload to blob
       success, result = store_blob(storage_info, PATH_TO_FILE)
   
       if success == True:
           print("Upload succeeded. Result is: \n") 
           print(result)
           print()
   
           device_client.notify_blob_upload_status(
               storage_info["correlationId"], True, 200, "OK: {}".format(PATH_TO_FILE)
           )
   
       else :
           # If the upload was not successful, the result is the exception object
           print("Upload failed. Exception is: \n") 
           print(result)
           print()
   
           device_client.notify_blob_upload_status(
               storage_info["correlationId"], False, result.status_code, str(result)
           )
   
   def main():
       device_client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
   
       try:
           print ("IoT Hub file upload sample, press Ctrl-C to exit")
           run_sample(device_client)
       except KeyboardInterrupt:
           print ("IoTHubDeviceClient sample stopped")
       finally:
           # Graceful exit
           device_client.shutdown()
   
   
   if __name__ == "__main__":
       main()
   

   This code creates an IoTHubDeviceClient and uses the following APIs to manage the file upload with your IoT hub:

   • get_storage_info_for_blob gets information from your IoT hub about the linked Storage Account you created previously. This information includes the hostname, container name, blob name, and a SAS token. The storage info is passed to the store_blob function (created in the previous step), so the BlobClient in that function can authenticate with Azure storage. The get_storage_info_for_blob method also returns a correlation_id, which is used in the notify_blob_upload_status method. The correlation_id is IoT Hub's way of marking which blob you're working on.

   • notify_blob_upload_status notifies IoT Hub of the status of your blob storage operation. You pass it the correlation_id obtained by the get_storage_info_for_blob method. It's used by IoT Hub to notify any service that might be listening for a notification on the status of the file upload task.

9. Save and close the FileUpload.py file.

Run the application

Now you're ready to run the application.

1. At a command prompt in your working folder, run the following command:

   python FileUpload.py
   

2. The following screenshot shows the output from the FileUpload app:

   

3. You can use the portal to view the uploaded file in the storage container you configured:

   

Next steps

In this article, you learned how to use the file upload feature of IoT Hub to simplify file uploads from devices. You can continue to explore this feature with the following articles:

• Create an IoT hub programmatically

• Azure IoT SDKs

Learn more about Azure Blob Storage with the following links:

• Azure Blob Storage documentation

• Azure Blob Storage for Python API documentation