Upload files from your device to the cloud with Azure IoT Hub (Java)
This article demonstrates how to file upload capabilities of IoT Hub upload a file to Azure blob storage, using Java.
The Send cloud-to-device messages with IoT Hub article shows the basic cloud-to-device messaging functionality of IoT Hub. The Configure message routing with IoT Hub tutorial shows a way to reliably store device-to-cloud messages in Azure blob storage. However, in some scenarios, you can't easily map the data your devices send into the relatively small device-to-cloud messages that IoT Hub accepts. For example:
- Videos
- Large files that contain images
- Vibration data sampled at high frequency
- Some form of preprocessed data.
These files are typically batch processed in the cloud, using tools such as Azure Data Factory or the Hadoop stack. When you need to upload files from a device, you can still use the security and reliability of IoT Hub. This article shows you how. View two samples from azure-iot-sdk-java in GitHub.
Note
IoT Hub supports many device platforms and languages (including C, .NET, and JavaScript) through Azure IoT device SDKs. Refer to the Azure IoT Developer Center to learn how to connect your device to Azure IoT Hub.
Important
File upload functionality on devices that use X.509 certificate authority (CA) authentication is in public preview, and preview mode must be enabled. It is generally available on devices that use X.509 thumbprint authentication or X.509 certificate attestation with Azure Device Provisioning Service. To learn more about X.509 authentication with IoT Hub, see Supported X.509 certificates.
Prerequisites
An IoT hub in your Azure subscription. If you don't have a hub yet, you can follow the steps in Create an IoT hub.
A device registered in your IoT hub. If you don't have a device in your IoT hub, follow the steps in Register a device.
Java SE Development Kit 8. Make sure you select Java 8 under Long-term support to get to downloads for JDK 8.
Port 8883 should be open in your firewall. The device sample in this article uses MQTT protocol, which communicates over port 8883. This port may be blocked in some corporate and educational network environments. For more information and ways to work around this issue, see Connecting to IoT Hub (MQTT).
Associate an Azure Storage account to IoT Hub
To upload files from a device, you must have an Azure Storage account and Azure Blob Storage container associated with your IoT hub. Once you associate the storage account and container with your IoT hub, your IoT hub can provide the elements of a SAS URI when requested by a device. The device can then use these elements to construct the SAS URI that it uses to authenticate with Azure Storage and upload files to the blob container.
To associate an Azure Storage account with your IoT hub:
Under Hub settings, select File upload on the left-pane of your IoT hub.
On the File upload pane, select Azure Storage Container. For this article, it's recommended that your storage account and IoT Hub be located in the same region.
If you already have a storage account you want to use, select it from the list.
To create a new storage account, select +Storage account. Provide a name for the storage account and make sure the Location is set to the same region as your IoT hub, then select OK. The new account is created in the same resource group as your IoT hub. When the deployment completes, select the storage account from the list.
After you select the storage account, the Containers pane opens.
On the Containers pane, select the blob container.
If you already have a blob container you want to use, select it from the list and click Select.
To create a new blob container, select + Container. Provide a name for the new container. For the purposes of this article, you can leave all other fields at their default. Select Create. When the deployment completes, select the container from the list and click Select.
Back on the File upload pane, make sure that file notifications are set to On. You can leave all other settings at their defaults. Select Save and wait for the settings to complete before moving on to the next section.
For more detailed instructions on how to create an Azure Storage account, see Create a storage account. For more detailed instructions on how to associate a storage account and blob container with an IoT hub, see Configure file uploads using the Azure portal.
Create a project using Maven
Create a directory for your project, and start a shell in that directory. On the command line, execute the following
mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false
This generates a directory with the same name as the artifactId and a standard project structure:
my-app
|-- pom.xml
-- src
-- main
-- java
-- com
-- mycompany
-- app
--App.Java
Using a text editor, replace the pom.xml file with the following:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>1.0-SNAPSHOT</version>
<name>my-app</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>com.microsoft.azure.sdk.iot</groupId>
<artifactId>iot-device-client</artifactId>
<version>1.30.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.29</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>with-deps</shadedClassifierName>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
Upload a file from a device app
Copy the file that you want to upload to the my-app folder in your project tree. Using a text editor, replace App.java with the following code. Supply your device connection string and file name where noted. You copied the device connection string when you registered the device.
package com.mycompany.app;
import com.azure.storage.blob.BlobClient;
import com.azure.storage.blob.BlobClientBuilder;
import com.microsoft.azure.sdk.iot.deps.serializer.FileUploadCompletionNotification;
import com.microsoft.azure.sdk.iot.deps.serializer.FileUploadSasUriRequest;
import com.microsoft.azure.sdk.iot.deps.serializer.FileUploadSasUriResponse;
import com.microsoft.azure.sdk.iot.device.DeviceClient;
import com.microsoft.azure.sdk.iot.device.IotHubClientProtocol;
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.net.URISyntaxException;
import java.util.Scanner;
public class App
{
/**
* Upload a single file to blobs using IoT Hub.
*
*/
public static void main(String[] args)throws IOException, URISyntaxException
{
String connString = "Your device connection string here";
String fullFileName = "Path of the file to upload";
System.out.println("Starting...");
System.out.println("Beginning setup.");
// File upload will always use HTTPS, DeviceClient will use this protocol only
// for the other services like Telemetry, Device Method and Device Twin.
IotHubClientProtocol protocol = IotHubClientProtocol.MQTT;
System.out.println("Successfully read input parameters.");
DeviceClient client = new DeviceClient(connString, protocol);
System.out.println("Successfully created an IoT Hub client.");
try
{
File file = new File(fullFileName);
if (file.isDirectory())
{
throw new IllegalArgumentException(fullFileName + " is a directory, please provide a single file name, or use the FileUploadSample to upload directories.");
}
System.out.println("Retrieving SAS URI from IoT Hub...");
FileUploadSasUriResponse sasUriResponse = client.getFileUploadSasUri(new FileUploadSasUriRequest(file.getName()));
System.out.println("Successfully got SAS URI from IoT Hub");
System.out.println("Correlation Id: " + sasUriResponse.getCorrelationId());
System.out.println("Container name: " + sasUriResponse.getContainerName());
System.out.println("Blob name: " + sasUriResponse.getBlobName());
System.out.println("Blob Uri: " + sasUriResponse.getBlobUri());
System.out.println("Using the Azure Storage SDK to upload file to Azure Storage...");
try
{
BlobClient blobClient =
new BlobClientBuilder()
.endpoint(sasUriResponse.getBlobUri().toString())
.buildClient();
blobClient.uploadFromFile(fullFileName);
}
catch (Exception e)
{
System.out.println("Exception encountered while uploading file to blob: " + e.getMessage());
System.out.println("Failed to upload file to Azure Storage.");
System.out.println("Notifying IoT Hub that the SAS URI can be freed and that the file upload failed.");
// Note that this is done even when the file upload fails. IoT Hub has a fixed number of SAS URIs allowed active
// at any given time. Once you are done with the file upload, you should free your SAS URI so that other
// SAS URIs can be generated. If a SAS URI is not freed through this API, then it will free itself eventually
// based on how long SAS URIs are configured to live on your IoT Hub.
FileUploadCompletionNotification completionNotification = new FileUploadCompletionNotification(sasUriResponse.getCorrelationId(), false);
client.completeFileUpload(completionNotification);
System.out.println("Notified IoT Hub that the SAS URI can be freed and that the file upload was a failure.");
client.closeNow();
return;
}
System.out.println("Successfully uploaded file to Azure Storage.");
System.out.println("Notifying IoT Hub that the SAS URI can be freed and that the file upload was a success.");
FileUploadCompletionNotification completionNotification = new FileUploadCompletionNotification(sasUriResponse.getCorrelationId(), true);
client.completeFileUpload(completionNotification);
System.out.println("Successfully notified IoT Hub that the SAS URI can be freed, and that the file upload was a success");
}
catch (Exception e)
{
System.out.println("On exception, shutting down \n" + " Cause: " + e.getCause() + " \nERROR: " + e.getMessage());
System.out.println("Shutting down...");
client.closeNow();
}
System.out.println("Press any key to exit...");
Scanner scanner = new Scanner(System.in);
scanner.nextLine();
System.out.println("Shutting down...");
client.closeNow();
}
}
Build and run the application
At a command prompt in the my-app folder, run the following command:
mvn clean package -DskipTests
When the build is complete, run the following command to run the application:
mvn exec:java -Dexec.mainClass="com.mycompany.app.App"
You can use the portal to view the uploaded file in the storage container you configured:
Receive a file upload notification
In this section, you create a Java console app that receives file upload notification messages from IoT Hub.
Create a directory for your project, and start a shell in that directory. On the command line, execute the following
mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=falseAt your command prompt, navigate to the new
my-appfolder.Using a text editor, replace the
pom.xmlfile in themy-appfolder with the following. Adding the service client dependency enables you to use the iothub-java-service-client package in your application to communicate with your IoT hub service:<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.mycompany.app</groupId> <artifactId>my-app</artifactId> <version>1.0-SNAPSHOT</version> <name>my-app</name> <!-- FIXME change it to the project's website --> <url>http://www.example.com</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.7</maven.compiler.source> <maven.compiler.target>1.7</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>com.microsoft.azure.sdk.iot</groupId> <artifactId>iot-device-client</artifactId> <version>1.30.1</version> </dependency> <dependency> <groupId>com.microsoft.azure.sdk.iot</groupId> <artifactId>iot-service-client</artifactId> <version>1.7.23</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <version>1.7.29</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.11</version> <scope>test</scope> </dependency> </dependencies> <build> <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) --> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.3</version> <configuration> <source>1.7</source> <target>1.7</target> </configuration> </plugin> <plugin> <artifactId>maven-shade-plugin</artifactId> <version>2.4</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> <shadedArtifactAttached>true</shadedArtifactAttached> <shadedClassifierName>with-deps</shadedClassifierName> </configuration> </execution> </executions> </plugin> </plugins> </pluginManagement> </build> </project>Note
You can check for the latest version of iot-service-client using Maven search.
Save and close the
pom.xmlfile.Get the IoT Hub service connection string.
To get the IoT Hub connection string for the service policy, follow these steps:
In the Azure portal, select Resource groups. Select the resource group where your hub is located, and then select your hub from the list of resources.
On the left-side pane of your IoT hub, select Shared access policies.
From the list of policies, select the service policy.
Copy the Primary connection string and save the value.
For more information about IoT Hub shared access policies and permissions, see Access control and permissions.
Using a text editor, open the
my-app\src\main\java\com\mycompany\app\App.javafile and replace the code with the following.package com.mycompany.app; import com.microsoft.azure.sdk.iot.service.*; import java.io.IOException; import java.net.URISyntaxException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; public class App { private static final String connectionString = "{Your service connection string here}"; private static final IotHubServiceClientProtocol protocol = IotHubServiceClientProtocol.AMQPS; public static void main(String[] args) throws Exception { ServiceClient sc = ServiceClient.createFromConnectionString(connectionString, protocol); FileUploadNotificationReceiver receiver = sc.getFileUploadNotificationReceiver(); receiver.open(); FileUploadNotification fileUploadNotification = receiver.receive(2000); if (fileUploadNotification != null) { System.out.println("File Upload notification received"); System.out.println("Device Id : " + fileUploadNotification.getDeviceId()); System.out.println("Blob Uri: " + fileUploadNotification.getBlobUri()); System.out.println("Blob Name: " + fileUploadNotification.getBlobName()); System.out.println("Last Updated : " + fileUploadNotification.getLastUpdatedTimeDate()); System.out.println("Blob Size (Bytes): " + fileUploadNotification.getBlobSizeInBytes()); System.out.println("Enqueued Time: " + fileUploadNotification.getEnqueuedTimeUtcDate()); } else { System.out.println("No file upload notification"); } receiver.close(); } }Save and close the
my-app\src\main\java\com\mycompany\app\App.javafile.Use the following command to build the app and check for errors:
mvn clean package -DskipTests
Run the application
Now you're ready to run the application.
At a command prompt in the my-app folder, run the following command:
mvn exec:java -Dexec.mainClass="com.mycompany.app.App"
The following screenshot shows the output from the read-file-upload-notification app:
Next steps
In this article, you learned how to use the file upload feature of IoT Hub to simplify file uploads from devices. You can continue to explore this feature with the following articles:
To further explore the capabilities of IoT Hub, see: