Parse Protobuf in Azure Stream Analytics

Azure Stream Analytics supports processing events in Protocol Buffer (Protobuf) data formats. You can use the built-in Protobuf deserializer when configuring your inputs. To use the built-in deserializer, specify the Protobuf definition file, message type, and prefix style.

To configure your Stream Analytics job to deserialize events in Protobuf:

1. After you create your Stream Analytics job, select Inputs.

2. Select Add input, and then select what input you want to configure to open the pane for input configuration.

3. Select Event serialization format to show a dropdown list, and then select Protobuf.

   

4. Complete the configuration by using the following guidance:

   Expand table

   Property name	Description
   Protobuf definition file	A file that specifies the structure and data types of your Protobuf events
   Message type	The message type that you want to deserialize
   Prefix style	The setting that determines how long a message is to deserialize Protobuf events correctly

   

To learn more about Protobuf data types, see the official Protocol Buffers documentation.

• The Protobuf deserializer takes only one Protobuf definition file at a time. Imports to custom-made Protobuf definition files aren't supported. For example:

  

  This Protobuf definition file refers to another Protobuf definition file in its imports. Because the Protobuf deserializer would have only the current Protobuf definition file and not know what carseat.proto is, it would be unable to deserialize correctly.

• When a Protobuf definition file contains a namespace or package, the message type must include it. For example:

  

  In the Protobuf deserializer in the portal, the message type must be Namespacetest.Volunteer instead of the usual Volunteer.

• If the original package or namespace is all uppercase or lowercase, the message type must include it with the first character of the package or namespace in uppercase. For example, if the namespace is nameSpaceTest, use nameSpaceTest.Volunteer. This rule applies to a message type without a namespace or package.

• You must specify your message in your protobuf definition file without the Optional keyword. In Proto 3, all fields are optional. For example:

  

  This Protobuf definition file shows a message that has Optional keyword. To deserialize correctly, you must remove the keyword:

  

• When sending messages that were serialized via google.protobuf, the prefix type should be set to base128 because that's the most cross-compatible type.

• Service messages aren't supported in the Protobuf deserializers. Your job throws an exception if you try to use a service message. For example:

  

• These data types aren't supported:

  • Any
  • One of (related to enumerations)
  • Durations
  • Struct
  • Field Mask (not supported by protobuf-net)
  • List Value
  • Value
  • Null Value
  • Empty