Adding support for long data type in Azure Time Series Insights Gen2
Note
The Time Series Insights service will be retired on 7 July 2024. Consider migrating existing environments to alternative solutions as soon as possible. For more information on the deprecation and migration, visit our documentation.
The addition of support for long data type affects how we store and index numeric data in Azure Time Series Insights Gen2 environments only. If you have a Gen1 environment, you can disregard these changes.
Beginning June 29 or June 30, 2020, depending on your region, your data will be indexed as Long and Double. If you have any questions or concerns about this change, submit a support ticket through the Azure portal and mention this communication.
If you're affected by any of the following cases, make the recommended changes:
- Case 1: You currently use Time Series Model variables and send only integral data types in your telemetry data.
- Case 2: You currently use Time Series Model variables and send both integral and nonintegral data types in your telemetry data.
- Case 3: You use categorical variables to map integer values to categories.
- Case 4: You use the JavaScript SDK to build a custom front-end application.
- Case 5: You're nearing the 1,000-property name limit in Warm Store and send both integral and nonintegral data. The property count can be viewed as a metric in the Azure portal.
If any of the cases apply to you, make changes to your model. Update the Time Series Expression (TSX) in your variable definition with the recommended changes. Update both:
- Azure Time Series Insights Explorer
- Any custom client that uses our APIs
Depending on your IoT solution and constraints, you might not have visibility into the data that's sent to your Azure Time Series Insights Gen2 environment. If you're unsure if your data is integral only or both integral and nonintegral, you have a few options:
- You can wait for the feature to be released. Then, explore your raw events in the explorer UI to understand which properties are saved in two separate columns.
- You can preemptively make the recommended changes for all numeric tags.
- You can temporarily route a subset of events to storage to better understand and explore your schema.
To store events, turn on event capture for Azure Event Hubs, or route from your IoT Hub to Azure Blob storage.
Data can also be observed through the Event Hub Explorer, or by using the Event Processor Host.
If you use IoT Hub, go to Read device-to-cloud messages from the built-in endpoint for how to access the built-in endpoint.
Note
You might experience a disruption if you don't make the recommended changes. For example, the affected Time Series Insights variables that are accessed via the query APIs or Time Series Insights explorer will return null (that is, show no data in the explorer).
Recommended changes
Case 1: Using Time Series Model variables and sending only integral data types in telemetry data
The recommended changes for Case 1 are the same as for Case 2. Follow the instructions in the section for Case 2.
Case 2: Using Time Series Model variables and sending both integral and nonintegral types in telemetry data
If you currently send integer telemetry data, your data will be divided into two columns:
- propertyValue_double
- propertyValue_long
Your integer data writes to propertyValue_long. Previously ingested (and future ingested) numeric data in propertyValue_double isn't copied over.
If you want to query data across these two columns for the propertyValue property, you need to use the coalesce() scalar function in your TSX. The function accepts arguments of the same DataType and returns the first non-null value in the argument list. For more information, see Azure Time Series Insights Gen2 data access concepts.
Variable definition in TSX - numeric
Previous variable definition:
New variable definition:
You can also use coalesce($event.propertyValue.Double, toDouble($event.propertyValue.Long)) as the custom Time Series Expression.
Inline variable definition using TSX query APIs - numeric
Previous variable definition:
"PropertyValueVariable": {
"kind": "numeric",
"value": {
"tsx": "$event.propertyValue.Double"
},
"filter": null,
"aggregation": {
"tsx": "avg($value)"
}
}
New variable definition:
"PropertyValueVariable ": {
"kind": "numeric",
"value": {
"tsx": "coalesce($event.propertyValue.Long, toLong($event.propertyValue.Double))"
},
"filter": null,
"aggregation": {
"tsx": "avg($value)"
}
}
You can also use coalesce($event.propertyValue.Double, toDouble($event.propertyValue.Long)) as the custom Time Series Expression.
Note
We recommend that you update these variables in all places they might be used. These places include Time Series Model, saved queries, and Power BI connector queries.
Case 3: Using categorical variables to map integer values to categories
If you currently use categorical variables that map integer values to categories, you're likely using the toLong function to convert data from Double type to Long type. Just like Cases 1 and 2, you need to coalesce the Double and Long DataType columns.
Variable definition in Time Series Explorer - categorical
Previous variable definition:
New variable definition:
You can also use coalesce($event.propertyValue.Double, toDouble($event.propertyValue.Long)) as the custom Time Series Expression.
Categorical variables still require the value to be of an integer type. The DataType of all the arguments in coalesce() must be of type Long in the custom Time Series Expression.
Inline variable definition using TSX query APIs - categorical
Previous variable definition:
"PropertyValueVariable_Long": {
"kind": "categorical",
"value": {
"tsx": "tolong($event.propertyValue.Double)"
},
"categories": [
{
"label": "Good",
"values": [0, 1, 2 ]
},
{
"label": "Bad",
"values": [ 3, 4 ]
} ],
"defaultCategory": {
"label": "Unknown"
}
}
New variable definition:
"PropertyValueVariable_Long": {
"kind": "categorical",
"value": {
"tsx": "coalesce($event.propertyValue.Long, tolong($event.propertyValue.Double))"
},
"categories": [
{
"label": "Good",
"values": [0, 1, 2 ]
},
{
"label": "Bad",
"values": [ 3, 4 ]
} ],
"defaultCategory": {
"label": "Unknown"
}
}
Categorical variables still require the value to be of an integer type. The DataType of all the arguments in coalesce() must be of type Long in the custom Time Series Expression.
Note
We recommend that you update these variables in all places they might be used. These places include Time Series Model, saved queries, and Power BI connector queries.
Case 4: Using the JavaScript SDK to build a custom front-end application
If you're affected by Cases 1 through 3 and build custom applications, you need to update your queries to use the coalesce() function, as demonstrated in the previous examples.
Case 5: Nearing Warm Store 1,000 property limit
If you're a Warm Store user with a large number of properties and believe that this change would push your environment over the 1,000 Warm Store property-name limit, submit a support ticket through the Azure portal and mention this communication.
Next steps
- View the full list of supported data types.