Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Streaming real-time mode has the following limitation:
The arbitrary stateful streaming operator, <op>, is not currently supported. Please see the Real-Time Mode User Guide for a list of supported operators.
Async progress tracking is not supported in real-time mode. Set option asyncProgressTrackingEnabled to false and retry your query.
Async state checkpointing is not supported in real-time mode. Set spark.databricks.streaming.statefulOperator.asyncCheckpoint.enabled to false and retry your query.
Real-time mode does not support union with batch data sources. Ensure that the union streaming query only contains streaming data sources.
The following configuration(s) are not supported:
<invalidReasons>. Set spark.databricks.streaming.realTimeMode.enabled to true in your cluster configuration, and remove <confNames> from your cluster configuration if present, and then restart your Spark cluster.
dropDuplicatesWithinWatermark is not supported. To work around this limitation, you can use dropDuplicates instead. Remember to pass your event-time column to dropDuplicates so that state can be evicted.
Event time based timers in transformWithState operator are not supported yet. If you would like to use the transformWithState operator, please specify processing time in your transformWithState operator and retry your query.
Real-time mode does not support union on two or more identical streaming data sources in a single query. This includes scenarios such as referencing the same source DataFrame more than once, or using two data sources with identical configurations for some sources. For Kafka, avoid reusing the same DataFrame and create different ones. For Kinesis, in addition to duplicating data frames, ensure that each DataFrame sets a unique value for the "consumerName" option. Sources provided in the query: <sources>
The input stream <className> is not supported. See the Real-Time Mode User Guide for a list of supported input streams.
The <errorType>(s): <message> not in the <errorType> allowlist for Real-Time Mode. To bypass this check, set spark.databricks.streaming.realTimeMode.allowlistCheck to false. By changing this, you agree to run the query at your own risk.
The output mode <outputMode> is not supported. To work around this limitation, set the output mode to Update. In the future, <outputMode> may be supported.
Session windows are not yet supported. The only windowed aggregations allowed are over tumbling and sliding windows.
There are <shuffleCount> org.apache.spark.sql.execution.exchange.ShuffleExchangeExec operators in the physical plan. Likely, you are calling the operations that require shuffle more than once, and RTM does not support this. To run the query, set spark.databricks.streaming.realTimeMode.allowlistCheck to false. By changing this, you agree to run the query at your own risk.
The <className> sink is currently not supported. See the Real-Time Mode User Guide for a list of supported sinks.
The following configuration(s) are not supported:
<invalidReasons>. Set spark.databricks.streaming.realTimeMode.enabled to true in your cluster configuration, and remove <confNames> from your SparkSession configuration if present, and then restart your Spark cluster.
Streaming queries in real-time mode cannot include stateful operators (e.g. aggregate, deduplicate, transformWithState) before a union. Please restructure your query to apply the union before any stateful operations.
No type of streaming joins are supported yet.