Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Streaming real-time mode has the following limitation:
The arbitrary stateful streaming operator, <op>, is not currently supported. Please see the Real-Time Mode User Guide for a list of supported operators.
Async progress tracking is not supported in real-time mode. Set option asyncProgressTrackingEnabled to false and retry your query.
Async state checkpointing is not supported in real-time mode. Set spark.databricks.streaming.statefulOperator.asyncCheckpoint.enabled to false and retry your query.
The real-time mode requires checkpoint v2. Set spark.sql.streaming.stateStore.checkpointFormatVersion to 2 and retry your query.
The real-time mode requires the concurrent stage DAG scheduler. Restart your spark cluster and set spark.databricks.dagScheduler.type to ConcurrentStageDAGScheduler.
dropDuplicatesWithinWatermark is not supported. To work around this limitation, you can use dropDuplicates instead. Remember to pass your event-time column to dropDuplicates so that state can be evicted.
Event time based timers in transformWithState operator are not supported yet. If you would like to use the transformWithState operator, please specify processing time in your transformWithState operator and retry your query.
The real-time mode requires the multi-shuffle manager. Set spark.shuffle.manager to org.apache.spark.shuffle.streaming.MultiShuffleManager and restart your spark cluster.
The output mode <outputMode> is not supported. To work around this limitation, set the output mode to Update. In the future, <outputMode> may be supported.
The real-time mode requires the RocksDB state store provider or providers that extends it. Set spark.sql.streaming.stateStore.providerClass to org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider and retry your query.
Session windows are not yet supported. The only windowed aggregations allowed are over tumbling and sliding windows.
Speculative execution is not supported in real-time mode. Set spark.speculation to false and restart your spark cluster.
No type of streaming joins are supported yet.