StreamingQueryListener

An abstract class for listening to events related to StreamingQuery. Subclass this class and implement its abstract methods to receive lifecycle event callbacks for streaming queries.

Syntax

from pyspark.sql.streaming import StreamingQueryListener

class MyListener(StreamingQueryListener):
    def onQueryStarted(self, event):
        pass

    def onQueryProgress(self, event):
        pass

    def onQueryIdle(self, event):
        pass

    def onQueryTerminated(self, event):
        pass

Methods

Method Description
onQueryStarted(event) Called when a query is started.
onQueryProgress(event) Called when there is some status update (ingestion rate updated, etc.)
onQueryIdle(event) Called when the query is idle and waiting for new data to process.
onQueryTerminated(event) Called when a query is stopped, with or without error.

Notes

The methods are not thread-safe as they may be called from different threads.

In Spark Connect mode, the listener does not have access to variables defined outside of it. Use self.spark instead of spark to access the session inside the listener in Connect mode.

Examples

from pyspark.sql.streaming import StreamingQueryListener

class MyListener(StreamingQueryListener):
    def onQueryStarted(self, event):
        # Do something with event.
        pass

    def onQueryProgress(self, event):
        # Do something with event.
        pass

    def onQueryIdle(self, event):
        # Do something with event.
        pass

    def onQueryTerminated(self, event):
        # Do something with event.
        pass

spark.streams.addListener(MyListener())
  • Last updated on