onQueryProgress (StreamingQueryListener)

Called when there is some status update (ingestion rate updated, etc.)

Syntax

onQueryProgress(event)

Parameters

Returns

None

Notes

This method is asynchronous. The status in StreamingQuery will always be the latest no matter when this method is called. Therefore, the status of StreamingQuery may be changed before or when you process the event. For example, you may find StreamingQuery is terminated when you are processing QueryProgressEvent.

Examples

from pyspark.sql.streaming import StreamingQueryListener

class MyListener(StreamingQueryListener):
    def onQueryStarted(self, event):
        pass

    def onQueryProgress(self, event):
        print(f"Query progress: {event.progress.numInputRows} rows processed")

    def onQueryIdle(self, event):
        pass

    def onQueryTerminated(self, event):
        pass

spark.streams.addListener(MyListener())