Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Registers a progress handler to be called when a progress update is received from the server.
Syntax
registerProgressHandler(handler)
Parameters
| Parameter | Type | Description |
|---|---|---|
handler |
ProgressHandler | A callable that follows the ProgressHandler interface. This handler is called on every progress update. |
Returns
None
Notes
This method is only supported with Spark Connect. With a regular Spark session it raises an exception.
Examples
def progress_handler(stages, inflight_tasks, done):
print(f"{len(stages)} Stages known, Done: {done}")
spark.registerProgressHandler(progress_handler)
res = spark.range(10).repartition(1).collect()
# 3 Stages known, Done: False
# 3 Stages known, Done: True
spark.clearProgressHandlers()