Enrich Delta Lake tables with custom metadata

  • Article

Databricks recommends always providing comments for tables and columns in tables. You can generate these comments using AI.

Unity Catalog also provides the ability to tag data. See Apply tags to Unity Catalog securable objects.

You can also log messages for individual commits to tables in a field in the Delta Lake transaction log.

Set user-defined commit metadata

You can specify user-defined strings as metadata in commits, either using the DataFrameWriter option userMetadata or the SparkSession configuration spark.databricks.delta.commitInfo.userMetadata. If both of them have been specified, then the option takes preference. This user-defined metadata is readable in the DESCRIBE HISTORY operation. See Work with Delta Lake table history.

SQL


SET spark.databricks.delta.commitInfo.userMetadata=some-comment
INSERT OVERWRITE target_table SELECT * FROM data_source

Python

df.write \
  .mode("overwrite") \
  .option("userMetadata", "some-comment") \
  .table("target_table")

Scala

df.write
  .mode("overwrite")
  .option("userMetadata", "some-comment")
  .table("target_table")