Delta LakeDelta Lake

• 09/03/2020
• • m

Delta Lake 是可以提高 Data Lake 可靠性的开源存储层。Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake 提供 ACID 事务和可缩放的元数据处理，并可以统一流处理和批数据处理。Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake 在现有 Data Lake 的顶层运行，与 Apache Spark API 完全兼容。Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs.

利用 Azure Databricks 上的 Delta Lake，便可以根据工作负载模式配置 Delta Lake。Delta Lake on Azure Databricks allows you to configure Delta Lake based on your workload patterns. Azure Databricks 还包括 Delta Engine，这为快速交互式查询提供了优化的布局和索引。Azure Databricks also includes Delta Engine, which provides optimized layouts and indexes for fast interactive queries.

本部分介绍 Azure Databricks 上的 Delta Lake。This section covers Delta Lake on Azure Databricks.

• 简介Introduction
  • 快速入门Quickstart
  • 资源Resources
• Delta Lake 快速入门Delta Lake quickstart
  • 创建表Create a table
  • 修改表Modify a table
  • 读取表Read a table
  • 优化表Optimize a table
  • 清理快照Clean up snapshots
• 介绍性笔记本Introductory notebooks
  • Delta Lake 快速入门 Python 笔记本Delta Lake Quickstart Python notebook
  • Delta Lake 快速入门 Scala 笔记本Delta Lake Quickstart Scala notebook
  • Delta Lake 快速入门 SQL 笔记本Delta Lake Quickstart SQL notebook
• 将数据引入到 Delta LakeIngest data into Delta Lake
  • 合作伙伴集成Partner integrations
  • COPY INTO SQL 命令COPY INTO SQL command
  • 自动加载程序Auto Loader
• 表批量读取和写入Table batch reads and writes
  • 创建表Create a table
  • 读取表Read a table
  • 写入表Write to a table
  • 架构验证Schema validation
  • 更新表架构Update table schema
  • 替换表架构Replace table schema
  • 表中的视图Views on tables
  • 表属性Table properties
  • 表元数据Table metadata
  • 笔记本Notebook
• 表流读取和写入Table streaming reads and writes
  • 用作流源的增量表Delta table as a stream source
  • 用作接收器的增量表Delta table as a sink
• 表删除、更新和合并Table deletes, updates, and merges
  • 从表中删除Delete from a table
  • 更新表Update a table
  • 使用合并操作在表中执行更新插入Upsert into a table using merge
  • 合并操作示例Merge examples
• 表实用工具命令Table utility commands
  • 清空Vacuum
  • 描述历史记录Describe History
  • 描述详细信息Describe Detail
  • Convert to DeltaConvert to Delta
  • 将 Delta 表转换为 Parquet 表Convert a Delta table to a Parquet table
  • 克隆 Delta 表Clone a Delta table
• 表版本控制Table versioning
• API 参考API reference
• 并发控制Concurrency control
  • 乐观并发控制Optimistic concurrency control
  • 写冲突Write conflicts
  • 使用分区和非连续命令条件来避免冲突Avoid conflicts using partitioning and disjoint command conditions
  • 冲突异常Conflict exceptions
• 迁移指南Migration guide
  • 将工作负荷迁移到 Delta LakeMigrate workloads to Delta Lake
• 最佳做法Best practices
  • 提供数据位置提示Provide data location hints
  • 选择正确的分区列Choose the right partition column
  • 压缩文件Compact files
  • 替换表的内容或架构Replace the content or schema of a table
• 常见问题解答 (FAQ)Frequently asked questions (FAQ)
  • 什么是 Delta Lake？What is Delta Lake?
  • Delta Lake 与 Apache Spark 之间存在何种关系？How is Delta Lake related to Apache Spark?
  • Delta Lake 使用哪种格式存储数据？What format does Delta Lake use to store data?
  • 如何使用 Delta Lake 读取和写入数据？How can I read and write data with Delta Lake?
  • Delta Lake 将数据存储在何处？Where does Delta Lake store the data?
  • 是否可将数据直接流式传入和流式传出 Delta 表？Can I stream data directly into and from Delta tables?
  • Delta Lake 是否支持使用 Spark Streaming DStream API 写入或读取数据？Does Delta Lake support writes or reads using the Spark Streaming DStream API?
  • 使用 Delta Lake 时，是否可以轻松将代码移植到其他 Spark 平台？When I use Delta Lake, will I be able to port my code to other Spark platforms easily?
  • 增量表与 Hive SerDe 表之间有何差别？How do Delta tables compare to Hive SerDe tables?
  • Delta Lake 不支持哪些 DDL 和 DML 功能？What DDL and DML features does Delta Lake not support?
  • Delta Lake 是否支持多表事务？Does Delta Lake support multi-table transactions?
  • 如何更改列的类型？How can I change the type of a column?
  • Delta Lake 支持多群集写入是什么意思？What does it mean that Delta Lake supports multi-cluster writes?
  • 是否可从不同的工作区修改增量表？Can I modify a Delta table from different workspaces?
  • 是否可以在 Databricks Runtime 的外部访问增量表？Can I access Delta tables outside of Databricks Runtime?
• 资源Resources
  • 博客文章Blog posts
  • 讲座Talks
  • 示例Examples
  • Delta Lake 事务日志规范Delta Lake transaction log specification