数据类型Data types

• 12/22/2020
• • m

支持的数据类型Supported data types

Apache Spark SQL 和 DataFrames 支持以下数据类型：Apache Spark SQL and DataFrames support the following data types:

• 数字类型Numeric types
  • ByteType：表示 1 个字节的带符号整数。ByteType: Represents 1-byte signed integer numbers. 数字范围是从 -128 到 127。The range of numbers is from -128 to 127.
  • ShortType：表示 2 个字节的带符号整数。ShortType: Represents 2-byte signed integer numbers. 数字范围是从 -32768 到 32767。The range of numbers is from -32768 to 32767.
  • IntegerType：表示 4 个字节的带符号整数。IntegerType: Represents 4-byte signed integer numbers. 数字范围是从 -2147483648 到 2147483647。The range of numbers is from -2147483648 to 2147483647.
  • LongType：表示 8 个字节的带符号整数。LongType: Represents 8-byte signed integer numbers. 数字范围是从 -9223372036854775808 到 9223372036854775807。The range of numbers is from -9223372036854775808 to 9223372036854775807.
  • FloatType：表示 4 个字节的单精度浮点数。FloatType: Represents 4-byte single-precision floating point numbers.
  • DoubleType：表示 8 个字节的双精度浮点数。DoubleType: Represents 8-byte double-precision floating point numbers.
  • DecimalType：表示任意精度的带符号十进制数字。DecimalType: Represents arbitrary-precision signed decimal numbers. 由 java.math.BigDecimal 在内部提供支持。Backed internally by java.math.BigDecimal. BigDecimal 由一个任意精度的非标度整数值和一个 32 位整数标度构成。A BigDecimal consists of an arbitrary precision integer unscaled value and a 32-bit integer scale.
• 字符串类型：String type:
  • StringType：表示字符串值。StringType: Represents character string values.
• 二进制类型：Binary type:
  • BinaryType：表示字节序列值。BinaryType: Represents byte sequence values.
• 布尔类型：Boolean type:
  • BooleanType：表示布尔值。BooleanType: Represents Boolean values.
• 日期/时间类型Datetime types
  • TimestampType：表示由字段 year、month、day、hour、minute 和 second 的值构成的值，使用会话本地时区。TimestampType: Represents values comprising values of fields year, month, day, hour, minute, and second, with the session local time-zone. 时间戳值表示绝对时间点。The timestamp value represents an absolute point in time.
  • DateType：表示由字段 year、month 和 day 的值构成的值，不包含时区。DateType: Represents values comprising values of fields year, month and day, without a time-zone.
• 复杂类型Complex types
  • ArrayType(elementType, containsNull)：表示由 elementType 类型的元素序列构成的值。ArrayType(elementType, containsNull): Represents values comprising a sequence of elements with the type of elementType. containsNull 指示 ArrayType 值中的元素是否可以具有 null 值。containsNull indicates if elements in a ArrayType value can have null values.
  • MapType(keyType, valueType, valueContainsNull)：表示由一组键值对构成的值。MapType(keyType, valueType, valueContainsNull): Represents values comprising a set of key-value pairs. 键的数据类型由 keyType 描述，而值的数据类型由 valueType 描述。The data type of keys is described by keyType and the data type of values is described by valueType. 对于 MapType 值，不允许键具有 null 值。For a MapType value, keys are not allowed to have null values. valueContainsNull 指示 MapType 值的值是否可以具有 null 值。valueContainsNull indicates if values of a MapType value can have null values.
  • StructType(fields)：表示多个值，其结构通过一系列 StructFields（字段）来描述。StructType(fields): Represents values with the structure described by a sequence of StructFields (fields).
    • StructField(name, dataType, nullable)：表示 StructType 内的字段。StructField(name, dataType, nullable): Represents a field in a StructType. 字段的名称由 name 指示。The name of a field is indicated by name. 字段的数据类型由 dataType 指示。The data type of a field is indicated by dataType. nullable 指示这些字段的值是否可以具有 null 值。nullable indicates if values of these fields can have null values.

语言映射Language mappings

ScalaScala

Spark SQL 数据类型是在包 org.apache.spark.sql.types 中定义的。Spark SQL data types are defined in the package org.apache.spark.sql.types. 可以通过导入此包来访问这些数据类型：You access them by importing the package:

import org.apache.spark.sql.types._

JavaJava

Spark SQL 数据类型是在包 org.apache.spark.sql.types 中定义的。Spark SQL data types are defined in the package org.apache.spark.sql.types. 若要访问或创建数据类型，请使用 org.apache.spark.sql.types.DataTypes 中提供的工厂方法。To access or create a data type, use factory methods provided in org.apache.spark.sql.types.DataTypes.

PythonPython

Spark SQL 数据类型是在包 pyspark.sql.types 中定义的。Spark SQL data types are defined in the package pyspark.sql.types. 可以通过导入此包来访问这些数据类型：You access them by importing the package:

from pyspark.sql.types import *

RR

SQLSQL

下表显示了每种数据类型的 Spark SQL 分析器中使用的类型名称和别名。The following table shows the type names as well as aliases used in Spark SQL parser for each data type.

特殊浮点值Special floating point values

Spark SQL 支持多个特殊的浮点值（不区分大小写）：Spark SQL supports several special floating point values in a case-insensitive manner:

• Inf、+Inf、Infinity、+Infinity：正无穷大Inf, +Inf, Infinity, +Infinity: positive infinity
  • FloatType：等效于 Scala Float.PositiveInfinity。FloatType: equivalent to Scala Float.PositiveInfinity.
  • DoubleType：等效于 Scala Double.PositiveInfinity。DoubleType: equivalent to Scala Double.PositiveInfinity.
• -Inf、-Infinity：负无穷大-Inf, -Infinity: negative infinity
  • FloatType：等效于 Scala Float.NegativeInfinity。FloatType: equivalent to Scala Float.NegativeInfinity.
  • DoubleType：等效于 Scala Double.NegativeInfinity。DoubleType: equivalent to Scala Double.NegativeInfinity.
• NaN：非数值NaN: not a number
  • FloatType：等效于 Scala Float.NaN。FloatType: equivalent to Scala Float.NaN.
  • DoubleType：等效于 Scala Double.NaN。DoubleType: equivalent to Scala Double.NaN.

正负无穷大语义Positive and negative infinity semantics

正负无穷大有特殊的处理方式。There is special handling for positive and negative infinity. 它们具有以下语义：They have the following semantics:

• 正无穷大乘以任何正值都会返回正无穷大。Positive infinity multiplied by any positive value returns positive infinity.
• 负无穷大乘以任何正值都会返回负无穷大。Negative infinity multiplied by any positive value returns negative infinity.
• 正无穷大乘以任何负值都会返回负无穷大。Positive infinity multiplied by any negative value returns negative infinity.
• 负无穷大乘以任何负值都会返回正无穷大。Negative infinity multiplied by any negative value returns positive infinity.
• 正/负无穷大乘以 0 返回 NaN。Positive/negative infinity multiplied by 0 returns NaN.
• 正/负无穷大等于自身。Positive/negative infinity is equal to itself.
• 在聚合中，所有正无穷大值都分组在一起。In aggregations, all positive infinity values are grouped together. 同样，所有负无穷大值都分组在一起。Similarly, all negative infinity values are grouped together.
• 正无穷大和负无穷大被视为联接键中的正常值。Positive infinity and negative infinity are treated as normal values in join keys.
• 正无穷大小于 NaN，并且大于任何其他值。Positive infinity sorts lower than NaN and higher than any other values.
• 负无穷大小于任何其他值。Negative infinity sorts lower than any other values.

NaN 语义NaN semantics

在处理与标准浮点语义不完全匹配的 float 或 double 类型时，NaN 具有特殊处理方式。There is special handling for NaN when dealing with float or double types that do not exactly match standard floating point semantics. 尤其是在下列情况下：Specifically:

• NaN = NaN 返回 true。NaN = NaN returns true.
• 在聚合中，所有 NaN 值都分组在一起。In aggregations, all NaN values are grouped together.
• NaN 被视为联接键中的正常值。NaN is treated as a normal value in join keys.
• NaN 值按升序排列后为最后一个值，大于任何其他数值。NaN values go last when in ascending order, larger than any other numeric value.

示例Examples

SELECT double('infinity') AS col;
+--------+
|     col|
+--------+
|Infinity|
+--------+

SELECT float('-inf') AS col;
+---------+
|      col|
+---------+
|-Infinity|
+---------+

SELECT float('NaN') AS col;
+---+
|col|
+---+
|NaN|
+---+

SELECT double('infinity') * 0 AS col;
+---+
|col|
+---+
|NaN|
+---+

SELECT double('-infinity') * (-1234567) AS col;
+--------+
|     col|
+--------+
|Infinity|
+--------+

SELECT double('infinity') < double('NaN') AS col;
+----+
| col|
+----+
|true|
+----+

SELECT double('NaN') = double('NaN') AS col;
+----+
| col|
+----+
|true|
+----+

SELECT double('inf') = double('infinity') AS col;
+----+
| col|
+----+
|true|
+----+

CREATE TABLE test (c1 int, c2 double);
INSERT INTO test VALUES (1, double('infinity'));
INSERT INTO test VALUES (2, double('infinity'));
INSERT INTO test VALUES (3, double('inf'));
INSERT INTO test VALUES (4, double('-inf'));
INSERT INTO test VALUES (5, double('NaN'));
INSERT INTO test VALUES (6, double('NaN'));
INSERT INTO test VALUES (7, double('-infinity'));
SELECT COUNT(*), c2 FROM test GROUP BY c2;
+---------+---------+
| count(1)|       c2|
+---------+---------+
|        2|      NaN|
|        2|-Infinity|
|        3| Infinity|
+---------+---------+