作业因 Azure Data Lake Storage (ADLS) CREATE 限制而失败Job failure due to Azure Data Lake Storage (ADLS) CREATE limits

问题Problem

当运行的作业涉及到在 Azure Data Lake Storage (ADLS)(Gen2)中创建文件时,会出现以下异常:When you run a job that involves creating files in Azure Data Lake Storage (ADLS), Gen2, the following exception occurs:

Caused by: java.io.IOException: CREATE failed with error 0x83090c25 (Files and folders are being created at too high a rate). [745c5836-264e-470c-9c90-c605f1c100f5] failed with error 0x83090c25 (Files and folders are being created at too high a rate). [2019-04-12T10:06:43.1117197-07:00] [ServerRequestId:745c5836-264e-470c-9c90-c605f1c100f5]
at com.microsoft.azure.datalake.store.ADLStoreClient.getRemoteException(ADLStoreClient.java:1191)
at com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1154)
at com.microsoft.azure.datalake.store.ADLStoreClient.createFile(ADLStoreClient.java:281)
at com.databricks.adl.AdlFileSystem.create(AdlFileSystem.java:348)
at com.databricks.spark.metrics.FileSystemWithMetrics.create(FileSystemWithMetrics.scala:280)
at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2$$anonfun$create$1$$anonfun$apply$10$$anonfun$apply$11.apply(DatabricksFileSystemV2.scala:483)

原因Cause

每个 ADLS 订阅级别都限制了每个时间单位内可创建的文件数。Each ADLS subscription level has a limit on the number of files that can be created per unit of time. 超出限制时,将中止创建文件,并且作业会失败。When the limit is exceeded, file creation is throttled, and the job fails.

此错误的潜在原因包括:Potential causes for this error include:

  • 你的应用程序创建了大量小文件。Your application creates a large number of small files.
  • 外部应用程序创建了大量文件。External applications create a large number of files.
  • 订阅的当前限制过低。The current limit for the subscription is too low.

解决方案Solution

如果你的应用程序或外部应用程序将要生成大量文件,你需要优化应用程序。If your application or an external application is generating a large number of files, then you need to optimize the application. 如果当前订阅的限制不适用于你的用例,请联系 Microsoft Azure 团队获取帮助。If the limit on your current subscription is not appropriate for your use case, then contact the Microsoft Azure Team for assistance.