无法访问 Azure HDInsight 中的 Data Lake 存储文件Unable to access Data Lake storage files in Azure HDInsight

本文介绍在与 Azure HDInsight 群集交互时出现的问题的故障排除步骤和可能的解决方法。This article describes troubleshooting steps and possible resolutions for issues when interacting with Azure HDInsight clusters.

问题:ACL 验证失败Issue: ACL verification failed

收到如下所示的错误消息:You receive an error message similar to:

LISTSTATUS failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.).

原因Cause

用户可能已撤消服务主体 (SP) 对文件/文件夹的权限。The user might have revoked permissions of service principal(SP) on files/folders.

解决方法Resolution

  1. 检查 SP 是否拥有遍历路径的“x”权限。Check that the SP has 'x' permissions to traverse along the path. 有关详细信息,请参阅权限For more information, see Permissions. 用于检查对 Data Lake 存储帐户中文件/文件夹的访问权限的示例 dfs 命令:Sample dfs command to check access to files/folders in Data Lake storage account:

    hdfs dfs -ls /<path to check access>
    
  2. 根据正在执行的读/写操作设置访问路径所需的权限。Set up required permissions to access the path based on the read/write operation being performed. 了解各种文件系统操作所需的权限。See here for permissions required for various file system operations.


问题:服务主体证书过期Issue: Service principal certificate expiry

收到如下所示的错误消息:You receive an error message similar to:

Token Refresh failed - Received invalid http response: 500

原因Cause

为服务主体访问权限提供的证书可能已过期。The certificate provided for Service principal access might have expired.

  1. 通过 SSH 连接到头节点。SSH into headnode. 使用以下 dfs 命令检查对存储帐户的访问权限:Check access to storage account using following dfs command:

    hdfs dfs -ls /
    
  2. 确认错误消息类似于以下内容:Confirm that the error message is similar to the following:

    {"stderr": "-ls: Token Refresh failed - Received invalid http response: 500, text = Response{protocol=http/1.1, code=500, message=Internal Server Error, url=http://gw0-abccluster.24ajrd4341lebfgq5unsrzq0ue.fx.internal.chinacloudapp.cn:909/api/oauthtoken}}...
    
  3. core-site.xml property - fs.azure.datalake.token.provider.service.urls 获取一个URL。Get one of the urls from core-site.xml property - fs.azure.datalake.token.provider.service.urls.

  4. 运行以下 curl 命令检索 OAuth 令牌。Run the following curl command to retrieve OAuth token.

    curl gw0-abccluster.24ajrd4341lebfgq5unsrzq0ue.fx.internal.chinacloudapp.cn:909/api/oauthtoken
    
  5. 如果服务主体有效,则输出应如下所示:The output for a valid service principal should be something like:

    {"AccessToken":"MIIGHQYJKoZIhvcNAQcDoIIGDjCCBgoCAQA…….","ExpiresOn":1500447750098}
    
  6. 如果服务主体证书已过期,则输出如下所示:If the service principal certificate has expired, the output will look something like this:

    Exception in OAuthTokenController.GetOAuthToken: 'System.InvalidOperationException: Error while getting the OAuth token from AAD for AppPrincipalId 23abe517-2ffd-4124-aa2d-7c224672cae2, ResourceUri https://management.core.chinacloudapi.cn/, AADTenantId https://login.windows.net/80abc8bf-86f1-41af-91ab-2d7cd011db47, ClientCertificateThumbprint C49C25705D60569884EDC91986CEF8A01A495783 ---> Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException: AADSTS70002: Error validating credentials. AADSTS50012: Client assertion contains an invalid signature. **[Reason - The key used is expired.**, Thumbprint of key used by client: 'C49C25705D60569884EDC91986CEF8A01A495783', Found key 'Start=08/03/2016, End=08/03/2017, Thumbprint=C39C25705D60569884EDC91986CEF8A01A4956D1', Configured keys: [Key0:Start=08/03/2016, End=08/03/2017, Thumbprint=C39C25705D60569884EDC91986CEF8A01A4956D1;]]
    Trace ID: e4d34f1c-a584-47f5-884e-1235026d5000
    Correlation ID: a44d870e-6f23-405a-8b23-9b44aebfa4bb
    Timestamp: 2017-10-06 20:44:56Z ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpWebRequestWrapper.<GetResponseSyncOrAsync>d__2.MoveNext()
    
  7. 可以通过 ping 网关 URL 获取 OAuth 令牌,来识别任何其他 Azure Active Directory 相关错误/证书相关错误。Any other Azure Active Directory related errors/certificate related errors can be recognized by pinging the gateway url to get the OAuth token.

  8. 如果尝试从 HDI 群集访问 ADLS 时遇到以下错误:If you are getting following error when attempting to access ADLS from the HDI Cluster. 按上述步骤检查证书是否已过期。Check if the Certificate has Expired by following the steps mentioned above.

    Error: java.lang.IllegalArgumentException: Token Refresh failed - Received invalid http response: 500, text = Response{protocol=http/1.1, code=500, message=Internal Server Error, url=http://clustername.hmssomerandomstringc.cx.internal.chinacloudapp.cn:909/api/oauthtoken}
    

解决方法Resolution

使用以下 PowerShell 脚本创建新证书或分配现有证书:Create a new Certificate or assign existing Certificate using the following PowerShell script:

$clusterName = 'CLUSTERNAME'
$resourceGroupName = 'RGNAME'
$subscriptionId = 'SUBSCRIPTIONID'
$appId = 'APPLICATIONID'
$generateSelfSignedCert = $false
$addNewCertKeyCredential = $true
$certFilePath = 'NEW_CERT_PFX_LOCAL_PATH'
$certPassword = Read-Host "Enter Certificate Password"

if($generateSelfSignedCert)
{
    Write-Host "Generating new SelfSigned certificate"
    
    $cert = New-SelfSignedCertificate -CertStoreLocation "cert:\CurrentUser\My" -Subject "CN=hdinsightAdlsCert" -KeySpec KeyExchange
    $certBytes = $cert.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Pkcs12, $certPassword);
    $certString = [System.Convert]::ToBase64String($certBytes)
}
else
{

    Write-Host "Reading the cert file from path $certFilePath"

    $cert = new-object System.Security.Cryptography.X509Certificates.X509Certificate2($certFilePath, $certPassword)
    $certString = [System.Convert]::ToBase64String([System.IO.File]::ReadAllBytes($certFilePath))
}


Login-AzureRmAccount

if($addNewCertKeyCredential)
{
    Write-Host "Creating new KeyCredential for the app"

    $keyValue = [System.Convert]::ToBase64String($cert.GetRawCertData())
    
    New-AzureRmADAppCredential -ApplicationId $appId -CertValue $keyValue -EndDate $cert.NotAfter -StartDate $cert.NotBefore

    Write-Host "Waiting for 30 seconds for the permissions to get propagated"

    Start-Sleep -s 30
}

Select-AzureRmSubscription -SubscriptionId $subscriptionId

Write-Host "Updating the certificate on HDInsight cluster."

Invoke-AzureRmResourceAction `
    -ResourceGroupName $resourceGroupName `
    -ResourceType 'Microsoft.HDInsight/clusters' `
    -ResourceName $clusterName `
    -ApiVersion '2015-03-01-preview' `
    -Action 'updateclusteridentitycertificate' `
    -Parameters @{ ApplicationId = $appId.ToString(); Certificate = $certString; CertificatePassword = $certPassword.ToString() } `
    -Force

若要分配现有证书,请创建一个证书,并准备好 .pfx 文件和密码。For assigning existing certificate, create a certificate, have the .pfx file and password ready. 使用准备好的 AppId 将证书与创建群集时使用的服务主体相关联。Associate the certificate with the service principal that the cluster was created with, using the AppId ready.

将参数替换为实际值后,执行 PowerShell 命令。Execute the PowerShell command after you substitute the parameters with the actual values.

后续步骤Next steps

如果你的问题未在本文中列出,或者无法解决问题,请访问以下渠道以获取更多支持:If you didn't see your problem or are unable to solve your issue, visit the following channel for more support: