Azure HDInsight 中的企业安全性常规信息和准则Enterprise security general information and guidelines in Azure HDInsight

部署安全的 HDInsight 群集时,可以遵循一些最佳做法来简化部署和群集管理。When deploying a secure HDInsight cluster, there are some best practices that should make the deployment and cluster management easier. 本文介绍了一些常规信息和准则。Some general information and guidelines are discussed here.

使用安全群集Use of secure cluster

  • 群集将由多个用户同时使用。Cluster will be used by multiple users at the same time.
  • 不同用户对同一数据的访问权限级别不同。Users have different levels of access to the same data.

不必要Not necessary

  • 你只需运行自动化作业(像单用户帐户一样),使用标准群集就足够了。You're going to run only automated jobs (like single user account), a standard cluster is good enough.
  • 你可以使用标准群集执行数据导入,在可供用户运行分析作业的不同安全群集上使用同一存储帐户。You can do the data import using a standard cluster and use the same storage account on a different secure cluster where users can run analytics jobs.

使用本地帐户Use of local account

  • 如果你使用共享用户帐户或本地帐户,则很难确定使用过该帐户更改配置或服务的人员。If you use a shared user account or a local account, then it will be difficult to identify who used the account to change the config or service.
  • 当用户不再是组织的成员时,使用本地帐户会出现问题。Using local accounts is problematic when users are no longer part of the organization.

RangerRanger

策略Policies

  • 默认情况下,Ranger 使用“拒绝”作为策略。By default, Ranger uses Deny as the policy.

  • 通过在其中启用了授权的服务进行数据访问时:When data access is made through a service where authorization is enabled:

    • 会调用 Ranger 授权插件并为其提供请求上下文。Ranger authorization plugin is invoked and given the context of the request.
    • Ranger 应用为服务配置的策略。Ranger applies the policies configured for the service. 如果 Ranger 策略失败,则会将访问检查推迟到文件系统。If the Ranger policies fail, the access check is deferred to the file system. 某些服务(如 MapReduce)仅检查提交请求的用户所拥有的文件/文件夹。Some services like MapReduce only check if the file / folder being owned by the same user who is submitting the request. Hive 之类的服务会检查所有权是否匹配或文件系统权限是否适当 (rwx)。Services like Hive, check for either ownership match or appropriate filesystem permissions (rwx).
  • 对于 Hive,用户除了拥有执行创建/更新/删除操作的权限外,还应当对存储上的目录和所有子目录拥有 rwx 权限。For Hive, in addition to having the permissions to do Create / Update / Delete permissions, the user should have rwxpermissions on the directory on storage and all sub directories.

  • 可以将策略应用于组(首选)而不是个人。Policies can be applied to groups (preferable) instead of individuals.

  • Ranger 授权程序会针对每个请求评估针对该服务的所有 Ranger 策略。Ranger authorizer will evaluate all Ranger policies for that service for each request. 此评估可能会影响接受作业或查询所耗费的时间。This evaluation could have an impact on the time take to accept the job or query.

存储访问Storage access

  • 如果存储类型为 WASB,则不涉及任何 OAuth 令牌。If the storage type is WASB, then no OAuth token is involved.
  • 如果 Ranger 已执行授权操作,则可使用托管标识进行存储访问。If Ranger has performed the authorization, then the storage access happens using the Managed Identity.
  • 如果 Ranger 未执行任何授权操作,则会使用用户的 OAuth 令牌进行存储访问。If Ranger didn't perform any authorization, then the storage access happens using the user's OAuth token.

分层命名空间Hierarchical name space

如果未启用分层命名空间,则:When hierarchical name space in not enabled:

  • 没有继承的权限。There are no inherited permissions.
  • 能够使用的唯一文件系统权限是存储数据 XXXX Azure 角色,该角色直接在 Azure 门户中分配给用户。Only filesystem permission that works is Storage Data XXXX Azure role, to be assigned to the user directly in Azure portal.

默认的 HDFS 权限Default HDFS permissions

  • 默认情况下,用户无权访问 HDFS 上的 / 文件夹(需要具有存储 blob 所有者角色才能成功访问)。By default, users don't have access to the / folder on HDFS (they need to be in the storage blob owner role for access to succeed).
  • 对于 MapReduce 和其他服务的暂存目录,系统会创建特定于用户的目录,并为其提供 sticky _wx 权限。For the staging directory for mapreduce and others, a user-specific directory is created and provided sticky _wx permissions. 用户可以在其下创建文件和文件夹,但不能查看其他项目。Users can create files and folders underneath, but can't look at other items.

URL 身份验证URL auth

如果启用了 URL 身份验证,则:If the url auth is enabled:

  • 配置将涉及 URL 身份验证中包含哪些前缀(例如 adl://)。The config will contain what prefixes are covered in the url auth (like adl://).
  • 如果访问权限是针对此 URL 的,则 Ranger 会检查用户是否在允许列表中。If the access is for this url, then Ranger will check if the user is in the allow list.
  • Ranger 不检查任何细化策略。Ranger won't check any of the fine grained policies.

资源组Resource groups

为每个群集使用新的资源组,以便区分群集资源。Use a new resource group for each cluster so that you can distinguish between cluster resources.

NSG、防火墙和内部网关NSGs, firewalls, and internal gateway

  • 使用网络安全组 (NSG) 锁定虚拟网络。Use network security groups (NSGs) to lock down virtual networks.
  • 使用防火墙处理出站访问策略。Use firewall to handle outbound access policies.
  • 请使用未向公共 Internet 开放的内部网关。Use the internal gateway that isn't open to the public internet.

Azure Active DirectoryAzure Active Directory

Azure Active Directory (Azure AD) 是 Microsoft 推出的基于云的标识和访问管理服务。Azure Active Directory (Azure AD) is Microsoft's cloud-based identity and access management service.

策略Policies

  • 禁用使用了基于 IP 地址的策略的条件访问策略。Disable conditional access policy using the IP address based policy. 这要求在部署群集的 VNET 上启用服务终结点。This requires service endpoints to be enabled on the VNETs where the clusters are deployed. 如果对 MFA 使用外部服务(不同于 AAD 的服务),则基于 IP 地址的策略将不起作用If you use an external service for MFA (something other than AAD), the IP address based policy won't work

  • 对于联合用户,AllowCloudPasswordValidation 策略是必需的。AllowCloudPasswordValidation policy is required for federated users. 由于 HDInsight 直接使用用户名/密码从 Azure AD 获取令牌,因此必须为所有联合用户启用此策略。Since HDInsight uses the username / password directly to get tokens from Azure AD, this policy has to be enabled for all federated users.

  • 如果需要使用受信任的 IP 来绕过条件访问,请启用服务终结点。Enable service endpoints if you require conditional access bypass using Trusted IPs.

Groups

  • 始终通过组来部署群集。Always deploy clusters with a group.
  • 使用 Azure AD 来管理组成员身份(比尝试管理群集中的各项服务更轻松)。Use Azure AD to manage group memberships (easier than trying to manage the individual services in the cluster).

用户帐户User accounts

  • 为每个方案使用唯一的用户帐户。Use a unique user account for each scenario. 例如,将一个帐户用于导入,将另一个帐户用于查询或其他处理作业。For example, use an account for import, use another for query or other processing jobs.
  • 使用基于组的 Ranger 策略,而不是使用个体策略。Use group-based Ranger policies instead of individual policies.
  • 计划如何管理不应再访问群集的用户。Have a plan on how to manage users who shouldn't have access to clusters anymore.

Azure Active Directory 域服务Azure Active Directory Domain Services

Azure Active Directory 域服务 (Azure AD DS) 提供与 Windows Server Active Directory 完全兼容的托管域服务,例如域加入、组策略、轻型目录访问协议 (LDAP) 和 Kerberos/NTLM 身份验证。Azure Active Directory Domain Services (Azure AD DS) provides managed domain services such as domain join, group policy, lightweight directory access protocol (LDAP), and Kerberos / NTLM authentication that is fully compatible with Windows Server Active Directory.

若要让安全群集加入域,Azure AD DS 是必需的。Azure AD DS is required for secure clusters to join a domain. HDInsight 不能依赖于本地域控制器或自定义域控制器,因为它引入了太多的故障点、凭据共享、DNS 权限等。HDInsight can't depend on on-premise domain controllers or custom domain controllers, as it introduces too many fault points, credential sharing, DNS permissions, and so on. 有关详细信息,请参阅 Azure AD DS 常见问题解答For more information, see Azure AD DS FAQs.

Azure AD DS 实例Azure AD DS instance

  • 创建使用 .onmicrosoft.com domain 的实例。Create the instance with the .onmicrosoft.com domain. 这样就不会有多个 DNS 服务器为域提供服务。This way, there won’t be multiple DNS servers serving the domain.
  • 为 LDAPS 创建自签名证书并将其上传到 Azure AD DS。Create a self-signed certificate for the LDAPS and upload it to Azure AD DS.
  • 使用对等互连虚拟网络来部署群集(当你有许多团队部署 HDInsight ESP 群集时,这会很有用)。Use a peered virtual network for deploying clusters (when you have a number of teams deploying HDInsight ESP clusters, this will be helpful). 这可以确保你不需要在具有域控制器的虚拟网络上打开端口 (NSG)。This ensures that you don't need to open up ports (NSGs) on the virtual network with domain controller.
  • 为虚拟网络正确配置 DNS(Azure AD DS 域名应该可以在没有任何 hosts 文件条目的情况下进行解析)。Configure the DNS for the virtual network properly (the Azure AD DS domain name should resolve without any hosts file entries).
  • 如果要限制出站流量,请确保已通读 HDInsight 中的防火墙支持一文If you're restricting outbound traffic, make sure that you have read through the firewall support in HDInsight

从 Azure AD 同步到 Azure AD DS 的属性Properties synced from Azure AD to Azure AD DS

  • Azure AD Connect 从本地同步到 Azure AD。Azure AD connect syncs from on-premise to Azure AD.
  • Azure AD DS 从 Azure AD 同步。Azure AD DS syncs from Azure AD.

Azure AD DS 定期从 Azure AD 同步对象。Azure AD DS syncs objects from Azure AD periodically. Azure 门户上的 Azure AD DS 边栏选项卡会显示同步状态。The Azure AD DS blade on the Azure portal displays the sync status. 在每个同步阶段,唯一属性可能会在出现冲突后重命名。During each stage of sync, unique properties may get into conflict and renamed. 请注意从 Azure AD 到 Azure AD DS 的属性映射。Pay attention to the property mapping from Azure AD to Azure AD DS.

有关详细信息,请参阅 Azure AD UserPrincipalName 填充Azure AD DS 同步的工作方式For more information, see Azure AD UserPrincipalName population, and How Azure AD DS synchronization works.

密码哈希同步Password hash sync

  • 密码的同步与其他对象类型不同。Passwords are synced differently from other object types. 在 Azure AD 和 Azure AD DS 中,仅会同步不可逆的密码哈希Only non-reversible password hashes are synced in Azure AD and Azure AD DS
  • 必须通过 AD Connect 启用从本地到 Azure AD 的同步On-premise to Azure AD has to be enabled through AD Connect
  • 从 Azure AD 到 Azure AD DS 的同步是自动的(延迟在 20 分钟内)。Azure AD to Azure AD DS sync is automatic (latencies are under 20 minutes).
  • 只有在密码已更改的情况下,才会同步密码哈希。Password hashes are synced only when there's a changed password. 启用密码哈希同步时,所有现有的密码都不会自动同步,因为它们是以不可逆方式存储的。When you enable password hash sync, all existing passwords don't get synced automatically as they're stored irreversibly. 更改密码时,密码哈希会同步。When you change the password, password hashes get synced.

计算机对象位置Computer objects location

每个群集都与单个 OU 相关联。Each cluster is associated with a single OU. OU 中预配了一个内部用户。An internal user is provisioned in the OU. 所有节点都以域加入方式加入到同一个 OU 中。All the nodes are domain joined into the same OU.

Active Directory 管理工具Active Directory administrative tools

有关如何在 Windows Server VM 上安装 Active Directory 管理工具的步骤,请参阅安装管理工具For steps on how to install the Active Directory administrative tools on a Windows Server VM, see Install management tools.

故障排除Troubleshooting

群集创建反复失败Cluster creation fails repeatedly

最常见原因:Most common reasons:

  • DNS 配置不正确,群集节点的域加入操作失败。DNS configuration isn't correct, domain join of cluster nodes fail.
  • NSG 过于严格,阻止加入域。NSGs are too restrictive, preventing domain join.
  • 托管标识没有足够的权限。Managed Identity doesn't have sufficient permissions.
  • 群集名称的前六个字符不唯一(与其他实时群集或已删除群集的相同)。Cluster name isn't unique on the first six characters (either with another live cluster, or with a deleted cluster).

身份验证设置和配置Authentication setup and configuration

用户主体名称 (UPN)User Principal Name (UPN)

  • 请为所有服务使用小写,不过 UPN 在 ESP 群集中不区分大小写Please use lowercase for all services - UPNs are not case sensitive in ESP clusters, but
  • UPN 前缀应与 Azure AD DS 中的 SAMAccountName 匹配。The UPN prefix should match both SAMAccountName in Azure AD-DS. 不要求与邮件字段匹配。Matching with the mail field is not required.

Ambari 配置中的 LDAP 属性LDAP properties in Ambari configuration

有关影响 HDInsight 群集配置的 Ambari 属性的完整列表,请参阅 Ambari LDAP 身份验证设置For a full list of the Ambari properties that affect your HDInsight cluster configuration, see Ambari LDAP Authentication Setup.

后续步骤Next steps