Azure 机器学习的 Git 集成Git integration for Azure Machine Learning

Git 是一种常用的版本控制系统,可用于共享和协作处理项目。Git is a popular version control system that allows you to share and collaborate on your projects.

Azure 机器学习完全支持用于跟踪工作的 Git 存储库 - 你可以将存储库直接克隆到共享工作区文件系统上,在本地工作站上使用 Git,或者从 CI/CD 管道使用 Git。Azure Machine Learning fully supports Git repositories for tracking work - you can clone repositories directly onto your shared workspace file system, use Git on your local workstation, or use Git from a CI/CD pipeline.

在向 Azure 机器学习提交作业时,如果源文件存储在本地 git 存储库中,那么系统会将有关存储库的信息作为训练过程的一部分进行跟踪。When submitting a job to Azure Machine Learning, if source files are stored in a local git repository then information about the repo is tracked as part of the training process.

由于 Azure 机器学习会跟踪来自本地 git 存储库的信息,因此它不会绑定到任何特定的中心存储库。Since Azure Machine Learning tracks information from a local git repo, it isn't tied to any specific central repository. 可以从 GitHub、GitLab、Bitbucket、Azure DevOps 或任何其他与 git 兼容的服务克隆存储库。Your repository can be cloned from GitHub, GitLab, Bitbucket, Azure DevOps, or any other git-compatible service.

将 Git 存储库克隆到你的工作区文件系统Clone Git repositories into your workspace file system

Azure 机器学习为工作区中的所有用户提供了一个共享文件系统。Azure Machine Learning provides a shared file system for all users in the workspace. 若要将 Git 存储库克隆到此文件共享,我们建议你创建一个计算实例并打开终端。To clone a Git repository into this file share, we recommend that you create a Compute Instance & open a terminal. 打开终端后,你可以访问完整的 Git 客户端,并可以通过 Git CLI 体验来克隆和使用 Git。Once the terminal is opened, you have access to a full Git client and can clone and work with Git via the Git CLI experience.

我们建议你将存储库克隆到你的用户目录中,避免与其他人直接在你的工作分支上发生冲突。We recommend that you clone the repository into your users directory so that others will not make collisions directly on your working branch.

你可以克隆你能够向其证明身份的任何 Git 存储库(GitHub、Azure Repos、BitBucket 等)You can clone any Git repository you can authenticate to (GitHub, Azure Repos, BitBucket, etc.)

有关克隆的详细信息,请参阅如何使用 Git CLI 页面上的指南。For more information about cloning, see the guide on how to use Git CLI.

通过 SSH 对 Git 帐户进行身份验证Authenticate your Git Account with SSH

生成新的 SSH 密钥Generate a new SSH key

  1. 在 Azure 机器学习笔记本选项卡中打开终端窗口Open the terminal window in the Azure Machine Learning Notebook Tab.

  2. 粘贴以下文本,并将其替换为电子邮件地址。Paste the text below, substituting in your email address.

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

这会创建一个新的 SSH 密钥,并使用提供的电子邮件作为标签。This creates a new ssh key, using the provided email as a label.

> Generating public/private rsa key pair.
  1. 当系统提示“输入用于保存密钥的文件”时,请按 Enter 键。When you're prompted to "Enter a file in which to save the key" press Enter. 这会接受默认文件位置。This accepts the default file location.

  2. 验证该默认位置为“/home/azureuser/.ssh”,然后按 Enter。Verify that the default location is '/home/azureuser/.ssh' and press enter. 否则,请指定位置“/home/azureuser/.ssh”。Otherwise specify the location '/home/azureuser/.ssh'.

提示

请确保 SSH 密钥保存在“/home/azureuser/.ssh”中。Make sure the SSH key is saved in '/home/azureuser/.ssh'. 此文件保存在计算实例上,只有计算实例的所有者才能访问This file is saved on the compute instance is only accessible by the owner of the Compute Instance

> Enter a file in which to save the key (/home/azureuser/.ssh/id_rsa): [Press enter]
  1. 在提示符下,键入安全密码。At the prompt, type a secure passphrase. 建议向 SSH 密钥添加密码,以提高安全性We recommend you add a passphrase to your SSH key for added security
> Enter passphrase (empty for no passphrase): [Type a passphrase]
> Enter same passphrase again: [Type passphrase again]

将公钥添加到 Git 帐户Add the public key to Git Account

  1. 在终端窗口中,复制公钥文件的内容。In your terminal window, copy the contents of your public key file. 如果重命名了该密钥,请将 id_rsa.pub 替换为公钥文件名。If you renamed the key, replace id_rsa.pub with the public key file name.
cat ~/.ssh/id_rsa.pub

提示

在终端中复制和粘贴Copy and Paste in Terminal

  • Windows:使用 Ctrl-Insert 复制,使用 Ctrl-Shift-vShift-Insert 粘贴。Windows: Ctrl-Insert to copy and use Ctrl-Shift-v or Shift-Insert to paste.
  • Mac OS:使用 Cmd-c 复制,使用 Cmd-v 粘贴。Mac OS: Cmd-c to copy and Cmd-v to paste.
  • FireFox/IE 可能不会正确支持剪贴板权限。FireFox/IE may not support clipboard permissions properly.
  1. 选择并复制剪贴板中的密钥输出。Select and copy the key output in the clipboard.

通过 SSH 克隆 Git 存储库Clone the Git repository with SSH

  1. 从 Git 存储库中复制 SSH Git 克隆 URL。Copy the SSH Git clone URL from the Git repo.

  2. 将 URL 粘贴到下面的 git clone 命令中,以使用 SSH Git 存储库 URL。Paste the url into the git clone command below, to use your SSH Git repo URL. 该 URL 应类似于:This will look something like:

git clone git@example.com:GitUser/azureml-example.git
Cloning into 'azureml-example'...

将看到如下所示的响应:You will see a response like:

The authenticity of host 'example.com (192.30.255.112)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.255.112' (RSA) to the list of known hosts.

SSH 可能会显示服务器的 SSH 指纹,并要求你对其进行验证。SSH may display the server's SSH fingerprint and ask you to verify it. 应验证显示的指纹是否与 SSH 公钥页中的某个指纹相匹配。You should verify that the displayed fingerprint matches one of the fingerprints in the SSH public keys page.

SSH 将在连接到未知主机时显示此指纹,以防止中间人攻击SSH displays this fingerprint when it connects to an unknown host to protect you from man-in-the-middle attacks. 接受主机的指纹后,SSH 将不再提示你,除非指纹发生更改。Once you accept the host's fingerprint, SSH will not prompt you again unless the fingerprint changes.

  1. 当系统询问是否要继续连接时,请键入 yesWhen you are asked if you want to continue connecting, type yes. Git 将克隆存储库,并设置原点远程,以便在将来的 Git 命令中使用 SSH 进行连接。Git will clone the repo and set up the origin remote to connect with SSH for future Git commands.

跟踪来自 Git 存储库的代码Track code that comes from Git repositories

从 Python SDK 或机器学习 CLI 提交训练运行时,训练模型所需的文件将上传到工作区。When you submit a training run from the Python SDK or Machine Learning CLI, the files needed to train the model are uploaded to your workspace. 如果可在开发环境中使用 git 命令,则上传过程会使用该命令检查文件是否存储在 git 存储库中。If the git command is available on your development environment, the upload process uses it to check if the files are stored in a git repository. 如果是,那么 git 存储库中的信息也会作为训练运行的一部分上传。If so, then information from your git repository is also uploaded as part of the training run. 此信息存储在训练运行的以下属性中:This information is stored in the following properties for the training run:

属性Property 用于获取值的 Git 命令Git command used to get the value 说明Description
azureml.git.repository_uri git ls-remote --get-url 从中克隆存储库的 URI。The URI that your repository was cloned from.
mlflow.source.git.repoURL git ls-remote --get-url 从中克隆存储库的 URI。The URI that your repository was cloned from.
azureml.git.branch git symbolic-ref --short HEAD 提交运行时的活动分支。The active branch when the run was submitted.
mlflow.source.git.branch git symbolic-ref --short HEAD 提交运行时的活动分支。The active branch when the run was submitted.
azureml.git.commit git rev-parse HEAD 为运行提交的代码的提交哈希。The commit hash of the code that was submitted for the run.
mlflow.source.git.commit git rev-parse HEAD 为运行提交的代码的提交哈希。The commit hash of the code that was submitted for the run.
azureml.git.dirty git status --porcelain . 如果分支/提交异常,则为 True;否则为 falseTrue, if the branch/commit is dirty; otherwise, false.

对于使用估计器、机器学习管道或脚本运行的运行,将发送此信息。This information is sent for runs that use an estimator, machine learning pipeline, or script run.

如果训练文件不在开发环境的 git 存储库中,或者 git 命令不可用,则不会跟踪与 git 相关的信息。If your training files are not located in a git repository on your development environment, or the git command is not available, then no git-related information is tracked.

提示

若要检查 git 命令在开发环境中是否可用,请打开 shell 会话、命令提示符、PowerShell 或其他命令行接口,并键入以下命令:To check if the git command is available on your development environment, open a shell session, command prompt, PowerShell or other command line interface and type the following command:

git --version

如果已安装且在路径中,则会收到类似于 git version 2.4.1 的响应。If installed, and in the path, you receive a response similar to git version 2.4.1. 有关在开发环境中安装 git 的详细信息,请参阅 Git 网站For more information on installing git on your development environment, see the Git website.

查看记录的信息View the logged information

Git 信息存储在训练运行的属性中。The git information is stored in the properties for a training run. 可以使用 Azure 门户、Python SDK 和 CLI 查看此信息。You can view this information using the Azure portal, Python SDK, and CLI.

Azure 门户Azure portal

  1. Azure 门户中,选择工作区。From the Azure portal, select your workspace.
  2. 选择“试验”,然后选择一个试验。Select Experiments, and then select one of your experiments.
  3. 从“运行号”列中选择一个运行。Select one of the runs from the RUN NUMBER column.
  4. 选择“输出 + 日志”,然后展开“日志”和“azureml”条目 。Select Outputs + logs, and then expand the logs and azureml entries. 选择以“###_azure”开头的链接。Select the link that begins with ###_azure.

记录的信息包含类似于以下 JSON 的文本:The logged information contains text similar to the following JSON:

"properties": {
    "_azureml.ComputeTargetType": "batchai",
    "ContentSnapshotId": "5ca66406-cbac-4d7d-bc95-f5a51dd3e57e",
    "azureml.git.repository_uri": "git@github.com:azure/machinelearningnotebooks",
    "mlflow.source.git.repoURL": "git@github.com:azure/machinelearningnotebooks",
    "azureml.git.branch": "master",
    "mlflow.source.git.branch": "master",
    "azureml.git.commit": "4d2b93784676893f8e346d5f0b9fb894a9cf0742",
    "mlflow.source.git.commit": "4d2b93784676893f8e346d5f0b9fb894a9cf0742",
    "azureml.git.dirty": "True",
    "AzureML.DerivedImageName": "azureml/azureml_9d3568242c6bfef9631879915768deaf",
    "ProcessInfoFile": "azureml-logs/process_info.json",
    "ProcessStatusFile": "azureml-logs/process_status.json"
}

Python SDKPython SDK

提交训练运行后,将返回 Run 对象。After submitting a training run, a Run object is returned. 此对象的 properties 属性包含记录的 git 信息。The properties attribute of this object contains the logged git information. 例如,以下代码可检索提交哈希:For example, the following code retrieves the commit hash:

run.properties['azureml.git.commit']

CLICLI

az ml run CLI命令可用于从运行中检索属性。The az ml run CLI command can be used to retrieve the properties from a run. 例如,以下命令返回名为 train-on-amlcompute 的试验中最后一次运行的属性:For example, the following command returns the properties for the last run in the experiment named train-on-amlcompute:

az ml run list -e train-on-amlcompute --last 1 -w myworkspace -g myresourcegroup --query '[].properties'

有关详细信息,请参阅 az ml run 参考文档。For more information, see the az ml run reference documentation.

后续步骤Next steps