自定义语音识别的 CI/CDCI/CD for Custom Speech

实施自动定型、测试和发布管理,以便在你应用对定型数据和测试数据的更新时,自定义语音识别模型可持续改进。Implement automated training, testing, and release management to enable continuous improvement of Custom Speech models as you apply updates to training and testing data. 通过 CI/CD 工作流的有效实施,可确保性能最佳的自定义语音识别模型的终结点始终可用。Through effective implementation of CI/CD workflows, you can ensure that the endpoint for the best-performing Custom Speech model is always available.

持续集成 (CI) 是经常在共享存储库中提交更新并对其执行自动生成的工程实践。Continuous integration (CI) is the engineering practice of frequently committing updates in a shared repository, and performing an automated build on it. 自定义语音识别的 CI 工作流可基于其数据源定型新模型,并对新模型执行自动测试,确保其性能比之前的模型更好。CI workflows for Custom Speech train a new model from its data sources and perform automated testing on the new model to ensure that it performs better than the previous model.

持续交付 (CD) 从 CI 过程获取模型,并为每个改进的自定义语音识别模型创建一个终结点。Continuous delivery (CD) takes models from the CI process and creates an endpoint for each improved Custom Speech model. CD 使终结点可轻松地集成到解决方案中。CD makes endpoints easily available to be integrated into solutions.

可使用自定义 CI/CD 解决方案,但对于可靠的预生成解决方案,请使用语音 DevOps 模板存储库,它将使用 GitHub Actions 执行 CI/CD 工作流。Custom CI/CD solutions are possible, but for a robust, pre-built solution, use the Speech DevOps template repository, which executes CI/CD workflows using GitHub Actions.

自定义语音识别的 CI/CD 工作流CI/CD workflows for Custom Speech

这些工作流的目的是确保每个自定义语音识别模型具有比上一个版本更好的识别准确性。The purpose of these workflows is to ensure that each Custom Speech model has better recognition accuracy than the previous build. 如果对测试数据和/或定型数据的更新提高了准确性,则这些工作流将创建新的自定义语音识别终结点。If the updates to the testing and/or training data improve the accuracy, these workflows create a new Custom Speech endpoint.

GitHub 和 Azure DevOps 等 Git 服务器可在发生特定 Git 事件(例如合并或拉取请求)时运行自动工作流。Git servers such as GitHub and Azure DevOps can run automated workflows when specific Git events happen, such as merges or pull requests. 例如,测试数据的更新被推送到主分支时,可触发 CI 工作流。For example, a CI workflow can be triggered when updates to testing data are pushed to the master branch. 不同的 Git 服务器具有不同的工具,但允许使用脚本命令行接口 (CLI) 命令,使命令可在生成服务器上执行。Different Git Servers will have different tooling, but will allow scripting command-line interface (CLI) commands so that they can execute on a build server.

在此过程中,工作流应存储数据、测试、测试文件、模型和终结点并对它们命名,以便可以追溯到它们的源提交或源版本。Along the way, the workflows should name and store data, tests, test files, models, and endpoints such that they can be traced back to the commit or version they came from. 对这些资产命名也很有帮助,这样便可轻松地查看哪些资产是在更新测试数据和定型数据之后创建的。It is also helpful to name these assets so that it is easy to see which were created after updating testing data versus training data.

测试数据更新的 CI 工作流CI workflow for testing data updates

CI/CD 工作流的主要目的是使用定型数据生成一个新模型,并使用测试数据测试该模型,确定与之前性能最佳的模型(“基准模型”)相比,字词错误率 (WER) 是否有所改进。The principal purpose of the CI/CD workflows is to build a new model using the training data, and to test that model using the testing data to establish whether the Word Error Rate (WER) has improved compared to the previous best-performing model (the "benchmark model"). 如果新模型的性能更佳,它将成为新的基准模型,将来的模型将与它进行比较。If the new model performs better, it becomes the new benchmark model against which future models are compared.

测试数据更新的 CI 工作流应使用更新的测试数据重新测试当前的基准模型,计算修改后的 WER。The CI workflow for testing data updates should retest the current benchmark model with the updated test data to calculate the revised WER. 这可确保在将新模型的 WER 与基准模型的 WER 进行比较时,这两种模型均已针对相同的测试数据进行了测试,并且你正在进行同类比较。This ensures that when the WER of a new model is compared to the WER of the benchmark, both models have been tested against the same test data and you're comparing like with like.

此工作流应触发对测试数据的更新,并执行以下操作:This workflow should trigger on updates to testing data and:

  • 针对更新的测试数据测试基准模型。Test the benchmark model against the updated testing data.
  • 使用更新的数据来存储测试输出,输出中包含基准模型的 WER。Store the test output, which contains the WER of the benchmark model, using the updated data.
  • 从这些测试得出的 WER 将成为未来模型必须超越的新基准 WER。The WER from these tests will become the new benchmark WER that future models must beat.
  • 对于测试数据的更新,不执行 CD 工作流。The CD workflow does not execute for updates to testing data.

定型数据更新的 CI 工作流CI workflow for training data updates

对定型数据的更新意味着对自定义模型的更新。Updates to training data signify updates to the custom model.

此工作流应触发对定型数据的更新,并执行以下操作:This workflow should trigger on updates to training data and:

  • 使用更新的定型数据定型新模型。Train a new model with the updated training data.
  • 针对测试数据测试新模型。Test the new model against the testing data.
  • 存储测试输出,其中包含 WER。Store the test output, which contains the WER.
  • 将新模型的 WER 与基准模型的 WER 进行比较。Compare the WER from the new model to the WER from the benchmark model.
  • 如果 WER 未改进,则停止工作流。If the WER does not improve, stop the workflow.
  • 如果 WER 得到改进,则执行 CD 工作流以创建自定义语音识别终结点。If the WER improves, execute the CD workflow to create a Custom Speech endpoint.

CD 工作流CD workflow

在定型数据的更新提高了模型的识别能力后,CD 工作流应自动执行,为该模型创建一个新的终结点,并使该终结点可用,这样它便可在解决方案中使用。After an update to the training data improves a model's recognition, the CD workflow should automatically execute to create a new endpoint for that model, and make that endpoint available such that it can be used in a solution.

发布管理Release management

大多数团队需要一个手动审查和批准过程才能实现对生产环境的部署。Most teams require a manual review and approval process for deployment to a production environment. 对于生产部署,你可能希望确保它在开发团队中的关键人员可提供支持时或在低流量时期执行。For a production deployment, you might want to make sure it happens when key people on the development team are available for support, or during low-traffic periods.

自定义语音识别工作流工具Tools for Custom Speech workflows

对自定义语音识别的 CI/CD 自动化工作流使用以下工具:Use the following tools for CI/CD automation workflows for Custom Speech:

  • Azure CLI 用于创建 Azure 服务主体身份验证、查询 Azure 订阅并将测试结果存储在 Azure Blob 中。Azure CLI to create an Azure service principal authentication, query Azure subscriptions, and store test results in Azure Blob.
  • Azure 语音 CLI 用于从命令行或自动工作流与语音服务交互。Azure Speech CLI to interact with the Speech Service from the command line or an automated workflow.

使用 GitHub Actions 的 DevOps 自定义语音识别解决方案DevOps solution for Custom Speech using GitHub Actions

有关已实现的 DevOps 自定义语音识别解决方案,请转到语音 DevOps 模板存储库For an already-implemented DevOps solution for Custom Speech, go to the Speech DevOps template repo. 创建模板的副本,并通过可靠的 DevOps 系统开始开发自定义模型,包括使用 GitHub Actions 进行测试、定型和版本控制。Create a copy of the template and begin development of custom models with a robust DevOps system that includes testing, training, and versioning using GitHub Actions. 存储库提供了示例测试和定型数据,帮助设置和解释工作流。The repository provides sample testing and training data to aid in setup and explain the workflow. 初始设置后,将示例数据替换为你的项目数据。After initial setup, replace the sample data with your project data.

语音 DevOps 模板存储库为执行以下操作提供了基础结构和详细指导:The Speech DevOps template repo provides the infrastructure and detailed guidance to:

  • 将模板存储库复制到 GitHub 帐户,然后为 GitHub Actions CI/CD 工作流创建 Azure 资源和服务主体Copy the template repository to your GitHub account, then create Azure resources and a service principal for the GitHub Actions CI/CD workflows.
  • 演练开发内部循环Walk through the "dev inner loop." 从功能分支更新定型数据和测试数据,使用临时开发模型测试更改,并提出拉取请求以提出更改意见并对此进行审查。Update training and testing data from a feature branch, test the changes with a temporary development model, and raise a pull request to propose and review the changes.
  • 在对主分支的拉取请求中更新定型数据后,使用 GitHub Actions CI 工作流定型模型。When training data is updated in a pull request to master, train models with the GitHub Actions CI workflow.
  • 执行自动准确性测试,确定模型的 字词错误率 (WER)。Perform automated accuracy testing to establish a model's Word Error Rate (WER). 将测试结果存储在 Azure Blob 中。Store the test results in Azure Blob.
  • 若 WER 得到改进,则执行 CD 工作流以创建一个终结点。Execute the CD workflow to create an endpoint when the WER improves.

后续步骤Next steps

详细了解具有语音识别的 DevOps:Learn more about DevOps with Speech: