Validate Azure Stack Hub system state
As an Azure Stack Hub operator, being able to determine the health and status of your system on demand is essential. The Azure Stack Hub validation tool (Test-AzureStack)) is a PowerShell cmdlet that lets you run a series of tests on your system to identify failures if present. You'll typically be asked to run this tool through the privileged end point (PEP) when you contact Azure Customer Services Support (Azure Support) with an issue. With the system-wide health and status information at hand, Azure Support can collect and analyze detailed logs, focus on the area where the error occurred, and work with you to fix the issue.
Running the validation tool and accessing results
You can use the PEP to run the validation tool. The tool can take a while to run. The length of the time depends on the number of virtual machines in your system. Each test returns a PASS/FAIL status in the PowerShell window.
Here's an outline of the end-to-end validation testing process:
Establish the trust. On an integrated system, run the following command from an elevated Windows PowerShell session to add the PEP as a trusted host on the hardened VM running on the hardware lifecycle host or the Privileged Access Workstation.
winrm s winrm/config/client '@{TrustedHosts="<IP Address of Privileged Endpoint>"}'If you're running the Azure Stack Development Kit (ASDK), sign in to the development kit host.
Access the PEP. Run the following commands to establish a PEP session:
Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcredTip
To access the PEP on an Azure Stack Development Kit (ASDK) host computer, use AzS-ERCS01 for -ComputerName.
Once you're in the PEP, run:
Test-AzureStackFor more information, see Parameter considerations and Use case examples.
If any tests report FAIL, run
Get-AzureStackLog. For instructions on an integrated system, see how to run Get-AzureStackLog on Azure Stack Hub integrated systems.The cmdlet gathers logs generated by Test-AzureStack. We recommend you don't collect logs and contact Azure Support instead if tests report WARN.
If you're instructed to run the validation tool by the Azure Support, the Azure Support representative will request the logs you collected to continue troubleshooting your issue.
Tests available
The validation tool lets you run a series of system-level tests and basic cloud scenarios that provide you with insight to the current state, allowing you to fix issues in your system.
Cloud infrastructure tests
These low impact tests work on an infrastructure level and provide you with information on various system components and functions. Currently, tests are grouped into the following categories:
| Test Category | Argument for -Include and -Ignore |
|---|---|
| Azure Stack Hub ACS Summary | AzsAcsSummary |
| Azure Stack Hub Action Plan Status | AzsActionPlanStatus |
| Azure Stack Hub Agent Health | AzsAgentHealth |
| Azure Stack AKS Arc Preview Summary | AzsAksArcPreviewSummary |
| Azure Stack Hub Active Directory Summary | AzsAdSummary |
| Azure Stack Hub Alert Summary | AzsAlertSummary |
| Azure Stack Hub Application Crash Summary | AzsApplicationCrashSummary |
| Azure Stack Hub Backup Share Accessibility Summary | AzsBackupShareAccessibility |
| Azure Stack Hub Control Plane Summary | AzsControlPlane |
| Azure Stack Hub DCB Validation | AzsDCBValidation |
| Azure Stack Hub Defender Summary | AzsDefenderSummary |
| Azure Stack Hub ECE Provisioning Status | AzsEceProvisioningStatus |
| Azure Stack Hub External Certificates Summary | AzsExternalCertificates |
| Azure Stack Hub GPU Configuration Summary | AzsGpuConfigurationSummary |
| Azure Stack Hub Hardware Monitor Enable | AzsGpuConfigurationSummary |
| Azure Stack Hub Hosting Infrastructure Firmware Summary | AzsHostingInfraFWSummary |
| Azure Stack Hub Hosting Infrastructure Summary | AzsHostingInfraSummary |
| Azure Stack Hub Hosting Infrastructure Utilization | AzsHostingInfraUtilization |
| Azure Stack Hub Identity Provider Connectivity | AzsIdentityProviderConnectivity |
| Azure Stack Hub Identity Provider Status | AzsIdentityProviderStatus |
| Azure Stack Hub Infrastructure Capacity | AzsInfraCapacity |
| Azure Stack Hub Infrastructure File Validation | AzsInfraFileValidation |
| Azure Stack Hub Infrastructure Performance | AzsInfraPerformance |
| Azure Stack Hub Infrastructure Role Summary | AzsInfraRoleSummary |
| Azure Stack Hub Infrastructure Update Memory | AzsInfraUpdateMemory |
| Azure Stack Hub Infrastructure Update Summary | AzsInfraUpdateSummary |
| Azure Stack Hub NCImos Validation | AzsNCImosValidation |
| Azure Stack Hub Network Infra | AzsNetworkInfra |
| Azure Stack Hub OEM Hardware | AzsOemHardware |
| Azure Stack Hub OEM Update Prerequisites | AzsOemUpdatePrereqs |
| Azure Stack Hub Portal and API Summary | AzsPortalAPISummary |
| Azure Stack Hub RDMA Operational Status | AzsRdmaOperationalStatus |
| Azure Stack Hub Resource Provider Authentication Status | AzsResourceProviderAuthStatus |
| Azure Stack Hub Scale Unit VM Events | AzsScaleUnitEvents |
| Azure Stack Hub Scale Unit VM Resources | AzsScaleUnitResources |
| Azure Stack Hub Scenarios | AzsScenarios |
| Azure Stack Hub Service Fabric Role Summary | AzsSFRoleSummary |
| Azure Stack Hub SQL Diagnostics | AzsSQLDiagnostics |
| Azure Stack Hub SQL Summary | AzsSQLSummary |
| Azure Stack Hub Stamp BMC Summary | AzsStampBMCSummary |
| Azure Stack Hub Storage Data Plane | AzsStorageDataPlane |
| Azure Stack Hub Storage Services Summary | AzsStorageSvcsSummary |
| Azure Stack Hub SQL Store Summary | AzsStoreSummary |
| Azure Stack Hub Usage Summary | AzsUsage |
| Azure Stack Hub Usage Bridge Summary | AzsUsageBridgeSummary |
| Azure Stack Hub VM Placement Summary | AzsVmPlacement |
Cloud scenario tests
In addition to the infrastructure tests in the previous section, you can also run cloud scenario tests to check functionality across infrastructure components. Cloud admin credentials are required to run these tests because they involve resource deployment.
Note
Currently you can't run cloud scenario tests using Active Directory Federated Services (AD FS) credentials.
The following cloud scenarios are tested by the validation tool:
- Resource group creation
- Plan creation
- Offer creation
- Storage account creation
- Virtual machine creation (VM)
- Blob storage operation
- Queue storage operation
- Table storage operation
Parameter considerations
The parameter List can be used to display all available test categories.
The parameters Include and Ignore can be used to include or exclude test categories. For more information about these arguments, see the following section.
Test-AzureStack -Include AzsSFRoleSummary, AzsInfraCapacityTest-AzureStack -Ignore AzsInfraPerformanceA tenant VM is deployed as part of the cloud scenario tests. You can use DoNotDeployTenantVm to disable this VM deployment.
You need to supply the ServiceAdminCredential parameter to run cloud scenario tests as described in the Use case examples section.
BackupSharePath and BackupShareCredential are used when testing infrastructure backup settings as shown in the Use case examples section.
DetailedResults can be used to get pass/fail/warning information for each test, as well as the overall run. When not specified, Test-AzureStack returns $true if there are no failures, and $false if there are failures.
TimeoutSeconds can be used to set a specific time for each group to complete.
The validation tool also supports common PowerShell parameters: Verbose, Debug, ErrorAction, ErrorVariable, WarningAction, WarningVariable, OutBuffer, PipelineVariable, and OutVariable. For more information, see About Common Parameters.
Use case examples
Run validation without cloud scenarios
Run the validation tool without the ServiceAdminCredential parameter to skip running cloud scenario tests:
New-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred -SessionOption (New-PSSessionOption -Culture en-US -UICulture en-US)
Test-AzureStack
Run validation with cloud scenarios
Supplying the validation tool with the ServiceAdminCredentials parameter runs the cloud scenario tests by default:
Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred
Test-AzureStack -ServiceAdminCredential "<Cloud administrator user name>"
If you wish to run ONLY cloud scenarios without running the rest of the tests, you can use the Include parameter to do so:
Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred
Test-AzureStack -ServiceAdminCredential "<Cloud administrator user name>" -Include AzsScenarios
The cloud admin user name must be typed in the UPN format: serviceadmin@contoso.partner.onmschina.cn (Microsoft Entra ID). When prompted, type the password to the cloud admin account.
Groups
To improve the operator experience, a Group parameter has been enabled to run multiple test categories at the same time. Currently, there are three groups defined: Default, UpdateReadiness, and SecretRotationReadiness.
Default: Considered to be a standard run of Test-AzureStack. This group is run by default if no other groups are selected.
UpdateReadiness: A check to see if the Azure Stack Hub instance can be updated. When the UpdateReadiness group is run, warnings are displayed as errors in the console output, and they should be considered as blockers for the update. The following categories are part of the UpdateReadiness group:
- AzsInfraFileValidation
- AzsActionPlanStatus
- AzsStampBMCSummary
SecretRotationReadiness: A check to see if the Azure Stack Hub instance is in a state in which secret rotation can be run. When the SecretRotationReadiness group is run, warnings are displayed as errors in the console output and they should be considered as blockers for secret rotation. The following categories are part of the SecretRotationReadiness Group:
- AzsAcsSummary
- AzsDefenderSummary
- AzsHostingInfraSummary
- AzsInfraCapacity
- AzsInfraRoleSummary
- AzsPortalAPISummary
- AzsSFRoleSummary
- AzsStorageSvcsSummary
- AzsStoreSummary
Group parameter example
The following example runs Test-AzureStack to test system readiness before installing an update or hotfix using Group. Before you start the installation of an update or hotfix, run Test-AzureStack to check the status of your Azure Stack Hub:
Test-AzureStack -Group UpdateReadiness
Run validation tool to test infrastructure backup settings
Before configuring infrastructure backup, you can test the backup share path and credential using the AzsBackupShareAccessibility test:
Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred
Test-AzureStack -Include AzsBackupShareAccessibility -BackupSharePath "\\<fileserver>\<fileshare>" -BackupShareCredential $using:backupcred
After configuring backup, you can run AzsBackupShareAccessibility to validate the share is accessible from the ERCS:
Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred
Test-AzureStack -Include AzsBackupShareAccessibility
To test new credentials with the configured backup share, run:
Enter-PSSession -ComputerName "<ERCS VM-name/IP address>" -ConfigurationName PrivilegedEndpoint -Credential $localcred
Test-AzureStack -Include AzsBackupShareAccessibility -BackupShareCredential "<PSCredential for backup share>"
Run validation tool to test network infrastructure
This test checks the connectivity of the network infrastructure bypassing the Azure Stack Hub software defined network (SDN). It demonstrates connectivity from a Public VIP to the configured DNS forwarders, NTP servers, and authentication endpoints. This includes connectivity to Azure when using Microsoft Entra ID as identity provider or the federated server when using AD FS as identity provider.
Include the debug parameter to get a detailed output of the command:
Test-AzureStack -Include AzsNetworkInfra -Debug
Next steps
To learn more about Azure Stack Hub diagnostics tools and issue logging, see Azure Stack Hub Diagnostic log collection.
To learn more about troubleshooting, see Azure Stack Hub troubleshooting.