Configure Istio-based service mesh add-on for Azure Kubernetes Service

Open-source Istio uses MeshConfig to define mesh-wide settings for the Istio service mesh. Istio-based service mesh add-on for AKS builds on top of MeshConfig and classifies different properties as supported, allowed, and blocked.

This article walks through how to configure Istio-based service mesh add-on for Azure Kubernetes Service and the support policy applicable for such configuration.

Prerequisites

This guide assumes you followed the documentation to enable the Istio add-on on an AKS cluster.

Set up configuration on cluster

  1. Find out which revision of Istio is deployed on the cluster:

    az aks show --name $CLUSTER --resource-group $RESOURCE_GROUP --query 'serviceMeshProfile'
    

    Output:

    {
      "istio": {
          "certificateAuthority": null,
          "components": {
          "egressGateways": null,
          "ingressGateways": null
          },
          "revisions": [
          "asm-1-18"
          ]
      },
      "mode": "Istio"
    }
    
  2. Create a ConfigMap with the name istio-shared-configmap-<asm-revision> in the aks-istio-system namespace. For example, if your cluster is running asm-1-18 revision of mesh, then the ConfigMap needs to be named as istio-shared-configmap-asm-1-18. Mesh configuration has to be provided within the data section under mesh.

    Example:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: istio-shared-configmap-asm-1-18
      namespace: aks-istio-system
    data:
      mesh: |-
        accessLogFile: /dev/stdout
        defaultConfig:
          holdApplicationUntilProxyStarts: true
    

    The values under defaultConfig are mesh-wide settings applied for Envoy sidecar proxy.

Caution

A default ConfigMap (for example, istio-asm-1-18 for revision asm-1-18) is created in aks-istio-system namespace on the cluster when the Istio add-on is enabled. However, this default ConfigMap gets reconciled by the managed Istio add-on and thus users should NOT directly edit this ConfigMap. Instead users should create a revision specific Istio shared ConfigMap (for example istio-shared-configmap-asm-1-18 for revision asm-1-18) in the aks-istio-system namespace, and then the Istio control plane will merge this with the default ConfigMap, with the default settings taking precedence.

Mesh configuration and upgrades

When you're performing canary upgrade for Istio, you need to create a separate ConfigMap for the new revision in the aks-istio-system namespace before initiating the canary upgrade. This way the configuration is available when the new revision's control plane is deployed on cluster. For example, if you're upgrading the mesh from asm-1-18 to asm-1-19, you need to copy changes over from istio-shared-configmap-asm-1-18 to create a new ConfigMap called istio-shared-configmap-asm-1-19 in the aks-istio-system namespace.

After the upgrade is completed or rolled back, you can delete the ConfigMap of the revision that was removed from the cluster.

Allowed, supported, and blocked MeshConfig values

Fields in MeshConfig are classified as allowed, supported, or blocked. To learn more about these categories, see the support policy for Istio add-on features and configuration options.

Mesh configuration and the list of allowed/supported fields are revision specific to account for fields being added/removed across revisions. The full list of allowed fields and the supported/unsupported ones within the allowed list is provided in the below table. When new mesh revision is made available, any changes to allowed and supported classification of the fields is noted in this table.

MeshConfig

Fields present in open source MeshConfig reference documentation that are not covered in the following table are blocked. For example, configSources is blocked.

Field Supported/Allowed Notes
proxyListenPort Allowed -
proxyInboundListenPort Allowed -
proxyHttpPort Allowed -
connectTimeout Allowed Configurable in DestinationRule
tcpKeepAlive Allowed Configurable in DestinationRule
defaultConfig Supported Used to configure ProxyConfig
outboundTrafficPolicy Supported Also configurable in Sidecar CR
extensionProviders Allowed -
defaultProviders Allowed -
accessLogFile Supported This field addresses the generation of access logs. For a managed experience on collection and querying of logs, refer to Azure Monitor Container Insights on AKS. It is encouraged to configure access logging via the Telemetry API.
accessLogFormat Supported This field addresses the generation of access logs. For a managed experience on collection and querying of logs, refer to Azure Monitor Container Insights on AKS
accessLogEncoding Supported This field addresses the generation of access logs. For a managed experience on collection and querying of logs, refer to Azure Monitor Container Insights on AKS
enableTracing Allowed It is encouraged to configure tracing via the Telemetry API.
enableEnvoyAccessLogService Supported This field addresses the generation of access logs. For a managed experience on collection and querying of logs, refer to Azure Monitor Container Insights on AKS
disableEnvoyListenerLog Supported This field addresses the generation of access logs. For a managed experience on collection and querying of logs, refer to Azure Monitor Container Insights on AKS
trustDomain Allowed -
trustDomainAliases Allowed -
caCertificates Allowed Configurable in DestinationRule
defaultServiceExportTo Allowed Configurable in ServiceEntry
defaultVirtualServiceExportTo Allowed Configurable in VirtualService
defaultDestinationRuleExportTo Allowed Configurable in DestinationRule
localityLbSetting Allowed Configurable in DestinationRule
dnsRefreshRate Allowed -
h2UpgradePolicy Allowed Configurable in DestinationRule
enablePrometheusMerge Allowed -
discoverySelectors Supported -
pathNormalization Allowed -
defaultHttpRetryPolicy Allowed Configurable in VirtualService
serviceSettings Allowed -
meshMTLS Allowed -
tlsDefaults Allowed -
ingressService Allowed Name of the Kubernetes service used for the istio ingress controller.
ingressSelector Allowed Defines which gateway deployment to use as the Ingress controller. This field corresponds to the Gateway.selector field, and will be set as istio: INGRESS_SELECTOR.

ProxyConfig (meshConfig.defaultConfig)

Fields present in open source MeshConfig reference documentation that are not covered in the following table are blocked.

Field Supported/Allowed Notes
tracingServiceName Allowed It is encouraged to configure tracing via the Telemetry API.
drainDuration Supported -
statsUdpAddress Allowed -
proxyAdminPort Allowed -
tracing Allowed It is encouraged to configure tracing via the Telemetry API.
concurrency Supported -
envoyAccessLogService Allowed It is encouraged to configure tracing via the Telemetry API.
envoyMetricsService Allowed It is encouraged to configure metrics collection via the Telemetry API.
proxyMetadata Allowed -
statusPort Allowed -
extraStatTags Allowed -
proxyStatsMatcher Allowed -
terminationDrainDuration Supported -
meshId Allowed -
holdApplicationUntilProxyStarts Supported -
caCertificatesPem Allowed -
privateKeyProvider Allowed -

Caution

Support scope of configurations: Mesh configuration allows for extension providers such as self-managed instances of Zipkin or Apache Skywalking to be configured with the Istio add-on. However, these extension providers are outside the support scope of the Istio add-on. Any issues associated with extension tools are outside the support boundary of the Istio add-on.

Common errors and troubleshooting tips

  • Ensure that the MeshConfig is indented with spaces instead of tabs.
  • Ensure that you're only editing the revision specific shared ConfigMap (for example istio-shared-configmap-asm-1-18) and not trying to edit the default ConfigMap (for example istio-asm-1-18).
  • The ConfigMap must follow the name istio-shared-configmap-<asm-revision> and be in the aks-istio-system namespace.
  • Ensure that all MeshConfig fields are spelled correctly. If they're unrecognized or if they aren't part of the allowed list, admission control denies such configurations.
  • When performing canary upgrades, check your revision specific ConfigMaps to ensure configurations exist for the revisions deployed on your cluster.
  • Certain MeshConfig options such as accessLogging may increase Envoy's resource consumption, and disabling some of these settings may mitigate Istio data plane resource utilization. It's also advisable to use the discoverySelectors field in the MeshConfig to help alleviate memory consumption for Istiod and Envoy.
  • If the concurrency field in the MeshConfig is misconfigured and set to zero, it causes Envoy to use up all CPU cores. Instead if this field is unset, number of worker threads to run is automatically determined based on CPU requests/limits.
  • Pod and sidecar race conditions in which the application starts before Envoy can be mitigated using the holdApplicationUntilProxyStarts field in the MeshConfig.