Ambari user configs migration

  • Article

After setting up HDInsight 5.x, it's necessary to update the user-defined configurations from the HDInsight 4.x cluster. Ambari doesn't currently provide a feature to export and import configurations. To overcome this limitation, we created a script that facilitates downloading the configurations and comparing them across clusters. However, this process involves a few manual steps, such as uploading the configurations to a storage directory, downloading them and then comparing them.

Script details

  • This step contains two python scripts.
    • Script to download the local cluster service configs from Ambari.
    • Script to compare the service config files and generate the differences.
  • All service configurations downloaded, but certain services and properties excluded from the comparison process. These excluded services and properties are as follows:
    • Excluded properties
    dfs.namenode.shared.edits.dir','hadoop.registry.zk.quorum','ha.zookeeper.quorum','hive.llap.zk.sm.connectionString','hive.cluster.delegation.token.store.zookeeper.connectString','hive.zookeeper.quorum','hive.metastore.uris','yarn.resourcemanager.hostname','hadoop.registry.zk.quorum','yarn.resourcemanager.hostname','yarn.node-labels.fs-store.root','javax.jdo.option.ConnectionURL','javax.jdo.option.ConnectionUserName','hive_database_name','hive_existing_mssql_server_database','yarn.log.server.url','yarn.timeline-service.sqldb-store.connection-username','yarn.timeline-service.sqldb-store.connection-url','fs.defaultFS', 'address'
    • Excluded Services:
      AMBARI_METRICS and WEBHCAT.

Workflow

To execute the migration process,

  1. Run the script on the HDInsight 4.x cluster to obtain the current service configurations from Ambari. The output saved on the local VM from where the script executed.
  2. Upload the output file to a public/common storage location, as it requires to download on the HDInsight 5.x cluster.
  3. Execute the script on the HDInsight 5.x cluster to retrieve the current service configurations from Ambari. Save the output on the local drive.
  4. Save the output.
  5. Download the HDInsight 4.x cluster configurations from the storage account to the HDInsight 5.x cluster.
  6. Run the script on the HDInsight 5.x cluster, where both the HDInsight 4.x and HDInsight 5.x configurations are present.

Execution

On HDInsight 4 Cluster (Old Cluster)

  1. ssh to headnode and run the following commands.

    mkdir hdinsights_ambari_utils
cd hdinsights_ambari_utils

  2. Run wget https://hdiconfigactions2.blob.core.chinacloudapi.cn/hdi-sre-workspace/hdinsights_upgrade_ambari_utils/ambari_export_cluster_configs.py to download the Python script.

    Screenshot showing wget command.

  3. Run python ambari_export_cluster_configs.py. Make sure that the username and password supplied within single quotes.

    Screenshot showing run-python-script.

  4. Do ls -ltr to check the configs files.

    Screenshot showing script output.

  5. You can see an output file with cluster name as Plutos.out.

  6. Upload the file to a storage container.

On HDInsight 5.x Cluster (New Cluster)

  1. ssh to headnode.

    mkdir hdinsights_ambari_utils
cd hdinsights_ambari_util

    Screenshot showing wget output.

  2. Run wget https://hdiconfigactions2.blob.core.chinacloudapi.cn/hdi-sre-workspace/hdinsights_upgrade_ambari_utils/ambari_export_cluster_configs.py to download the Python script.

    Screenshot showing python script output.

  3. Execute the script python ambari_export_cluster_configs.py. Make sure that the username and password is supplied within single quotes

  4. Check for the configs files.

    Screenshot showing Ambari python script.

  5. You can see an output file with cluster name Sugar.out.

  6. Download the old cluster Sugar.out file.

    Screenshot showing wget command output.

    Screenshot showing python results.

  7. Run wget https://hdiconfigactions2.blob.core.chinacloudapi.cn/hdi-sre-workspace/hdinsights_upgrade_ambari_utils/compare_ambari_cluster_configs.py to download the Python script.

  8. Run compare_ambari_cluster_configs.py script.

  9. Run

    sshuser@hn0-sugar:~/hdinsights_ambari_utils$ python,
compare_ambari_cluster_configs.py plutos out sugar.out

    Screenshot showing python compare command.

  10. You can see the difference in the output. Screenshot showing python code sample.

  11. Run the command 'ls -ltr'.

  12. Additionally, both the clusters configs are stored here, which you can use them for future reference. Screenshot showing list of output files.