Ambari user configs migration
After setting up HDInsight 5.x, it's necessary to update the user-defined configurations from the HDInsight 4.x cluster. Ambari doesn't currently provide a feature to export and import configurations. To overcome this limitation, we created a script that facilitates downloading the configurations and comparing them across clusters. However, this process involves a few manual steps, such as uploading the configurations to a storage directory, downloading them and then comparing them.
Script details
- This step contains two python scripts.
- Script to download the local cluster service configs from Ambari.
- Script to compare the service config files and generate the differences.
- All service configurations downloaded, but certain services and properties excluded from the comparison process. These excluded services and properties are as follows:
- Excluded properties
dfs.namenode.shared.edits.dir','hadoop.registry.zk.quorum','ha.zookeeper.quorum','hive.llap.zk.sm.connectionString','hive.cluster.delegation.token.store.zookeeper.connectString','hive.zookeeper.quorum','hive.metastore.uris','yarn.resourcemanager.hostname','hadoop.registry.zk.quorum','yarn.resourcemanager.hostname','yarn.node-labels.fs-store.root','javax.jdo.option.ConnectionURL','javax.jdo.option.ConnectionUserName','hive_database_name','hive_existing_mssql_server_database','yarn.log.server.url','yarn.timeline-service.sqldb-store.connection-username','yarn.timeline-service.sqldb-store.connection-url','fs.defaultFS', 'address'
- Excluded Services:
AMBARI_METRICS
andWEBHCAT
.
Workflow
To execute the migration process,
- Run the script on the HDInsight 4.x cluster to obtain the current service configurations from Ambari. The output saved on the local VM from where the script executed.
- Upload the output file to a public/common storage location, as it requires to download on the HDInsight 5.x cluster.
- Execute the script on the HDInsight 5.x cluster to retrieve the current service configurations from Ambari. Save the output on the local drive.
- Save the output.
- Download the HDInsight 4.x cluster configurations from the storage account to the HDInsight 5.x cluster.
- Run the script on the HDInsight 5.x cluster, where both the HDInsight 4.x and HDInsight 5.x configurations are present.
Execution
On HDInsight 4 Cluster (Old Cluster)
ssh to headnode and run the following commands.
mkdir hdinsights_ambari_utils cd hdinsights_ambari_utils
Run
wget https://hdiconfigactions2.blob.core.chinacloudapi.cn/hdi-sre-workspace/hdinsights_upgrade_ambari_utils/ambari_export_cluster_configs.py
to download the Python script.Run
python ambari_export_cluster_configs.py
. Make sure that the username and password supplied within single quotes.Do
ls -ltr
to check the configs files.You can see an output file with cluster name as
Plutos.out
.Upload the file to a storage container.
On HDInsight 5.x Cluster (New Cluster)
ssh to headnode.
mkdir hdinsights_ambari_utils cd hdinsights_ambari_util
Run
wget https://hdiconfigactions2.blob.core.chinacloudapi.cn/hdi-sre-workspace/hdinsights_upgrade_ambari_utils/ambari_export_cluster_configs.py
to download the Python script.Execute the script
python ambari_export_cluster_configs.py
. Make sure that the username and password is supplied within single quotesCheck for the configs files.
You can see an output file with cluster name
Sugar.out
.Download the old cluster
Sugar.out
file.Run
wget https://hdiconfigactions2.blob.core.chinacloudapi.cn/hdi-sre-workspace/hdinsights_upgrade_ambari_utils/compare_ambari_cluster_configs.py
to download the Python script.Run
compare_ambari_cluster_configs.py
script.Run
sshuser@hn0-sugar:~/hdinsights_ambari_utils$ python, compare_ambari_cluster_configs.py plutos out sugar.out
Run the command 'ls -ltr'.
Additionally, both the clusters configs are stored here, which you can use them for future reference.