Scenario: hbase hbck command returns inconsistencies in Azure HDInsight

This article describes troubleshooting steps and possible resolutions for issues when interacting with Azure HDInsight clusters. If you're using hbase-2.x, see How to use Apache HBase HBCK2 tool

Region xxx on HDFS, but not listed in hbase:meta or deployed on any region server.

Varies.

1. Fix the meta table by running:

   hbase hbck -ignorePreCheckPermission -fixMeta
   

2. Assign regions to RegionServers by running:

   hbase hbck -ignorePreCheckPermission -fixAssignment
   

Region xxx not deployed on any RegionServer. It means the region is in hbase:meta, but offline.

Varies.

Bring regions online by running:

hbase hbck -ignorePreCheckPermission -fixAssignment

Alternatively, run assign <region-hash> on hbase-shell to force assign this region

Varies.

Manually merge those overlapped regions. Go to HBase HMaster Web UI table section, select the table link, which has the issue. You see start key/end key of each region belonging to that table. Then merge those overlapped regions. In HBase shell, do merge_region 'xxxxxxxx','yyyyyyy', true. For example:

RegionA, startkey:001, endkey:010,

RegionB, startkey:001, endkey:080,

RegionC, startkey:010, endkey:080.

In this scenario, you need to merge RegionA and RegionC and get RegionD with the same key range as RegionB, then merge RegionB and RegionD. xxxxxxx and yyyyyy are the hash string at the end of each region name. Be careful here not to merge two discontinuous regions. After each merge, like merge A and C, HBase will start a compaction on RegionD. Wait for the compaction to finish before doing another merge with RegionD. You can find the compaction status on that region server page in HBase HMaster UI.

Can't load .regioninfo for region /hbase/data/default/tablex/regiony.

It's most likely due to region partial deletion when RegionServer crashes or VM reboots. Currently, the Azure Storage is a flat blob file system and some file operations aren't atomic.

Manually clean up these remaining files and folders:

1. Execute hdfs dfs -ls /hbase/data/default/tablex/regiony to check what folders/files are still under it.

2. Execute hdfs dfs -rmr /hbase/data/default/tablex/regiony/filez to delete all child files/folders

3. Execute hdfs dfs -rmr /hbase/data/default/tablex/regiony to delete the region folder.

If you didn't see your problem or are unable to solve your issue, visit one of the following channels for more support:

• If you need more help, you can submit a support request from the Azure portal. Select Support from the menu bar or open the Help + support hub. For more detailed information, review How to create an Azure support request. Access to Subscription Management and billing support is included with your Azure subscription, and Technical Support is provided through one of the Azure Support Plans.