HBCK2 is the repair tool for Apache HBase clusters.
Problems in operation are bugs. The need for an HBCK2 fix is meant as workaround until the bug is fixed and deployed in a new HBase version.
HBCK2 is the successor to hbck, the repair tool that shipped with HBase 1.x (A.K.A hbck1).
Use HBCK2 in place of hbck1 making repairs against hbase-2.x clusters.
hbck1 should not be run against an HBase 2.x installation as it may do damage.
While hbck1 is still included in HBase 2.x to avoid surprises, it is now deprecated and will be removed in HBase 3.x.
The write-facility (-fix) of hbck1 has been removed.
It can report on the state of a HBase 2.x cluster but its assessments will be inaccurate since it does not understand all internal workings of HBase 2.x.
HBCK2 does not work the way hbck1 used to, even for the case where commands are similarly named across the two versions. See the next section for how the tools differ.
HBCK2 performs a single, discrete task each time it is run. It does not presume a tool can analyze all about the running cluster and then repair 'all problems' found as hbck1 used suggest.
HBCK2 is for fixes. For listings of inconsistencies or blockages in the running cluster, you go elsewhere, to the logs and UI of the running cluster Master. Once an issue has been identified, you use the HBCK2 tool to ask the Master to effect fixes or to skip-over bad state. Asking the Master to make the fixes rather than try and effect the repair locally in a fix-it tool's context is another important difference between HBCK2 and hbck1. More on how this interactive fix-it process works and on HBCK2 workings can be found in sections that follow.
Releases can be found under the HBase distribution directory. See the HBASE Downloads Page.
Run:
$ mvn install
The built HBCK2 jar will be in the target subdirectory.
The HBCK2 jar does not include dependencies; it is not built as a 'fat' jar.
Dependencies must be provided. Building, adjusting the target hbase version in the
top-level pom to match your deployment will make for the smoothest operation when run
against your deployment (See the parent pom.xml hbase-operator-tools for the
hbase.version to set).
Where runtime interaction between HBCK2 and running cluster can get interesting is when HBCK2 is in advance of your hbase deploy such that your hbase does not support all APIs in current HBCK2. Where HBCK2 does not have needed server-side support it should fail gracefully. Use an older release or upgrade your cluster (if you can).
The easiest means of 'providing' HBCK2 its dependencies is by launching
HBCK2 via the $HBASE_HOME/bin/hbase script. The bin/hbase script natively
makes mention of hbck -- there is a hbck option listed in the help output.
By default, running bin/hbase hbck, will run the built-in hbck1 tool.
To run HBCK2, you need to point at a built HBCK2 jar using the -j option
as in:
$ ${HBASE_HOME}/bin/hbase --config /etc/hbase-conf hbck -j ~/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar
/etc/hbase-conf is where the deployment's configuration lives.
The HBCK2 jar is at ~/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar.
The above command with no options or arguments passed will dump out the HBCK2 help:
usage: HBCK2 [OPTIONS] COMMAND <ARGS>
Options:
-d,--debug run with debug output
-h,--help output this help message
-p,--hbase.zookeeper.property.clientPort <arg> port of hbase ensemble
-q,--hbase.zookeeper.quorum <arg> hbase ensemble
-s,--skip skip hbase version check
(PleaseHoldException)
-v,--version this hbck2 version
-z,--zookeeper.znode.parent <arg> parent znode of hbase
ensemble
Command:
addFsRegionsMissingInMeta [OPTIONS]
[<NAMESPACE|NAMESPACE:TABLENAME>...|-i <INPUTFILES>...]
Options:
-i,--inputFiles take one or more files of namespace or table names
-o,--outputFile name/prefix of the file(s) to dump region names
-n,--numLines number of lines to be written to each output file
To be used when regions missing from hbase:meta but directories
are present still in HDFS. Can happen if user has run _hbck1_
'OfflineMetaRepair' against an hbase-2.x cluster. Needs hbase:meta
to be online. For each table name passed as parameter, performs diff
between regions available in hbase:meta and region dirs on HDFS.
Then for dirs with no hbase:meta matches, it reads the 'regioninfo'
metadata file and re-creates given region in hbase:meta. Regions are
re-created in 'CLOSED' state in the hbase:meta table, but not in the
Masters' cache, and they are not assigned either. To get these
regions online, run the HBCK2 'assigns'command printed when this
command-run completes.
NOTE: If using hbase releases older than 2.3.0, a rolling restart of
HMasters is needed prior to executing the set of 'assigns' output.
An example adding missing regions for tables 'tbl_1' in the default
namespace, 'tbl_2' in namespace 'n1' and for all tables from
namespace 'n2':
$ HBCK2 addFsRegionsMissingInMeta default:tbl_1 n1:tbl_2 n2
Returns HBCK2 an 'assigns' command with all re-inserted regions.
SEE ALSO: reportMissingRegionsInMeta
SEE ALSO: fixMeta
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains <NAMESPACE|NAMESPACE:TABLENAME>, one per line.
For example:
$ HBCK2 addFsRegionsMissingInMeta -i fileName1 fileName2
If -o or --outputFile is specified, the output file(s) can be passed as
input to assigns command via -i or -inputFiles option.
If -n or --numLines is specified, and say it is set to 100, this will
create files with prefix as value passed by -o or --outputFile option.
Each file will have 100 region names (max.), one per line.
For example:
$ HBCK2 addFsRegionsMissingInMeta -o outputFilePrefix -n 100
-i fileName1 fileName2
But if -n is not specified, but -o is specified, it will dump all
region names in a single file, one per line.
NOTE: -n option is applicable only if -o option is specified.
assigns [OPTIONS] [<ENCODED_REGIONNAME>...|-i <INPUT_FILE>...]
Options:
-o,--override override ownership by another procedure
-i,--inputFiles take one or more files of encoded region names
-b,--batchSize number of regions to process in a batch
A 'raw' assign that can be used even during Master initialization (if
the -skip flag is specified). Skirts Coprocessors. Pass one or more
encoded region names. 1588230740 is the hard-coded name for the
hbase:meta region and de00010733901a05f5a2a3a382e27dd4 is an example of
what a user-space encoded region name looks like. For example:
$ HBCK2 assigns 1588230740 de00010733901a05f5a2a3a382e27dd4
Returns the pid(s) of the created AssignProcedure(s) or -1 if none.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains encoded region names, one per line. For example:
$ HBCK2 assigns -i fileName1 fileName2
If -b or --batchSize is specified, the command processes those many
regions at a time in a batch-ed manner; Consider using this option,
if the list of regions is huge, to avoid CallTimeoutException.
For example:
$ HBCK2 assigns -i fileName1 fileName2 -b 500
By default, batchSize is set to -1 i.e. no batching is done.
bypass [OPTIONS] [<PID>...|-i <INPUT_FILE>...]
Options:
-o,--override override if procedure is running/stuck
-r,--recursive bypass parent and its children. SLOW! EXPENSIVE!
-w,--lockWait milliseconds to wait before giving up; default=1
-i,--inputFiles take one or more input files of PID's
-b,--batchSize number of procedures to process in a batch
Pass one (or more) procedure 'pid's to skip to procedure finish. Parent
of bypassed procedure will also be skipped to the finish. Entities will
be left in an inconsistent state and will require manual fixup. May
need Master restart to clear locks still held. Bypass fails if
procedure has children. Add 'recursive' if all you have is a parent pid
to finish parent and children. This is SLOW, and dangerous so use
selectively. Does not always work.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains PID's, one per line. For example:
$ HBCK2 bypass -i fileName1 fileName2
If -b or --batchSize is specified, the command processes those many
procedures at a time in a batch-ed manner; Consider using this option,
if the list of procedures is huge, to avoid CallTimeoutException.
For example:
$ HBCK2 bypass -i fileName1 fileName2 -b 500
By default, batchSize is set to -1 i.e. no batching is done.
extraRegionsInMeta [<NAMESPACE|NAMESPACE:TABLENAME>...|
-i <INPUT_FILE>...]
Options:
-f, --fix fix meta by removing all extra regions found.
-i,--inputFiles take one or more input files of namespace or
table names
Reports regions present on hbase:meta, but with no related
directories on the file system. Needs hbase:meta to be online.
For each table name passed as parameter, performs diff
between regions available in hbase:meta and region dirs on the given
file system. Extra regions would get deleted from Meta
if passed the --fix option.
NOTE: Before deciding on use the "--fix" option, it's worth check if
reported extra regions are overlapping with existing valid regions.
If so, then "extraRegionsInMeta --fix" is indeed the optimal solution.
Otherwise, "assigns" command is the simpler solution, as it recreates
regions dirs in the filesystem, if not existing.
An example triggering extra regions report for tables 'table_1'
and 'table_2', under default namespace:
$ HBCK2 extraRegionsInMeta default:table_1 default:table_2
An example triggering extra regions report for table 'table_1'
under default namespace, and for all tables from namespace 'ns1':
$ HBCK2 extraRegionsInMeta default:table_1 ns1
Returns list of extra regions for each table passed as parameter, or
for each table on namespaces specified as parameter.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains <NAMESPACE|NAMESPACE:TABLENAME>, one per line.
For example:
$ HBCK2 extraRegionsInMeta -i fileName1 fileName2
filesystem [OPTIONS] [<TABLENAME>...|-i <INPUT_FILE>...]
Options:
-f, --fix sideline corrupt hfiles, bad links, and references.
-i,--inputFiles take one or more input files of table names
Report on corrupt hfiles, references, broken links, and integrity.
Pass '--fix' to sideline corrupt files and links. '--fix' does NOT
fix integrity issues; i.e. 'holes' or 'orphan' regions. Pass one or
more tablenames to narrow checkup. Default checks all tables and
restores 'hbase.version' if missing. Interacts with the filesystem
only! Modified regions need to be reopened to pick-up changes.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains table names, one per line. For example:
$ HBCK2 filesystem -i fileName1 fileName2
fixMeta
Do a server-side fix of bad or inconsistent state in hbase:meta.
Available in hbase 2.2.1/2.1.6 or newer versions. Master UI has
matching, new 'HBCK Report' tab that dumps reports generated by
most recent run of _catalogjanitor_ and a new 'HBCK Chore'. It
is critical that hbase:meta first be made healthy before making
any other repairs. Fixes 'holes', 'overlaps', etc., creating
(empty) region directories in HDFS to match regions added to
hbase:meta. Command is NOT the same as the old _hbck1_ command
named similarily. Works against the reports generated by the last
catalog_janitor and hbck chore runs. If nothing to fix, run is a
noop. Otherwise, if 'HBCK Report' UI reports problems, a run of
fixMeta will clear up hbase:meta issues. See 'HBase HBCK' UI
for how to generate new report.
SEE ALSO: reportMissingRegionsInMeta
generateMissingTableDescriptorFile [OPTIONS] [<TABLENAME>...]
Trying to fix an orphan table by generating a missing table descriptor
file. This command will have no effect if the table folder is missing
or if the .tableinfo is present (we don't override existing table
descriptors). This command will first check it the TableDescriptor is
cached in HBase Master in which case it will recover the .tableinfo
accordingly. If TableDescriptor is not cached in master then it will
create a default .tableinfo file with the following items:
- the table name
- the column family list determined based on the file system
- the default properties for both TableDescriptor and
ColumnFamilyDescriptors
If the .tableinfo file was generated using default parameters then
make sure you check the table / column family properties later (and
change them if needed).
This method does not change anything in HBase, only writes the new
.tableinfo file to the file system. Orphan tables can cause e.g.
ServerCrashProcedures to stuck, you might need to fix these still
after you generated the missing table info files. If no tables are
specified, .tableinfo will be generated for all missing table
descriptors.
recoverUnknown
Schedule ServerCrashProcedure(SCP) for RegionServers that are reported
as unknown.
Returns the pid(s) of the created ServerCrashProcedure(s) or -1 if
no procedure created (see master logs for why not).
Command support added in hbase versions 2.2.7, 2.3.5, 2.4.3,
2.5.0 or newer.
regionInfoMismatch
Options:
-f,--fix Update hbase:meta with the corrections
It is recommended to first run this utility without the fix
option to ensure that the utility is generating the correct
serialized RegionInfo data structures. Inspect the output to
confirm that the hbase:meta rowkey matches the new RegionInfo.
This tool will read hbase:meta and report any regions whose rowkey
and cell value differ in their encoded region name. HBASE-23328
illustrates a problem for read-replica enabled tables in which
the encoded region name (the MD5 hash) does not match between
the rowkey and the value. This problem is generally harmless
for normal operation, but can break other HBCK2 tools.
Run this command to determine if any regions are affected by
this bug and use the -f/--fix option to then correct any
affected regions.
replication [OPTIONS] [<TABLENAME>...|-i <INPUT_FILE>...]
Options:
-f, --fix fix any replication issues found.
-i,--inputFiles take one or more input files of table names
Looks for undeleted replication queues and deletes them if passed the
'--fix' option. Pass a table name to check for replication barrier and
purge if '--fix'.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains <TABLENAME>, one per line. For example:
$ HBCK2 replication -i fileName1 fileName2
reportMissingRegionsInMeta [<NAMESPACE|NAMESPACE:TABLENAME>...|
-i <INPUT_FILE>...]
Options:
-i,--inputFiles take one or more files of namespace or table names
To be used when regions missing from hbase:meta but directories
are present still in HDFS. Can happen if user has run _hbck1_
'OfflineMetaRepair' against an hbase-2.x cluster. This is a CHECK only
method, designed for reporting purposes and doesn't perform any
fixes, providing a view of which regions (if any) would get re-added
to hbase:meta, grouped by respective table/namespace. To effectively
re-add regions in meta, run addFsRegionsMissingInMeta.
This command needs hbase:meta to be online. For each namespace/table
passed as parameter, it performs a diff between regions available in
hbase:meta against existing regions dirs on HDFS. Region dirs with no
matches are printed grouped under its related table name. Tables with
no missing regions will show a 'no missing regions' message. If no
namespace or table is specified, it will verify all existing regions.
It accepts a combination of multiple namespace and tables. Table names
should include the namespace portion, even for tables in the default
namespace, otherwise it will assume as a namespace value.
An example triggering missing regions report for tables 'table_1'
and 'table_2', under default namespace:
$ HBCK2 reportMissingRegionsInMeta default:table_1 default:table_2
An example triggering missing regions report for table 'table_1'
under default namespace, and for all tables from namespace 'ns1':
$ HBCK2 reportMissingRegionsInMeta default:table_1 ns1
Returns list of missing regions for each table passed as parameter, or
for each table on namespaces specified as parameter.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains <NAMESPACE|NAMESPACE:TABLENAME>, one per line.
For example:
$ HBCK2 reportMissingRegionsInMeta -i fileName1 fileName2
setRegionState [<ENCODED_REGIONNAME> <STATE>|-i <INPUT_FILE>...]
Options:
-i,--inputFiles take one or more input files of encoded region names
and states.
To set the replica region's state, it needs the primary region's
encoded regionname and replica id. The command will be
setRegionState <PRIMARY_ENCODED_REGIONNAME>,<replicaId> <STATE>
Possible region states:
OFFLINE, OPENING, OPEN, CLOSING, CLOSED, SPLITTING, SPLIT,
FAILED_OPEN, FAILED_CLOSE, MERGING, MERGED, SPLITTING_NEW,
MERGING_NEW, ABNORMALLY_CLOSED
WARNING: This is a very risky option intended for use as last resort.
Example scenarios include unassigns/assigns that can't move forward
because region is in an inconsistent state in 'hbase:meta'. For
example, the 'unassigns' command can only proceed if passed a region
in one of the following states: SPLITTING|SPLIT|MERGING|OPEN|CLOSING
Before manually setting a region state with this command, please
certify that this region is not being handled by a running procedure,
such as 'assign' or 'split'. You can get a view of running procedures
in the hbase shell using the 'list_procedures' command. An example
setting region 'de00010733901a05f5a2a3a382e27dd4' to CLOSING:
$ HBCK2 setRegionState de00010733901a05f5a2a3a382e27dd4 CLOSING
Returns "0" if region state changed and "1" otherwise.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains <ENCODED_REGIONNAME> <STATE>, one pair per line.
For example:
$ HBCK2 setRegionState -i fileName1 fileName2
setTableState [<TABLENAME> <STATE>|-i <INPUT_FILE>...]
Options:
-i,--inputFiles take one or more files of table names and states
Possible table states: ENABLED, DISABLED, DISABLING, ENABLING
To read current table state, in the hbase shell run:
hbase> get 'hbase:meta', '<TABLENAME>', 'table:state'
A value of \x08\x00 == ENABLED, \x08\x01 == DISABLED, etc.
Can also run a 'describe "<TABLENAME>"' at the shell prompt.
An example making table name 'user' ENABLED:
$ HBCK2 setTableState users ENABLED
Returns whatever the previous table state was.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains <TABLENAME> <STATE>, one pair per line.
For example:
$ HBCK2 setTableState -i fileName1 fileName2
scheduleRecoveries [<SERVERNAME>...|-i <INPUT_FILE>...]
Options:
-i,--inputFiles take one or more input files of server names
Schedule ServerCrashProcedure(SCP) for list of RegionServers. Format
server name as '<HOSTNAME>,<PORT>,<STARTCODE>' (See HBase UI/logs).
Example using RegionServer 'a.example.org,29100,1540348649479':
$ HBCK2 scheduleRecoveries a.example.org,29100,1540348649479
Returns the pid(s) of the created ServerCrashProcedure(s) or -1 if
no procedure created (see master logs for why not).
Command support added in hbase versions 2.0.3, 2.1.2, 2.2.0 or newer.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains <SERVERNAME>, one per line. For example:
$ HBCK2 scheduleRecoveries -i fileName1 fileName2
unassigns [OPTIONS] [<ENCODED_REGIONNAME>...|-i <INPUT_FILE>...]
Options:
-o,--override override ownership by another procedure
-i,--inputFiles take one or more input files of encoded region names
-b,--batchSize number of regions to process in a batch
A 'raw' unassign that can be used even during Master initialization
(if the -skip flag is specified). Skirts Coprocessors. Pass one or
more encoded region names. 1588230740 is the hard-coded name for the
hbase:meta region and de00010733901a05f5a2a3a382e27dd4 is an example
of what a userspace encoded region name looks like. For example:
$ HBCK2 unassigns 1588230740 de00010733901a05f5a2a3a382e27dd4
Returns the pid(s) of the created UnassignProcedure(s) or -1 if none.
SEE ALSO, org.apache.hbase.hbck1.OfflineMetaRepair, the offline
hbase:meta tool. See the HBCK2 README for how to use.
If -i or --inputFiles is specified, pass one or more input file names.
Each file contains encoded region names, one per line. For example:
$ HBCK2 unassigns -i fileName1 fileName2
If -b or --batchSize is specified, the tool processes those many
regions at a time in a batch-ed manner; Consider using this option,
if the list of regions is huge, to avoid CallTimeoutException.
For example:
$ HBCK2 unassigns -i fileName1 fileName2 -b 500
By default, batchSize is set to -1 i.e. no batching is done.
Note that when you pass bin/hbase the hbck argument, it will by
default use the shaded client to get to the targeted hbase cluster.
This is sufficient for most HBCK2 usage. If you run into complaints
like the below:
bin/hbase --config hbase-conf hbck
2019-08-30 05:04:54,467 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
at org.apache.hadoop.hbase.util.CommonFSUtils.getRootDir(CommonFSUtils.java:361)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3605)
... it is because the HDFS jars are not on the CLASSPATH. The default is NOT
to bundle HDFS jars on the CLASSPATH when running hbck via bin/hbase. Define
HADOOP_HOME in the environment so bin/hbase can find your local Hadoop
installation, and then it will load its HDFS jars.
HBCK2 is currently a simple tool that does one thing at a time only.
In hbase-2.x, the Master is the final arbiter of all state, so a general principle for most HBCK2 commands is that it asks the Master to effect all repair. This means a Master must be up before you can run HBCK2 commands.
The HBCK2 implementation approach is to make use of an
HbckService hosted on the Master. The Service publishes a few methods for the HBCK2 tool to
pull on. Therefore, for HBCK2 commands relying on Master's HbckService facade,
first thing HBCK2 does is poke the cluster to ensure the service is available.
This will fail if the remote Server does not publish the Service or if the
HbckService is lacking the requested method. For the latter case, if you can,
update your cluster to obtain more fix facility.
HBCK2 versions should be able to work across multiple hbase-2 releases. It will
fail with a complaint if it is unable to run. There is no HbckService in versions
of hbase before 2.0.3 and 2.1.1. HBCK2 will not work against these versions.
Next we look first at how you 'find' issues in your running cluster followed by a section on how you 'fix' found problems.
While hbck1 performed analysis reporting your cluster GOOD or BAD, HBCK2 is less presumptuous. In hbase-2.x, the operator figures what needs fixing and then uses tooling including HBCK2 to do fixup. The operator may have to go a few rounds of back and forth running HBCK2 then checking cluster state.
To figure cluster issues, make use of the following utilities and emissions.
The Master runs all assignments, server crash handling, cluster start and
stop, etc. In hbase-2.x, all that the Master does has been cast as
Procedures run on a state machine engine. See
Procedure Framework
and Assignment Manager
for detail on how this new infrastructure works. Each Procedure has a
unique Procedure id, its pid, that it lists on each logging.
Following the pid, you can trace the lifecycle of a Procedure in the
Master logs as Procedures transition from start, through each of the
Procedure's various stages to finish. Some Procedures spawn sub-procedures,
wait on their Children, and then themselves finish. Each child logs
its pid but also its ppid; its parent's pid.
Generally all run problem free but if some unforeseen circumstance arises, the assignment framework may sustain damage requiring operator intervention. Below we will discuss some such scenarios, but they can manifest in the Master log as a Region being STUCK or a Procedure transitioning an entity -- a Region or a Table -- may be blocked because another Procedure holds the exclusive lock and is not letting go.
STUCK Procedures look like this:
2018-09-12 15:29:06,558 WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=va1001.example.org,22101,1536173230599, table=IntegrationTestBigLinkedList_20180626110336, region=dbdb56242f17610c46ea044f7a42895b
This section about midway down in Master UI home-page shows a list of tables with columns for whether the table is ENABLED, ENABLING, DISABLING, or DISABLED among other attributes. Also listed are columns with counts of Regions in their various transition states: OPEN, CLOSED, etc. A read of this table is good for figuring if the Regions of this table have a proper disposition. For example if a table is ENABLED and there are Regions that are not in the OPEN state and the Master Log is silent about any ongoing assigns, then something is amiss.
This page off the Master UI home page under the Procedures & Locks menu item in the page heading lists all ongoing Procedures and Locks as well as the current set of Master Procedure WALs (named pv2-0000000000000000###.log under the MasterProcWALs directory in your hbase install). On startup, on a large cluster when furious assigning is afoot, this page is filled with lists of Procedures and Locks. The count of MasterProcWALs will bloat too. If after the cluster settles, there is a stuck Lock or Procedure or the count of WALs doesn't ever come down but only grows, then operator intervention is needed to remove the blockage.
Lists of locks and procedures can also be obtained via the hbase shell:
$ echo "list_locks"| hbase shell &> /tmp/locks.txt
$ echo "list_procedures"| hbase shell &> /tmp/procedures.txt
Starting with HBase 2.3.0/2.1.6/2.2.1, the Master UI now includes a HBCK Report page located at /hbck.jsp.
This pages displays the output from two inspections run by the Master at regular intervals.
- The first is performed by the
CatalogJanitorand reports any overlaps in regions or holes inhbase:meta. - The second inspection is a background 'chore' process that compares
hbase:metaand filesystem content, and makes a note of any anomalies in the HBCK Report section.
If you want to force a run of these inspectors, refer to the HBCK Report page for instructions.
Look at the fixMeta command to fix overlaps and holes found by these inspections.
The Canary tool is useful verifying the state of assign. It can be run with a table focus or against the whole cluster.
For example, to check cluster assigns:
$ hbase canary -f false -t 6000000 &>/tmp/canary.log
The -f false tells the Canary to keep going across failed Region fetches and the -t 6000000 tells the Canary run for ~two hours maximum. When done, check out /tmp/canary.log. Grep for ERROR lines to find problematic Region assigns.
You can do a probe like the Canary's in the hbase shell. For example, given a Region that has a start row of d1dddd0c belonging to the table testtable, do as follows:
hbase> scan 'testtable', {STARTROW => 'd1dddd0c', LIMIT => 10}
For an overview on parsing a Region name into its constituent parts, see RegionInfo API.
To figure the list of Regions that are not OPEN on an ENABLED or ENABLING table, read the hbase:meta table info:state column. For example, to find the state of all Regions in the table IntegrationTestBigLinkedList_20180626064758, do the following:
$ echo " scan 'hbase:meta', {ROWPREFIXFILTER => 'IntegrationTestBigLinkedList_20180626064758,', COLUMN => 'info:state'}"| hbase shell > /tmp/t.txt
...then grep for OPENING or CLOSING Regions.
To move an OPENING issue to OPEN, so it agrees with a table's
ENABLED state, use the assign command in the hbase shell to
queue a new Assign Procedure (watch the Master logs to see the
Assign run). If many Regions to assign, use the HBCK2 tool. It
can do bulk assigning.
When making repair, make sure hbase:meta is consistent first before you go about fixing any other issue type such as a filesystem deviance. Deviance in the filesystem or problems with assign should be addressed after the hbase:meta has been put in order. If hbase:meta is out of whack, the Master cannot make proper placements when adopting orphan filesystem data or making region assignments.
Other general principles to keep in mind include a Region cannot be assigned if it is in CLOSING state (or the inverse, unassigned if in OPENING state) without first transitioning via CLOSED: Regions must always move from CLOSED, to OPENING, to OPEN, and then to CLOSING, CLOSED.
When making repair, do fixup of a table-at-a-time.
Also, if a table is DISABLED, you cannot assign a Region. In the Master logs, you will see that the Master will report that the assign has been skipped because the table is DISABLED. You may want to assign a Region because it is currently in the OPENING state and you want it in the CLOSED state so it agrees with the table's DISABLED state. In this situation, you may have to temporarily set the table status to ENABLED, just so you can do the assign, and then set it back again after the unassign. HBCK2 has facility to allow you to do this. See the HBCK2 usage output.
What follows is a mix of notes and prescription that comes of experience running hbase-2.x so far. The root issues that brought on states described below has been fixed in later versions of HBase so upgrade if you can to avoid scenarios described.
Generally, on assign, the Master will persist until successful. An assign takes an exclusive lock on the Region. This precludes a concurrent assign or unassign from running. An assign against a locked Region will wait until the lock is released before making progress. See the [Procedures & Locks] section above for current list of outstanding Locks.
This should never happen. If it does, here is what it looks like:
2018-10-01 22:07:42,792 WARN org.apache.hadoop.hbase.master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=CLOSING, ts=1538456302300, server=ve1017.example.org,22101,1538449648131}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined.
The Master is unable to continue startup because there is no Procedure to assign hbase:meta (or hbase:namespace). To inject one, use the HBCK2 tool:
HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase org.apache.hbase.HBCK2 -skip assigns 1588230740
...where 1588230740 is the encoded name of the hbase:meta Region. Pass the '-skip' option to stop HBCK2 doing a version check against the remote master. If the remote master is not up, the version check will prompt a 'Master is initializing response' or 'PleaseHoldException' and drop the assign attempt. The '-skip' command punts on version check and will land the scheduled assign.
The same may happen to the hbase:namespace system table. Look for the encoded Region name of the hbase:namespace Region and do similar to what we did for hbase:meta. In this latter case, the Master actually prints out a helpful message that looks like the following:
2019-07-09 22:08:38,966 WARN [master/localhost:16000:becomeActiveMaster] master.HMaster: hbase:namespace,,1562733904278.9559cf72b8e81e1291c626a8e781a6ae. is NOT online; state={9559cf72b8e81e1291c626a8e781a6ae state=CLOSED, ts=1562735318897, server=null}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined.
To schedule an assign for the hbase:namespace table noted in the above log line, you would do:
$ ${HBASE_HOME}/bin/hbase --config /etc/hbase-conf hbck -j ./hbase-hbck2-1.0.0-SNAPSHOT.jar -skip assigns 9559cf72b8e81e1291c626a8e781a6ae
... passing the encoded name for the namespace region (the encoded name will differ per deploy).
There have been some unusual cases where table regions have been removed from hbase:meta table. Some triage on such cases revealed these were operator-induced. Users had run the obsolete hbck1 OfflineMetaRepair tool against an HBCK2 cluster. OfflineMetaRepair is a well known tool for fixing hbase:meta table related issues on HBase 1.x versions. The original version is not compatible with HBase 2.x or higher versions, and it has undergone some adjustments so in the extreme, it can now be run via HBCK2.
In most of these cases, regions end up missing in hbase:meta at random, but hbase may still be operational. In such situations, problem can be addressed with the Master online, using the addFsRegionsMissingInMeta command in HBCK2. This command is less disruptive to hbase than a full hbase:meta rebuild covered later, and it can be used even for recovering the namespace table region.
There can also be situations where table regions have been removed file system, but still have related entries on hbase:meta table. This may happen due to problems on splitting, manual operation mistakes (like deleting/moving the region dir manually), or even meta info data loss issues such as HBASE-21843.
Such problem can be addressed with the Master online, using the extraRegionsInMeta --fix command in HBCK2. This command is less disruptive to hbase than a full hbase:meta rebuild covered later. Also useful when this happens on versions that don't support fixMeta hbck2 option (any prior to "2.0.6", "2.1.6", "2.2.1", "2.3.0","3.0.0").
If hbase:meta corruption is not too critical, hbase would still be able to bring it online. Even if namespace region is among the missing regions, it will still be possible to scan hbase:meta during the initialization period, where Master will be waiting for namespace to be assigned. To verify this situation, a hbase:meta scan command can be executed as below. If it does not time out or show any errors, hbase:meta is online:
echo "scan 'hbase:meta', {COLUMN=>'info:regioninfo'}" | hbase shell
HBCK2 addFsRegionsMissingInMeta can be used if the above does not show any errors. It reads region metadata info available on the FS region directories to recreate regions in hbase:meta. Since it can run with hbase partially operational, it attempts to disable online tables that are affected by the reported problem and it is going to readd regions to hbase:meta. It can check for specific tables/namespaces, or all tables from all namespaces. An example below shows adding missing regions for tables 'tbl_1' in the default namespace, 'tbl_2' in namespace 'n1', and for all tables from namespace 'n2':
$ HBCK2 addFsRegionsMissingInMeta default:tbl_1 n1:tbl_2 n2
As it operates independently of Master, once it finishes successfully, additional steps are required to actually have the re-added regions assigned. These are listed below:
-
addFsRegionsMissingInMeta outputs an assigns command with all regions that got re-added. This command needs to be executed later, so copy and save it for convenience.
-
For HBase versions prior to 2.3.0, after addFsRegionsMissingInMeta finished successfully and output has been saved, restart all running HBase Masters.
-
Once Master's are restarted and hbase:meta is already online (check if Web UI is accessible), run assigns command from addFsRegionsMissingInMeta output saved per instructions from #1.
NOTE: If namespace region is among the missing regions, you will need to add --skip flag at the beginning of assigns command returned.
Should a cluster suffer a catastrophic loss of the hbase:meta table, a rough rebuild is possible using the following recipe.
In outline, we stop the cluster; run the HBCK2 OfflineMetaRepair tool which reads directories and metadata dropped into the filesystem
making a best effort at reconstructing a viable hbase:meta table; restart your cluster; inject an assign to bring the system
namespace table online; and then finally, re-assign userspace tables you'd like enabled (the rebuilt hbase:meta creates a table with all tables offline and no regions assigned).
Stop the cluster.
Run the rebuild hbase:meta command from HBCK2. This will move aside the original hbase:meta and put in place a newly rebuilt one. Below is an example of how to run the tool. It adds the -details flag so the tool dumps info on the regions its found in hdfs:
$ HBASE_CLASSPATH_PREFIX=~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar ./bin/hbase org.apache.hbase.hbck1.OfflineMetaRepair -details
Start the cluster up. It won’t come up fully. It will be stuck because the namespace table is not online and there is no assign procedure in the procedure store for this contingency. The hbase master log will show this state. Here is an example of what it will log:
2019-07-10 18:30:51,090 WARN [master/localhost:16000:becomeActiveMaster] master.HMaster: hbase:namespace,,1562808216225.725a0fe6c2c869d3d0a9ed82bfa80fa3. is NOT online; state={725a0fe6c2c869d3d0a9ed82bfa80fa3 state=CLOSED, ts=1562808619952, server=null}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
To assign the namespace table region, you cannot use the shell. If you use the shell, it will fail with a PleaseHoldException because the master is not yet up (it is waiting for the namespace table to come online before it declares itself ‘up’). You have to use the HBCK2 assigns command. To assign, you will need the namespace encoded name. It shows in the log quoted above: i.e. 725a0fe6c2c869d3d0a9ed82bfa80fa3 in this case. You will also have to pass the -skip command to ‘skip’ the master version check (without it, your HBCK2 invocation will also elicit the above PleaseHoldException because the master is not yet up). Here is an example adding an assign of the namespace table:
$ HBASE_CLASSPATH_PREFIX=~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar ./bin/hbase org.apache.hbase.HBCK2 -skip assigns 725a0fe6c2c869d3d0a9ed82bfa80fa3
If the invocation comes back with ‘Connection refused’, is the Master up? The Master will shut down after a while if it can’t initialize itself. Just restart the cluster/master and rerun the above assigns command.
When the assigns runs successfully, you’ll see it emit the likes of the following. The ‘48’ on the end is the pid of the assign procedure schedule. If the pid returned is ‘-1’, then the master startup has not progressed sufficiently… retry. Or, the encoded regionname is incorrect. Check.
$ HBASE_CLASSPATH_PREFIX=~/checkouts/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar ./bin/hbase org.apache.hbase.HBCK2 -skip assigns 725a0fe6c2c869d3d0a9ed82bfa80fa3
18:40:43.817 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18:40:44.315 [main] INFO org.apache.hbase.HBCK2 - hbck support check skipped
[48]
Check the master logs. The master should have come up. You’ll see successful completion of pid=48. Look for a line like this to verify successful master launch:
master.HMaster: Master has completed initialization 132.515sec
It might take a while to appear.
The rebuild of hbase:meta adds the user tables in DISABLED state and the regions in CLOSED mode. Re-enable tables via the shell to bring all table regions back online.
Do it one-at-a-time or see the enable_all ".*" command to enable all tables in one shot.
The rebuild meta will likely be missing edits and may need subsequent repair and cleaning using facility outlined higher up in this README.
HBCK2 can check for hanging references and corrupt HFiles. You can ask it to sideline bad files, which may be needed to get over humps where regions won't online or reads are failing. See the filesystem command in the HBCK2 listing. Pass one or more tablename (or 'none' to check all tables). It will report bad files. Pass the --fix option to effect repairs.
At an extreme, as a last resource, if the Master is distraught and all
attempts at fixup only turn up undoable locks or Procedures that won't finish, and/or
the set of MasterProcWALs is growing without bound, it is
possible to wipe the Master state clean. Just move aside the
/hbase/MasterProcWALs/ directory under your hbase install and
restart the Master process. It will come back as a tabula rasa without
memory of the bad times past.
If at the time of the erasure, all Regions were happily assigned or offlined, then on Master restart, the Master should pick up and continue as though nothing happened. But if there were Regions-In-Transition at the time, then the operator will have to intervene to bring outstanding assigns/unassigns to their terminal point. Read the hbase:meta info:state columns as described above to figure what needs assigning/unassigning. Having erased all history moving aside the MasterProcWALs, none of the entities should be locked so you are free to bulk assign/unassign.
For how to fix orphan regions reported by the 'HBCK Chore',
see the advanced section on the completebulkload tool in the refguide,
'Adopting' Stray Data.