HadoopExam Learning Resources

CCD-410 Certifcation CCA-500 Hadoop Administrator Exam HBase Certifcation CCB-400 Data Science Certifcation Hadoop Training with Hands On Lab Hadoop Package Deal

BUG: HBase canary failing on kerberized CDH5.4.1 install

We're running CDH5.4.1 managed by Cloudera Manager 5.3.  We have successfully configured the cluster to enable Kerberos and HA.  Currently, we have enabled YARN, HDFS, and HBase.  While looking through the HBase logs, we found that the canary is failing to connect after enabling Kerberos.  I have tried with jdk7_55 and jdk7_79.

The canary is throwing these two errors about once a second and then quitting.
2015-05-18 15:51:31,788 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
2015-05-18 15:51:32,115 ERROR org.apache.hadoop.hbase.AuthUtil: Error while trying to perform the initial login
To the best of my knowledge, Kerberos is working; I can see the region server startup and authenticate to the kerberos domain without error and CM shows that the general health of the cluster is ok.
Looking at stderr for the hbase startup, the canary is quitting with:
+ echo 'Mon May 18 17:13:04 CDT 2015 RS pid:87973 ' 'Starting the canary'
+ /opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/3361-hbase-REGIONSERVER org.apache.hadoop.hbase.tool.Canary -t 15000 -daemon -interval 6 -regionserver dtbrc42.skipper.hdp.indeed.net
Exception in thread "main" java.io.IOException: Running in secure mode, but config doesn't have a keytab
	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:383)
	at org.apache.hadoop.hbase.security.User.login(User.java:250)
	at org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:115)
	at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:61)
	at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:776)
if I look at the KRB5CCNAME cache file, I can see that there's a valid ticket in there:
[hbase@dtbrc42 ~]$  export CONF_DIR=/var/run/cloudera-scm-agent/process/3361-hbase-REGIONSERVER
[hbase@dtbrc42 ~]$      export KRB5CCNAME=$CONF_DIR/krb5cc_$(id -u)
[hbase@dtbrc42 ~]$ klist
Ticket cache: FILE:/var/run/cloudera-scm-agent/process/3361-hbase-REGIONSERVER/krb5cc_483
Valid starting     Expires            Service principal
renew until 05/23/15 15:44:24
  userProvider.login("hbase.client.keytab.file", "hbase.client.kerberos.principal", host);
neither of those exist in the config files located in /var/run/cloudera-scm-agent/process/3361-hbase-REGIONSERVER or in /etc/hbase/conf.cloudera.base.
I attempted adding them to the process instance directory files and running the canary by hand, but that fails. If I add them to the files in /etc/hbase/conf.cloudera.base, that seems to work ... or at least, I don't get any further login errors and I see the logs filling with:
2015-05-18 18:50:19,048 INFO org.apache.hadoop.hbase.tool.Canary: No RegionServerInfo found, regionServerName:dtbrc42.prod.hdp.example.net
2015-05-18 18:50:26,048 INFO org.apache.hadoop.hbase.tool.Canary: No RegionServerInfo found, regionServerName:dtbrc42.prod.hdp.example.net
2015-05-18 18:50:33,049 INFO org.apache.hadoop.hbase.tool.Canary: No RegionServerInfo found, regionServerName:dtbrc42.prod.hdp.example.net

Thanks for reporting this Travis. We have an internal bug tracking 
this (OPSAPS-26468), and it should be fixed in a future release. 

For the moment a good workaround is to add the below to the RS 
hbase-site.xml safety valve: 


P.s. Please note though that that use of CDH newer than the CM version is 
unsupported and such a combination is not tested for. The CM release 
will not be able to use any newer CDH changes/features. Please also 
upgrade your CM version to at least equal of 
CDH version. 

Add comment

Security code

You are here: Home Question & Answer Hadoop Questions BUG: HBase canary failing on kerberized CDH5.4.1 install