HadoopExam Learning Resources

HadoopExam Training, Interview Questions, Certifications, Projects, POC and Hands On exercise access

    40000+ Learners upgraded/switched career    Testimonials

BUG: HBase canary failing on kerberized CDH5.4.1 install

We're running CDH5.4.1 managed by Cloudera Manager 5.3.  We have successfully configured the cluster to enable Kerberos and HA.  Currently, we have enabled YARN, HDFS, and HBase.  While looking through the HBase logs, we found that the canary is failing to connect after enabling Kerberos.  I have tried with jdk7_55 and jdk7_79.

The canary is throwing these two errors about once a second and then quitting.
2015-05-18 15:51:31,788 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
2015-05-18 15:51:32,115 ERROR org.apache.hadoop.hbase.AuthUtil: Error while trying to perform the initial login
To the best of my knowledge, Kerberos is working; I can see the region server startup and authenticate to the kerberos domain without error and CM shows that the general health of the cluster is ok.
Looking at stderr for the hbase startup, the canary is quitting with:
+ echo 'Mon May 18 17:13:04 CDT 2015 RS pid:87973 ' 'Starting the canary'
+ /opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/3361-hbase-REGIONSERVER org.apache.hadoop.hbase.tool.Canary -t 15000 -daemon -interval 6 -regionserver dtbrc42.skipper.hdp.indeed.net
Exception in thread "main" java.io.IOException: Running in secure mode, but config doesn't have a keytab
	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:383)
	at org.apache.hadoop.hbase.security.User.login(User.java:250)
	at org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:115)
	at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:61)
	at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:776)
if I look at the KRB5CCNAME cache file, I can see that there's a valid ticket in there:
[hbase@dtbrc42 ~]$  export CONF_DIR=/var/run/cloudera-scm-agent/process/3361-hbase-REGIONSERVER
[hbase@dtbrc42 ~]$      export KRB5CCNAME=$CONF_DIR/krb5cc_$(id -u)
[hbase@dtbrc42 ~]$ klist
Ticket cache: FILE:/var/run/cloudera-scm-agent/process/3361-hbase-REGIONSERVER/krb5cc_483
Valid starting     Expires            Service principal
renew until 05/23/15 15:44:24
  userProvider.login("hbase.client.keytab.file", "hbase.client.kerberos.principal", host);
neither of those exist in the config files located in /var/run/cloudera-scm-agent/process/3361-hbase-REGIONSERVER or in /etc/hbase/conf.cloudera.base.
I attempted adding them to the process instance directory files and running the canary by hand, but that fails. If I add them to the files in /etc/hbase/conf.cloudera.base, that seems to work ... or at least, I don't get any further login errors and I see the logs filling with:
2015-05-18 18:50:19,048 INFO org.apache.hadoop.hbase.tool.Canary: No RegionServerInfo found, regionServerName:dtbrc42.prod.hdp.example.net
2015-05-18 18:50:26,048 INFO org.apache.hadoop.hbase.tool.Canary: No RegionServerInfo found, regionServerName:dtbrc42.prod.hdp.example.net
2015-05-18 18:50:33,049 INFO org.apache.hadoop.hbase.tool.Canary: No RegionServerInfo found, regionServerName:dtbrc42.prod.hdp.example.net

Thanks for reporting this Travis. We have an internal bug tracking 
this (OPSAPS-26468), and it should be fixed in a future release. 

For the moment a good workaround is to add the below to the RS 
hbase-site.xml safety valve: 


P.s. Please note though that that use of CDH newer than the CM version is 
unsupported and such a combination is not tested for. The CM release 
will not be able to use any newer CDH changes/features. Please also 
upgrade your CM version to at least equal of 
CDH version. 

Visit Home Page : http://hadoopexam.com for more detail . As you are not blacklisted user.

You are here: Home Question & Answer Hadoop Questions BUG: HBase canary failing on kerberized CDH5.4.1 install