kerberos authentication error when using HDFS compatibility

[chip-factual]

When running a drake script that accesses hdfs, I'm getting a kerberos authentication error, even after running kinit. I've verified with a few other engineers that they're experiencing the same issue.

For example, when executing:

test.out <- hdfs:/tmp/test.out
  cat $INPUT > $OUTPUT

I get:

chip@chip-d48 ~/code/drake (master) > ./bin/drake -w /home/chip/code/test.d
Detecting DRAKE_HOME......
Using DRAKE_HOME: .
Hadoop 2.6.0-cdh5.4.0
15/06/10 20:56:08 WARN security.UserGroupInformation: PriviledgedActionException as:chip (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/06/10 20:56:08 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/06/10 20:56:08 WARN security.UserGroupInformation: PriviledgedActionException as:chip (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/06/10 20:56:08 WARN security.UserGroupInformation: PriviledgedActionException as:chip (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/06/10 20:56:08 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/06/10 20:56:08 WARN security.UserGroupInformation: PriviledgedActionException as:chip (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/06/10 20:56:08 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over hp13.la.prod.factual.com/10.20.6.113:9001 after 1 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "chip-d48/10.20.10.103"; destination host is: "hp13.la.prod.factual.com":9001;
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
...

cc @BMayneFactual

[dirtyvagabond]

hi @chip-factual , which version of drake?

[chip-factual]

Drake Version 0.1.7

[mavericklou]

@chip-factual Have you successfully run any HDFS related drake file on chip-d48? I guess chip-d48 is an openstack instance that upgraded from cdh4. I had same issue on my previous dev instance, and had no problem with new instance created with cdh5 support.

[chip-factual]

@mavericklou I did successfully run HDFS related drake files back when we were still using cdh4, so it is probably a cdh5 compatibility issue. I'll see about creating a new instance to get around it.

[BMayneFactual]

fwiw, I created a new one a while back and then got blocked by #167 so I couldn't complete the experiment.

[BMayneFactual]

haven't gotten around to trying again recently