[java] Fix handling of SERVICE_UNAVAILABLE errors

When the Java client sends a write, and it is rejected due to, e.g., a
soft memory limit error, it causes the client to invalidate the leader's
location for the tablet. The client then has to do a GetTableLocations
lookup on the master to refresh the cache. This is silly because the
location is valid and the leader is where the write must go. This patch
corrects the handling of certain RPC-level errors including the handling
of SERVICE_UNAVAILABLE. It will reduce greatly the number of unnecessary
calls to the master. These usually aren't a problem because master
lookups are fast, but it's a waste of time and resources nonetheless.

Change-Id: Id3437c779322e756a6e1fbc19f464f765741d58b
Reviewed-on: http://gerrit.cloudera.org:8080/13308
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <[email protected]>
Reviewed-by: Andrew Wong <[email protected]>

java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java

@@ -2063,7 +2063,8 @@ public void run(final Timeout timeout) {
    * Remove the tablet server from the RemoteTablet's locations. Right now nothing is removing
    * the tablet itself from the caches.
    */
-  private void invalidateTabletCache(RemoteTablet tablet, ServerInfo info,
+  private void invalidateTabletCache(RemoteTablet tablet,
+                                     ServerInfo info,
                                      String errorMessage) {
     final String uuid = info.getUuid();
     LOG.info("Invalidating location {} for tablet {}: {}",

java/kudu-client/src/main/java/org/apache/kudu/client/RpcProxy.java

@@ -409,11 +409,21 @@ private static void failOrRetryRpc(AsyncKuduClient client,
         .build());
 
     RemoteTablet tablet = rpc.getTablet();
-    // Note As of the time of writing (03/11/16), a null tablet doesn't make sense, if we see a null
-    // tablet it's because we didn't set it properly before calling sendRpc().
+    // Note: As of the time of writing (03/11/16), a null tablet doesn't make sense, if we see a
+    // null tablet it's because we didn't set it properly before calling sendRpc().
     if (tablet == null) {  // Can't retry, dunno where this RPC should go.
       rpc.errback(exception);
+      return;
+    }
+    if (exception instanceof InvalidAuthnTokenException) {
+      client.handleInvalidAuthnToken(rpc);
+    } else if (exception instanceof InvalidAuthzTokenException) {
+      client.handleInvalidAuthzToken(rpc, exception);
+    } else if (exception.getStatus().isServiceUnavailable()) {
+      client.handleRetryableError(rpc, exception);
     } else {
+      // If we don't really know anything about the exception, invalidate the location for the
+      // tablet, opening the possibility of retrying on a different server.
       client.handleTabletNotFound(rpc, exception, connection.getServerInfo());
     }
   }