[SPARK-25266][CORE] Fix memory leak in Barrier Execution Mode

## What changes were proposed in this pull request? BarrierCoordinator uses Timer and TimerTask. `TimerTask#cancel()` is invoked in ContextBarrierState#cancelTimerTask but `Timer#purge()` is never invoked. Once a TimerTask is scheduled, the reference to it is not released until `Timer#purge()` is invoked even though `TimerTask#cancel()` is invoked. ## How was this patch tested? I checked the number of instances related to the TimerTask using jmap. Closes apache#22258 from sarutak/fix-barrierexec-oom. Authored-by: sarutak <[email protected]> Signed-off-by: Xiangrui Meng <[email protected]>
dc-aps · Aug 29, 2018 · 3864480 · 3864480
1 parent 1fd59c1
commit 3864480
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 0 deletions.
diff --git a/core/src/main/scala/org/apache/spark/BarrierCoordinator.scala b/core/src/main/scala/org/apache/spark/BarrierCoordinator.scala
@@ -123,6 +123,7 @@ private[spark] class BarrierCoordinator(
     private def cancelTimerTask(): Unit = {
       if (timerTask != null) {
         timerTask.cancel()
+        timer.purge()
         timerTask = null
       }
     }

diff --git a/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala b/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
@@ -140,6 +140,7 @@ class BarrierTaskContext(
         throw e
     } finally {
       timerTask.cancel()
+      timer.purge()
     }
   }
-Original file line number
+Diff line change
@@ Expand Up / @@ -140,6 +140,7 @@ class BarrierTaskContext( @@
             throw e
         } finally {
           timerTask.cancel()
+          timer.purge()
         }
       }
@@ Expand Down @@