Skip to content

Commit

Permalink
[SPARK-2144] ExecutorsPage reports incorrect # of RDD blocks
Browse files Browse the repository at this point in the history
This is reproducible whenever we drop a block because of memory pressure.

This is because StorageStatusListener actually never removes anything from the block maps of its StorageStatuses. Instead, when a block is dropped, it sets the block's storage level to `StorageLevel.NONE`, when it should just remove it from the map.

This PR includes this simple fix.

Author: Andrew Or <[email protected]>

Closes apache#1080 from andrewor14/ui-blocks and squashes the following commits:

fcf9f1a [Andrew Or] Remove BlockStatus if it is no longer cached
  • Loading branch information
andrewor14 authored and pwendell committed Jun 17, 2014
1 parent 23a12ce commit 09deb3e
Showing 1 changed file with 5 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,11 @@ class StorageStatusListener extends SparkListener {
val filteredStatus = storageStatusList.find(_.blockManagerId.executorId == execId)
filteredStatus.foreach { storageStatus =>
updatedBlocks.foreach { case (blockId, updatedStatus) =>
storageStatus.blocks(blockId) = updatedStatus
if (updatedStatus.storageLevel == StorageLevel.NONE) {
storageStatus.blocks.remove(blockId)
} else {
storageStatus.blocks(blockId) = updatedStatus
}
}
}
}
Expand Down

0 comments on commit 09deb3e

Please sign in to comment.