Skip to content

Commit

Permalink
Secondary Sort suggestion to reduce memory footprint at reducer (link…
Browse files Browse the repository at this point in the history
  • Loading branch information
pralabhkumar authored and akshayrai committed Jan 10, 2018
1 parent a208c31 commit 7a27a3f
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions app/views/help/mapreduce/helpMemory.scala.html
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ <h4>Suggestions</h4>
<br>
For example, see if there is any mapper/reducer input data skew (some task processes larger input than others).
<br>
Also check if you are doing memory intensive operation at reducer side for e.g. count distinct. These operations can be rewritten using Secondary Sort concept (sort the values), which will reduce the memory consumption
<br>
<strong>This is a new heuristic. We are still working on that. Any feedback is welcomed!</strong>
<br>

Expand Down

0 comments on commit 7a27a3f

Please sign in to comment.