Skip to content

Commit

Permalink
[FLINK-4094] [core] Add a warning to only use off heap memory with pr…
Browse files Browse the repository at this point in the history
…e-allocation=true

This closes apache#2336
  • Loading branch information
Ramkrishna authored and StephanEwen committed Aug 5, 2016
1 parent 8d4a64d commit 119364c
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 1 deletion.
2 changes: 1 addition & 1 deletion docs/setup/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ The default fraction for managed memory can be adjusted using the `taskmanager.m

- `taskmanager.memory.segment-size`: The size of memory buffers used by the memory manager and the network stack in bytes (DEFAULT: 32768 (= 32 KiBytes)).

- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false)
- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false). When `taskmanager.memory.off-heap` is set to `true`, then it is advised that this configuration is also set to `true`. If this configuration is set to `false` cleaning up of the allocated offheap memory happens only when the configured JVM parameter MaxDirectMemorySize is reached by triggering a full GC.

### Memory and Performance Debugging

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@
import org.apache.flink.core.memory.MemorySegment;
import org.apache.flink.core.memory.MemoryType;
import org.apache.flink.util.MathUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
* The memory manager governs the memory that Flink uses for sorting, hashing, and caching. Memory
Expand All @@ -53,6 +55,7 @@
*/
public class MemoryManager {

private static final Logger LOG = LoggerFactory.getLogger(MemoryManager.class);
/** The default memory page size. Currently set to 32 KiBytes. */
public static final int DEFAULT_PAGE_SIZE = 32 * 1024;

Expand Down Expand Up @@ -163,6 +166,10 @@ public MemoryManager(long memorySize, int numberOfSlots, int pageSize,
this.memoryPool = new HeapMemoryPool(memToAllocate, pageSize);
break;
case OFF_HEAP:
if(!preAllocateMemory) {
LOG.warn("It is advisable to set 'taskmanager.memory.preallocate' to true when" +
" the memory type 'taskmanager.memory.off-heap' is set to true.");
}
this.memoryPool = new HybridOffHeapMemoryPool(memToAllocate, pageSize);
break;
default:
Expand Down

0 comments on commit 119364c

Please sign in to comment.