Skip to content

Commit

Permalink
[hotfix][docs] Fixed memory units in "Migration Guide" under "Memory …
Browse files Browse the repository at this point in the history
…Configuration" page. This closes apache#19506
JustDoDT authored and MartijnVisser committed Apr 19, 2022
1 parent 6fdc759 commit 789eb8a
Showing 6 changed files with 38 additions and 38 deletions.
4 changes: 2 additions & 2 deletions docs/content.zh/docs/deployment/memory/mem_migration.md
Original file line number Diff line number Diff line change
@@ -293,10 +293,10 @@ Flink 通过设置上述 JVM 内存限制降低内存泄漏问题的排查难度
本节描述 Flink 自带的默认 `flink-conf.yaml` 文件中的变化。

原本的 TaskManager 总内存(`taskmanager.heap.size`)被新的配置项 [`taskmanager.memory.process.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-process-size) 所取代。
默认值从 1024Mb 增加到了 1728Mb
默认值从 1024MB 增加到了 1728MB

原本的 JobManager 总内存(`jobmanager.heap.size`)被新的配置项 [`jobmanager.memory.process.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-process-size) 所取代。
默认值从 1024Mb 增加到了 1600Mb
默认值从 1024MB 增加到了 1600MB

请参考[如何配置总内存]({{< ref "docs/deployment/memory/mem_setup" >}}#configure-total-memory)。

28 changes: 14 additions & 14 deletions docs/content.zh/docs/deployment/memory/mem_setup.md
Original file line number Diff line number Diff line change
@@ -123,35 +123,35 @@ Flink 进程启动时,会根据配置的和自动推导出的各内存部分
这些内存部分的大小必须在相应的最大值、最小值范围内,否则 Flink 将无法启动。
最大值、最小值具有默认值,也可以通过相应的配置参数进行设置。
例如,如果仅配置下列参数:
- *进程总内存* = 1000Mb
- *JVM 开销最小值* = 64Mb
- *JVM 开销最大值* = 128Mb
- *进程总内存* = 1000MB
- *JVM 开销最小值* = 64MB
- *JVM 开销最大值* = 128MB
- *JVM 开销占比* = 0.1

那么 *JVM 开销*的实际大小将会是 1000Mb x 0.1 = 100Mb,在 64-128Mb 的范围内。
那么 *JVM 开销*的实际大小将会是 1000MB x 0.1 = 100MB,在 64-128MB 的范围内。

如果将最大值、最小值设置成相同大小,那相当于明确指定了该内存部分的大小。

如果没有明确指定内存部分的大小,Flink 会根据总内存和占比计算出该内存部分的大小。
计算得到的内存大小将受限于相应的最大值、最小值范围。
例如,如果仅配置下列参数:
- *进程总内存* = 1000Mb
- *JVM 开销最小值* = 128Mb
- *JVM 开销最大值* = 256Mb
- *进程总内存* = 1000MB
- *JVM 开销最小值* = 128MB
- *JVM 开销最大值* = 256MB
- *JVM 开销占比* = 0.1

那么 *JVM 开销*的实际大小将会是 128Mb,因为根据总内存和占比计算得到的内存大小 100Mb 小于最小值。
那么 *JVM 开销*的实际大小将会是 128MB,因为根据总内存和占比计算得到的内存大小 100MB 小于最小值。

如果配置了总内存和其他内存部分的大小,那么 Flink 也有可能会忽略给定的占比。
这种情况下,受限的等比内存部分的实际大小是总内存减去其他所有内存部分后剩余的部分。
这样推导得出的内存大小必须符合最大值、最小值范围,否则 Flink 将无法启动。
例如,如果仅配置下列参数:
- *进程总内存* = 1000Mb
- *任务堆内存* = 100Mb(或 JobManager 的 *JVM 堆内存*
- *JVM 开销最小值* = 64Mb
- *JVM 开销最大值* = 256Mb
- *进程总内存* = 1000MB
- *任务堆内存* = 100MB(或 JobManager 的 *JVM 堆内存*
- *JVM 开销最小值* = 64MB
- *JVM 开销最大值* = 256MB
- *JVM 开销占比* = 0.1

*进程总内存*中所有其他内存部分均有默认大小,包括 TaskManager 的*托管内存*默认占比或 JobManager 的默认*堆外内存*
因此,*JVM 开销*的实际大小不是根据占比算出的大小(1000Mb x 0.1 = 100Mb),而是*进程总内存*中剩余的部分。
这个剩余部分的大小必须在 64-256Mb 的范围内,否则将会启动失败。
因此,*JVM 开销*的实际大小不是根据占比算出的大小(1000MB x 0.1 = 100MB),而是*进程总内存*中剩余的部分。
这个剩余部分的大小必须在 64-256MB 的范围内,否则将会启动失败。
6 changes: 3 additions & 3 deletions docs/content.zh/docs/deployment/memory/mem_setup_tm.md
Original file line number Diff line number Diff line change
@@ -190,14 +190,14 @@ Flink 会负责管理网络内存,保证其实际用量不会超过配置大
| :------------------------------------------- | :---------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------ |
| 任务堆内存 | [`taskmanager.memory.task.heap.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-task-heap-size) | 无穷大 |
| 任务堆外内存 | [`taskmanager.memory.task.off-heap.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-task-off-heap-size) | 无穷大 |
| 托管内存 | [`taskmanager.memory.managed.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-managed-size) | 128Mb |
| 网络内存 | [`taskmanager.memory.network.min`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-network-max) | 64Mb |
| 托管内存 | [`taskmanager.memory.managed.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-managed-size) | 128MB |
| 网络内存 | [`taskmanager.memory.network.min`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-network-max) | 64MB |

<br/>

本地执行模式下,上面列出的所有内存部分均可以但不是必须进行配置。
如果未配置,则会采用默认值。
其中,[任务堆内存](#task-operator-heap-memory)*任务堆外内存*的默认值无穷大(*Long.MAX_VALUE* 字节),以及[托管内存](#managed-memory)的默认值 128Mb 均只针对本地执行模式。
其中,[任务堆内存](#task-operator-heap-memory)*任务堆外内存*的默认值无穷大(*Long.MAX_VALUE* 字节),以及[托管内存](#managed-memory)的默认值 128MB 均只针对本地执行模式。

<span class="label label-info">提示</span>
这种情况下,任务堆内存的大小与实际的堆空间大小无关。
4 changes: 2 additions & 2 deletions docs/content/docs/deployment/memory/mem_migration.md
Original file line number Diff line number Diff line change
@@ -279,10 +279,10 @@ The direct or native off-heap memory consumers can now be addressed by the follo
This section describes the changes of the default `flink-conf.yaml` shipped with Flink.

The total memory for TaskManagers (`taskmanager.heap.size`) is replaced by [`taskmanager.memory.process.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-process-size)
in the default `flink-conf.yaml`. The value increased from 1024Mb to 1728Mb.
in the default `flink-conf.yaml`. The value increased from 1024MB to 1728MB.

The total memory for JobManagers (`jobmanager.heap.size`) is replaced by [`jobmanager.memory.process.size`]({{< ref "docs/deployment/config" >}}#jobmanager-memory-process-size)
in the default `flink-conf.yaml`. The value increased from 1024Mb to 1600Mb.
in the default `flink-conf.yaml`. The value increased from 1024MB to 1600MB.

See also [how to configure total memory now]({{< ref "docs/deployment/memory/mem_setup" >}}#configure-total-memory).

28 changes: 14 additions & 14 deletions docs/content/docs/deployment/memory/mem_setup.md
Original file line number Diff line number Diff line change
@@ -131,34 +131,34 @@ Check also the detailed memory model for [TaskManager]({{< ref "docs/deployment/
The size of those components always has to be between its maximum and minimum value, otherwise Flink startup will fail.
The maximum and minimum values have defaults or can be explicitly set by corresponding configuration options.
For example, if you only set the following memory options:
- *total Process memory* = 1000Mb,
- *JVM Overhead min* = 64Mb,
- *JVM Overhead max* = 128Mb,
- *total Process memory* = 1000MB,
- *JVM Overhead min* = 64MB,
- *JVM Overhead max* = 128MB,
- *JVM Overhead fraction* = 0.1

then the *JVM Overhead* will be 1000Mb x 0.1 = 100Mb which is within the range 64-128Mb.
then the *JVM Overhead* will be 1000MB x 0.1 = 100MB which is within the range 64-128MB.

Notice if you configure the same maximum and minimum value it effectively fixes the size to that value.

If you do not explicitly configure the component memory, then Flink will use the fraction to calculate the memory size
based on the total memory. The calculated value is capped by its corresponding min/max options.
For example, if only the following memory options are set:
- *total Process memory* = 1000Mb,
- *JVM Overhead min* = 128Mb,
- *JVM Overhead max* = 256Mb,
- *total Process memory* = 1000MB,
- *JVM Overhead min* = 128MB,
- *JVM Overhead max* = 256MB,
- *JVM Overhead fraction* = 0.1

then the *JVM Overhead* will be 128Mb because the size derived from fraction is 100Mb, and it is less than the minimum.
then the *JVM Overhead* will be 128MB because the size derived from fraction is 100MB, and it is less than the minimum.

It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined.
In this case, the *JVM Overhead* is the rest of the total memory. The derived value still has to be within its min/max
range otherwise the configuration fails. For example, suppose only the following memory options are set:
- *total Process memory* = 1000Mb,
- *task heap* = 100Mb, (similar example can be for *JVM Heap* in the JobManager)
- *JVM Overhead min* = 64Mb,
- *JVM Overhead max* = 256Mb,
- *total Process memory* = 1000MB,
- *task heap* = 100MB, (similar example can be for *JVM Heap* in the JobManager)
- *JVM Overhead min* = 64MB,
- *JVM Overhead max* = 256MB,
- *JVM Overhead fraction* = 0.1

All other components of the *total Process memory* have default values, including the default *Managed Memory* fraction
(or *Off-heap* memory in the JobManager). Then the *JVM Overhead* is not the fraction (1000Mb x 0.1 = 100Mb), but the rest
of the *total Process memory* which will either be within the range 64-256Mb or fail.
(or *Off-heap* memory in the JobManager). Then the *JVM Overhead* is not the fraction (1000MB x 0.1 = 100MB), but the rest
of the *total Process memory* which will either be within the range 64-256MB or fail.
6 changes: 3 additions & 3 deletions docs/content/docs/deployment/memory/mem_setup_tm.md
Original file line number Diff line number Diff line change
@@ -174,15 +174,15 @@ then all components are ignored except for the following:
| :------------------------------------------- | :---------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------ |
| Task heap | [`taskmanager.memory.task.heap.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-task-heap-size) | infinite |
| Task off-heap | [`taskmanager.memory.task.off-heap.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-task-off-heap-size) | infinite |
| Managed memory | [`taskmanager.memory.managed.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-managed-size) | 128Mb |
| Network memory | [`taskmanager.memory.network.min`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-network-max) | 64Mb |
| Managed memory | [`taskmanager.memory.managed.size`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-managed-size) | 128MB |
| Network memory | [`taskmanager.memory.network.min`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-network-min) <br /> [`taskmanager.memory.network.max`]({{< ref "docs/deployment/config" >}}#taskmanager-memory-network-max) | 64MB |

<br/>

All of the components listed above can be but do not have to be explicitly configured for local execution.
If they are not configured they are set to their default values. [Task heap memory](#task-operator-heap-memory) and
*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](#managed-memory)
has a default value of 128Mb only for the local execution mode.
has a default value of 128MB only for the local execution mode.

<span class="label label-info">Note</span> The task heap size is not related in any way to the real heap size in this case.
It can become relevant for future optimizations coming with next releases. The actual JVM Heap size of the started

0 comments on commit 789eb8a

Please sign in to comment.