Skip to content

Commit

Permalink
# 优化flink 状态后端stateBack相关说明
Browse files Browse the repository at this point in the history
  • Loading branch information
lei-zuquan committed Sep 4, 2020
1 parent 2092056 commit 06a7f12
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.runtime.state.StateBackend;
import org.apache.flink.runtime.state.filesystem.FsStateBackend;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
Expand Down Expand Up @@ -35,7 +36,8 @@ public static void main(String[] args) throws Exception {
env.setRestartStrategy(RestartStrategies.fixedDelayRestart(3, 1000));

// 设置状态数据存储的后端,本地文件系统;默认:状态保存在 TaskManager 的内存中,检查点保存在 JobManager 的内存中
env.setStateBackend(new FsStateBackend("file:\\\\lei_test_project\\idea_workspace\\FlinkTutorial\\check_point_dir"));
StateBackend stateBackend = new FsStateBackend("file:\\\\lei_test_project\\idea_workspace\\FlinkTutorial\\check_point_dir");
env.setStateBackend(stateBackend);
// 生产环境将StateBackend保存到分布式文件系统
//env.setStateBackend(new FsStateBackend("hdfs://node-01:8020/user/root/sqoop/flink_state_backend"));

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ Flink 架构体系的一大特性是:有状态计算
增量计算,Failover这些机制都需要 State的支撑

存储实现
基于内存的HeapStateBackend - 在debug模式使用,不建议在生产模式下应用;
基于HDFS的FsStateBackend - 分布式文件持久化,每次读写都产生网络IO,整体性能不佳;
基于内存的 MemoryStateBackend - 在debug模式使用,不建议在生产模式下应用;
基于HDFS的 FsStateBackend - 分布式文件持久化,每次读写都产生网络IO,整体性能不佳;
基于RocksDB的RocksDBStateBackend - 本地文件 + 异步HDFS持久化;
还有一个是基于Niagara(Alibaba 内部实现) NiagaraStateBackend - 分布式持久化 - 在Alibaba生产环境应用:

Expand Down

0 comments on commit 06a7f12

Please sign in to comment.