Skip to content

Commit

Permalink
rename JobHandler to JobExecutor
Browse files Browse the repository at this point in the history
  • Loading branch information
ponfee committed Aug 4, 2024
1 parent 2ef07dc commit 2f70486
Show file tree
Hide file tree
Showing 64 changed files with 467 additions and 495 deletions.
6 changes: 3 additions & 3 deletions README.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

A distributed task scheduling and distributed computing framework, in addition to having regular task scheduling functions, also provides: pausing/cancelling running tasks, resuming paused tasks, task splitting, failure retries, broadcast tasks, task dependencies, workflow tasks (DAG), separate deployment of managers and executors, Web management backend, and other capabilities.

Lightweight, easy to use, especially suitable for long task execution. It has good scalability, extensibility, and stability, and has been tested in production.
Lightweight and easy to use, it is particularly suitable for the execution of long tasks. Powerful and reliable, it has been tested in production environments.

## Architecture

Expand Down Expand Up @@ -68,13 +68,13 @@ disjob # Main project①
- Supervisor and Worker discover each other through the registry center, supported: Database, Redis, Consul, Nacos, Zookeeper, Etcd
- Supervisor is responsible for generating tasks and dispatching them to Worker for execution, supported: Redis, Http
- Need to specify the group, Job tasks will only be dispatched to the specified group of Workers for execution
- Provides the ability to split tasks, override the method [JobHandler#split](disjob-worker/src/main/java/cn/ponfee/disjob/worker/handle/JobSplitter.java) to split many tasks, then distributed and parallel execution
- Provides the ability to split tasks, override the method [JobExecutor#split](disjob-worker/src/main/java/cn/ponfee/disjob/worker/executor/JobSplitter.java) to split many tasks, then distributed and parallel execution
- Supports pausing and cancelling running tasks, paused tasks can be resumed for execution, failed tasks support retry
- Supports savepoint task execution snapshot, so that manually or abnormally paused tasks can be resumed from the savepoint
- If a task throw [PauseTaskException](disjob-worker/src/main/java/cn/ponfee/disjob/worker/exception/PauseTaskException.java) at executing, then will pause all instance tasks (even if dispatched other worker machine tasks)
- Supports broadcast tasks, broadcast tasks will be dispatched to all workers under the group for execution
- Supports dependencies jobs, multiple Jobs configured with dependencies will be executed in the established dependency order
- Supports DAG workflows, can configure jobHandler as a complex DAG expression, such as: A->B,C,(D->E)->D,F->G
- Supports DAG workflows, can configure jobExecutor as a complex DAG expression, such as: A->B,C,(D->E)->D,F->G
- Provides a Web management backend, job configuration, task monitoring, etc.

## [Download From Maven Central](https://central.sonatype.com/namespace/cn.ponfee)
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

一款分布式的任务调度与分布式计算框架,除了具备常规的任务调度功能外,还提供暂停/取消运行中的任务、恢复执行被暂停的任务、任务分片、失败重试、广播任务、任务依赖、工作流任务(DAG)、管理器与执行器分离部署、Web管理后台等能力。

轻量级,简单易用,特别适合长任务的执行。有较好的伸缩性、扩展性、稳定性,历经生产检验。
轻量级,简单易用,特别适合长任务的执行。功能强大,稳定可靠,历经生产检验。

## Architecture

Expand Down Expand Up @@ -68,13 +68,13 @@ disjob # 主项目①
- Supervisor与Worker通过注册中心相互发现,支持的注册中心有:Database、Redis、Consul、Nacos、Zookeeper、Etcd
- Supervisor负责生成任务,把任务派发给Worker执行,支持的任务派发方式有:Redis、Http
- 需要指定Job的分组(group),Job的任务只会派发给指定组的Worker执行
- 提供任务分片的能力,重写拆分方法[JobHandler#split](disjob-worker/src/main/java/cn/ponfee/disjob/worker/handle/JobSplitter.java)即可拆分为多个任务,实现分布式任务及并行执行
- 提供任务分片的能力,重写拆分方法[JobExecutor#split](disjob-worker/src/main/java/cn/ponfee/disjob/worker/executor/JobSplitter.java)即可拆分为多个任务,实现分布式任务及并行执行
- 支持暂停和取消运行中的任务,已暂停的任务可恢复继续执行,执行失败的任务支持重试
- 支持任务保存(Savepoint)其执行状态,让手动或异常暂停的任务能从上一次的执行状态中恢复继续执行
- 任务在执行时若抛出[PauseTaskException](disjob-worker/src/main/java/cn/ponfee/disjob/worker/exception/PauseTaskException.java),会暂停对应实例下的全部任务(包括分派到其它worker机器的任务)
- 支持广播任务,广播任务会派发给group下的所有worker执行
- 支持Job间的依赖,多个Job配置好依赖关系后便会按既定的依赖顺序依次执行
- 支持DAG工作流,可把`jobHandler`配置为复杂的DAG表达式,如:A->B,C,(D->E)->D,F->G
- 支持Job间的父子依赖,多个Job配置好依赖关系后便会按既定的依赖顺序依次执行
- 支持DAG工作流,可把`jobExecutor`配置为复杂的DAG表达式,如:A->B,C,(D->E)->D,F->G
- 支持执行中的Task在Worker(服务)发布完成后自动恢复继续执行
- 提供Web管理后台,通过界面进行作业配置,任务监控等

Expand Down Expand Up @@ -243,7 +243,7 @@ Worker接收到子任务后,会提交到框架自定义的线程池中执行

假如我们的这几台机器资源需要临时做其它的事情,想把当前的统计任务暂停一段时间。No problem!框架是支持`暂停执行中的任务`,只需要在管理后台的`任务实例`页面,找到该任务点击`暂停`按钮即可。在暂停时任务会接收到一个中断信号,收到中断信号时同样可以在代码中使用`Savepoint`保存当前的执行快照。

当其它事情处理完后,我们可以在管理后台的`任务实例`页面,找到被暂停的这个任务,点击`恢复`按钮,此时任务会从上一次保存的状态中恢复继续执行。
当其它事情执行完后,我们可以在管理后台的`任务实例`页面,找到被暂停的这个任务,点击`恢复`按钮,此时任务会从上一次保存的状态中恢复继续执行。

8. **异常中断**

Expand All @@ -253,15 +253,15 @@ Worker接收到子任务后,会提交到框架自定义的线程池中执行

现在这个质数统计的总任务已经执行完了,共10个子任务,每个子任务都统计出了它的那部分结果。Disjob能自动帮我汇总结果吗?Yes!框架提供了非常强大且方便的表达式来编排任务,如:`A->B,C,(D->E)->D,F->G`,现在我们就可以创建一个汇总任务,然后再把这两个任务编排在一起。

以下是本例质数统计中的job数据,只列了一些主要字段,其中`jobHandler`编排了这两个任务处理器(见项目源码)
以下是本例质数统计中的job数据,只列了一些主要字段,其中`jobExecutor`编排了这两个任务执行器(见项目源码)

```json5
{
"group": "app-test",
"jobName": "prime-count-dag",
"jobState": 1, // job状态:0-禁用;1-启用;
"jobType": 2, // job类型:1-常规;2-工作流(DAG);
"jobHandler": "cn.ponfee.disjob.test.handler.PrimeCountJobHandler -> cn.ponfee.disjob.test.handler.PrimeAccumulateJobHandler",
"jobExecutor": "cn.ponfee.disjob.test.executor.PrimeCountJobExecutor -> cn.ponfee.disjob.test.executor.PrimeAccumulateJobExecutor",
"jobParam": "{\"m\":1,\"n\":10000000000,\"blockSize\":100000000,\"parallel\":10}",
"triggerType": 2,
"triggerValue": "2023-09-02 18:00:00"
Expand Down
20 changes: 10 additions & 10 deletions disjob-admin/ruoyi-admin/src/main/resources/templates/main.html
Original file line number Diff line number Diff line change
Expand Up @@ -178,13 +178,13 @@ <h2 tabindex="-1" dir="auto"><a id="user-content-features" class="anchor" aria-h
<li>Supervisor与Worker通过注册中心相互发现,支持的注册中心有:Database、Redis、Consul、Nacos、Zookeeper、Etcd</li>
<li>Supervisor负责生成任务,把任务派发给Worker执行,支持的任务派发方式有:Redis、Http</li>
<li>需要指定Job的分组(group),Job的任务只会派发给指定组的Worker执行</li>
<li>提供任务分片的能力,重写拆分方法<a href="https://github.com/dromara/disjob/blob/master/disjob-core/src/main/java/cn/ponfee/disjob/core/handle/JobSplitter.java">JobHandler#split</a>即可拆分为多个任务,实现分布式任务及并行执行</li>
<li>提供任务分片的能力,重写拆分方法<a href="https://github.com/dromara/disjob/blob/master/disjob-worker/src/main/java/cn/ponfee/disjob/worker/executor/JobSplitter.java">JobExecutor#split</a>即可拆分为多个任务,实现分布式任务及并行执行</li>
<li>支持暂停和取消运行中的任务,已暂停的任务可恢复继续执行,执行失败的任务支持重试</li>
<li>支持任务保存(Savepoint)其执行状态,让手动或异常暂停的任务能从上一次的执行状态中恢复继续执行</li>
<li>任务在执行时若抛出<a href="https://github.com/dromara/disjob/blob/master/disjob-core/src/main/java/cn/ponfee/disjob/core/exception/PauseTaskException.java">PauseTaskException</a>,会暂停对应任务实例下的全部任务(包括分布在不同worker机器中的任务)</li>
<li>任务在执行时若抛出<a href="https://github.com/dromara/disjob/blob/master/disjob-worker/src/main/java/cn/ponfee/disjob/worker/exception/PauseTaskException.java">PauseTaskException</a>,会暂停对应任务实例下的全部任务(包括分布在不同worker机器中的任务)</li>
<li>支持广播任务,广播任务会派发给group下的所有worker执行</li>
<li>支持Job间的依赖,多个Job配置好依赖关系后便会按既定的依赖顺序依次执行</li>
<li>支持DAG工作流,可把<code>jobHandler</code>配置为复杂的DAG表达式,如:A-&gt;B,C,(D-&gt;E)-&gt;D,F-&gt;G</li>
<li>支持DAG工作流,可把<code>jobExecutor</code>配置为复杂的DAG表达式,如:A-&gt;B,C,(D-&gt;E)-&gt;D,F-&gt;G</li>
<li>提供Web管理后台,通过界面进行作业配置,任务监控等</li>
</ul>
<h2 tabindex="-1" dir="auto"><a id="user-content-comparison" class="anchor" aria-hidden="true" tabindex="-1" href="#comparison"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a>Comparison</h2>
Expand Down Expand Up @@ -349,9 +349,9 @@ <h2 tabindex="-1" dir="auto"><a id="user-content-user-guide" class="anchor" aria
<li>Samples项目配置文件</li>
</ol>
<ul dir="auto">
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/conf-supervisor/application-mysql.yml">Supervisor角色Mysql配置</a></li>
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/conf-supervisor/application-supervisor.yml">Supervisor角色核心配置</a></li>
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/conf-worker/application-worker.yml">Worker角色核心配置</a>(Spring-boot应用)</li>
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/disjob-samples-conf-supervisor/src/main/resources/application-mysql.yml">Supervisor角色Mysql配置</a></li>
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/disjob-samples-conf-supervisor/src/main/resources/application-supervisor.yml">Supervisor角色核心配置</a></li>
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/disjob-samples-conf-worker/src/main/resources/application-worker.yml">Worker角色核心配置</a>(Spring-boot应用)</li>
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/disjob-samples-springboot-common/src/main/resources/application-redis.yml">Redis配置</a>(Worker与Supervisor共用,使用Redis做注册中心或任务派发时需要配置)</li>
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/disjob-samples-springboot-common/src/main/resources/application-web.yml">Spring-boot Web相关配置</a>(Worker与Supervisor共用)</li>
<li><a href="https://github.com/dromara/disjob/blob/master/disjob-samples/disjob-samples-frameless-worker/src/main/resources/worker-conf.yml">Worker角色普通Java-main应用配置</a></li>
Expand Down Expand Up @@ -441,18 +441,18 @@ <h2 tabindex="-1" dir="auto"><a id="user-content-example" class="anchor" aria-hi
<ol start="7" dir="auto">
<li><strong>异常中断</strong></li>
</ol>
<p dir="auto">子任务在执行过程中若抛出框架的<a href="https://github.com/dromara/disjob/blob/master/disjob-core/src/main/java/cn/ponfee/disjob/core/exception/PauseTaskException.java">PauseTaskException</a>,则会<code>暂停</code>对应任务实例下全部的10个子任务(包括派发在不同机器中的任务)。同样如果抛出<a href="https://github.com/dromara/disjob/blob/master/disjob-core/src/main/java/cn/ponfee/disjob/core/exception/CancelTaskException.java">CancelTaskException</a>则会<code>取消</code>对应任务实例下全部的10个子任务。如果抛出其它类型的异常时,只会<code>取消</code>当前子任务,对应任务实例下其它的子任务不受影响。</p>
<p dir="auto">子任务在执行过程中若抛出框架的<a href="https://github.com/dromara/disjob/blob/master/disjob-worker/src/main/java/cn/ponfee/disjob/worker/exception/PauseTaskException.java">PauseTaskException</a>,则会<code>暂停</code>对应任务实例下全部的10个子任务(包括派发在不同机器中的任务)。同样如果抛出<a href="https://github.com/dromara/disjob/blob/master/disjob-worker/src/main/java/cn/ponfee/disjob/worker/exception/CancelTaskException.java">CancelTaskException</a>则会<code>取消</code>对应任务实例下全部的10个子任务。如果抛出其它类型的异常时,只会<code>取消</code>当前子任务,对应任务实例下其它的子任务不受影响。</p>
<ol start="8" dir="auto">
<li><strong>任务编排</strong></li>
</ol>
<p dir="auto">现在这个质数统计的总任务已经执行完了,共10个子任务,每个子任务都统计出了它的那部分结果。Disjob能自动帮我汇总结果吗?Yes!框架提供了非常强大且方便的表达式来编排任务,如:<code>A-&gt;B,C,(D-&gt;E)-&gt;D,F-&gt;G</code>,现在我们就可以创建一个汇总任务,然后再把这两个任务编排在一起。</p>
<p dir="auto">以下是本例质数统计中的job数据,只列了一些主要字段,其中<code>jobHandler</code>编排了这两个任务处理器(见项目源码)</p>
<p dir="auto">以下是本例质数统计中的job数据,只列了一些主要字段,其中<code>jobExecutor</code>编排了这两个任务执行器(见项目源码)</p>
<div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
&quot;group&quot;: &quot;app-test&quot;,
&quot;jobName&quot;: &quot;prime-count-dag&quot;,
&quot;jobState&quot;: 1, // job状态:0-禁用;1-启用;
&quot;jobType&quot;: 2, // job类型:1-常规;2-工作流(DAG);
&quot;jobHandler&quot;: &quot;cn.ponfee.disjob.test.handler.PrimeCountJobHandler -&gt; cn.ponfee.disjob.test.handler.PrimeAccumulateJobHandler&quot;,
&quot;jobExecutor&quot;: &quot;cn.ponfee.disjob.test.executor.PrimeCountJobExecutor -&gt; cn.ponfee.disjob.test.handler.PrimeAccumulateJobExecutor&quot;,
&quot;jobParam&quot;: &quot;{\&quot;m\&quot;:1,\&quot;n\&quot;:10000000000,\&quot;blockSize\&quot;:100000000,\&quot;parallel\&quot;:10}&quot;,
&quot;triggerType&quot;: 2,
&quot;triggerValue&quot;: &quot;2023-09-02 18:00:00&quot;
Expand All @@ -461,7 +461,7 @@ <h2 tabindex="-1" dir="auto"><a id="user-content-example" class="anchor" aria-hi
<span class="pl-s">"jobName"</span>: <span class="pl-s">"prime-count-dag"</span><span class="pl-kos">,</span>
<span class="pl-s">"jobState"</span>: <span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c">// job状态:0-禁用;1-启用;</span>
<span class="pl-s">"jobType"</span>: <span class="pl-c1">2</span><span class="pl-kos">,</span> <span class="pl-c">// job类型:1-常规;2-工作流(DAG);</span>
<span class="pl-s">"jobHandler"</span>: <span class="pl-s">"cn.ponfee.disjob.test.handler.PrimeCountJobHandler -&gt; cn.ponfee.disjob.test.handler.PrimeAccumulateJobHandler"</span><span class="pl-kos">,</span>
<span class="pl-s">"jobExecutor"</span>: <span class="pl-s">"cn.ponfee.disjob.test.executor.PrimeCountJobExecutor -&gt; cn.ponfee.disjob.test.handler.PrimeAccumulateJobExecutor"</span><span class="pl-kos">,</span>
<span class="pl-s">"jobParam"</span>: <span class="pl-s">"{\"m\":1,\"n\":10000000000,\"blockSize\":100000000,\"parallel\":10}"</span><span class="pl-kos">,</span>
<span class="pl-s">"triggerType"</span>: <span class="pl-c1">2</span><span class="pl-kos">,</span>
<span class="pl-s">"triggerValue"</span>: <span class="pl-s">"2023-09-02 18:00:00"</span>
Expand Down
2 changes: 1 addition & 1 deletion disjob-admin/ruoyi-disjob/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
</dependencyManagement>

<dependencies>
<!-- 引入`disjob-test`模块中的一些内置`JobHandler`用于测试,实际开发中建议去除此依赖 -->
<!-- 引入`disjob-test`模块中的一些内置`JobExecutor`用于测试,实际开发中建议去除此依赖 -->
<dependency>
<groupId>cn.ponfee</groupId>
<artifactId>disjob-test</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,10 @@
*/
@Configuration
@DbSnowflakeIdGenerator(jdbcTemplateRef = SPRING_BEAN_NAME_JDBC_TEMPLATE)
@ComponentScan("cn.ponfee.disjob.test.handler") // 加载一些测试的JobHandler,只用于demo演示使用(开发时建议删掉这行)
@JacksonDateConfigurer // 解决日期反序列化报错的问题
@EnableSupervisor // disjob-admin必须启用Supervisor角色,即:必须加@EnableSupervisor注解
@EnableWorker // 若要取消worker角色可去掉@EnableWorker注解(生产建议Supervisor与Worker分开部署,即去掉@EnableWorker注解)
@ComponentScan("cn.ponfee.disjob.test.executor") // 加载一些测试的JobExecutor,只用于demo演示使用(开发时建议删掉这行)
@JacksonDateConfigurer // 解决日期反序列化报错的问题
@EnableSupervisor // disjob-admin必须启用Supervisor角色,即:必须加@EnableSupervisor注解
@EnableWorker // 若要取消worker角色可去掉@EnableWorker注解(生产建议Supervisor与Worker分开部署,即去掉@EnableWorker注解)
public class DisjobAdminConfiguration {

}
Loading

0 comments on commit 2f70486

Please sign in to comment.