Skip to content
/ glow Public
forked from chrislusf/glow

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

Notifications You must be signed in to change notification settings

Solertis/glow

Repository files navigation

glow

Ideally, job can be run on a big machine. For example, "sort | uniq | wc" for word count for large files. However, with super large files, doing the same thing distributedly requires lots of plumbing effort.

Glow aims to make it easy to distribute work to remote machines.

What's special?

  1. If any task can be done by unix pipe style, it can be done by Glow.
  2. Glow provides tools to distribute tasks: map by hashing, reduce by grouping.
  3. Each command is a json sent to leader. So more powerful tools can build on them.

What is the architecture?

  1. Each participating machine installs an agent talking to a leader.
  2. On any computer, user submits a job to leader.
  3. Agent picks up one role of the job, starts the process, setups the input/output. All participating agents thus forms a pipeline.
  4. user feed data into the pipeline.
  5. When all results arrived, user shuts down the pipeline, or leaves it running continuously by feeding more data.

What is the pipeline?

  1. Pipeline can flow 11, 1m, 1mn
  2. pipeline can ajust flow ratio to scale up or down 1m to 1m*n or n~m, on demand.

About

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 99.0%
  • Shell 1.0%