Parallel::ForkManager is used for operations that you would like to do in parallel. Typical use is a downloader which could be retrieving hundreds and/or thousands of files.
Parallel::ForkManager, as its name suggests, uses fork to handle parallel
processing instead of threads. If you've used fork before, you're aware
that you need to be responsible for managing (i.e. cleaning up) the
processes that were created as a result of the fork.
Parallel::ForkManager handles this for you such that you start and
finish without having to worry about child processes along
the way. Further, Parallel::ForkManager provides useful callbacks
that you can use when a child starts and/or finishes, or while you're
waiting for a child to complete.
The code for a downloader that uses Net::HTTP would look like this:
require "rubygems"
require "net/http"
require "forkmanager"
my_urls = %w(url1 url2 urlN)
max_proc = 30
my_timeout = 5
pm = Parallel::ForkManager.new(max_proc)
my_urls.each do |my_url|
pm.start(my_url) && next # blocks until new fork slot is available
# doing stuff here with my_url will be in a child
url = URI.parse(my_url)
begin
http = Net::HTTP.new(url.host, url.port)
http.open_timeout = http.read_timeout = my_timeout
res = http.get(url.path)
status = res.code
if status.to_i != 200
print "Cannot get #{url.path} from #{url.host}!\n"
pm.finish(255)
else
pm.finish(0)
end
rescue Timeout::Error, Errno::ECONNREFUSED #> e
print "*** ERROR: #{my_url}: #{e.message}!\n"
pm.finish(255)
end
end
pm.wait_all_children
First you need to instantiate the ForkManager with the new constructor.
You must specify the maximum number of processes to be created. If you
specify 0, then no fork will be done; this is good for debugging purposes.
Next, use pm.start to do the fork. pm.start returns nil in the child
process, and child pid in the parent process. The && next skips the internal
loop in the parent process. Note: pm.start dies if the fork fails.
pm.finish terminates the child process (assuming a fork was done in the
start).
Note: You cannot use pm.start if you are already in the child process.
If you want to manage another set of subprocesses in the child process,
you must instantiate another Parallel::ForkManager object!
Parallel::ForkManager is a Ruby port of Perl Parallel::ForkManager 1.14. It was originally ported from Perl Parallel::ForkManager 0.7.5 but was recently updated to integrate features implemented in Perl Parallel::ForkManager versions 0.7.6 - 1.14. Bug reports and feature requests are always welcome.
Do not use Parallel::ForkManager in an environment where other child
processes can affect the run of the main program, so using this module
is not recommended in an environment where fork / wait is already used.
If you want to use more than one copy of the Parallel::ForkManager then you have to make sure that all children processes are terminated - before you use the second object in the main program.
You are free to use a new copy of Parallel::ForkManager in the child processes, although I don't think it makes sense.
Nathan Patwardhan [email protected]
Copyright (c) 2008 - 2020 Nathan Patwardhan
Distributes under the same terms as Ruby
Nathan Patwardhan [email protected], based on Perl Parallel::ForkManager documentation by Noah Robin [email protected] and dLux [email protected].
- dLux [email protected] (author, original Perl module)
- Gábor Szabó [email protected] (co-maintainer)
- Michael Gang (bug report)
- Noah Robin [email protected] (documentation tweaks)
- Chuck Hirstius [email protected] (callback exit status, original Perl example)
- Grant Hopwood [email protected] (win32 port)
- Mark Southern [email protected] (bugfix)
- Ken Clarke www.perlprogrammer.net (data structure retrieval)
- Robert Klemme [email protected], David A. Black [email protected] (general awesomeness)
- Roger Pack [email protected] (bugfix, fork semantics in start, doc changes)
- Mike Stok [email protected] (test cases, percussion, backing vocals)
- Akinori MUSHA email@redacted