Gentoo Development Guide

MULTIPROCESSING.ECLASS

Section: portage (5)
Updated: Dec 2014
Index Return to Main Contents
 

NAME

multiprocessing.eclass - parallelization with bash (wtf?)  

DESCRIPTION

The multiprocessing eclass contains a suite of functions that allow ebuilds to quickly run things in parallel using shell code.

It has two modes: pre-fork and post-fork. If you don't want to dive into any more nuts & bolts, just use the pre-fork mode. For main threads that mostly spawn children and then wait for them to finish, use the pre-fork mode. For main threads that do a bit of processing themselves, use the post-fork mode. You may mix & match them for longer computation loops.  

EXAMPLE

# First initialize things:
multijob_init

# Then hash a bunch of files in parallel:
for n in {0..20} ; do
        multijob_child_init md5sum data.${n} > data.${n}
done

# Then wait for all the children to finish:
multijob_finish
 

FUNCTIONS

bashpid
Return the process id of the current sub shell. This is to support bash versions older than 4.0 that lack $BASHPID support natively. Simply do: echo ${BASHPID:-$(bashpid)}

Note: Using this func in any other way than the one above is not supported.

makeopts_jobs [${MAKEOPTS}]
Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number specified therein. Useful for running non-make tools in parallel too. i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the number as bash normalizes it to [0, 255]. If the flags haven't specified a -j flag, then "1" is shown as that is the default `make` uses. Since there's no way to represent infinity, we return 999 if the user has -j without a number.
makeopts_loadavg [${MAKEOPTS}]
Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set for load-average. For make and ninja based builds this will mean new jobs are not only limited by the jobs-value, but also by the current load - which might get excessive due to I/O and not just due to CPU load. Be aware that the returned number might be a floating-point number. Test whether your software supports that.
multijob_init [${MAKEOPTS}]
Setup the environment for executing code in parallel. You must call this before any other multijob function.
multijob_child_init [--pre|--post] [command to run in background]
This function has two forms. You can use it to execute a simple command in the background (and it takes care of everything else), or you must call this first thing in your forked child process.

The --pre/--post options allow you to select the child generation mode.

# 1st form: pass the command line as arguments:
multijob_child_init ls /dev
# Or if you want to use pre/post fork modes:
multijob_child_init --pre ls /dev
multijob_child_init --post ls /dev

# 2nd form: execute multiple stuff in the background (post fork):
(
multijob_child_init
out=`ls`
if echo "${out}" | grep foo ; then
        echo "YEAH"
fi
) &
multijob_post_fork

# 2nd form: execute multiple stuff in the background (pre fork):
multijob_pre_fork
(
multijob_child_init
out=`ls`
if echo "${out}" | grep foo ; then
        echo "YEAH"
fi
) &
multijob_pre_fork
You must call this in the parent process before forking a child process. If the parallel limit has been hit, it will wait for one child to finish and return its exit status.
multijob_post_fork
You must call this in the parent process after forking a child process. If the parallel limit has been hit, it will wait for one child to finish and return its exit status.
multijob_finish_one
Wait for a single process to exit and return its exit code.
multijob_finish
Wait for all pending processes to exit and return the bitwise or of all their exit codes.
redirect_alloc_fd <var> <file> [redirection]
Find a free fd and redirect the specified file via it. Store the new fd in the specified variable. Useful for the cases where we don't care about the exact fd #.
 

AUTHORS

Brian Harring <ferringb@gentoo.org>
Mike Frysinger <vapier@gentoo.org>
 

MAINTAINERS

base-system@gentoo.org
 

REPORTING BUGS

Please report bugs via http://bugs.gentoo.org/  

FILES

/usr/portage/eclass/multiprocessing.eclass  

SEE ALSO

ebuild(5)
http://sources.gentoo.org/eclass/multiprocessing.eclass?view=log


 

Index

NAME
DESCRIPTION
EXAMPLE
FUNCTIONS
AUTHORS
MAINTAINERS
REPORTING BUGS
FILES
SEE ALSO

This document was created by man2html, using the manual pages.
Time: 03:25:03 GMT, December 21, 2014