Gentoo Development Guide

MULTIPROCESSING.ECLASS

Section: portage (5)
Updated: May 2013
Index Return to Main Contents

NAME

multiprocessing.eclass - parallelization with bash (wtf?)

DESCRIPTION

The multiprocessing eclass contains a suite of functions that allow ebuilds to quickly run things in parallel using shell code.

It has two modes: pre-fork and post-fork. If you don't want to dive into any more nuts & bolts, just use the pre-fork mode. For main threads that mostly spawn children and then wait for them to finish, use the pre-fork mode. For main threads that do a bit of processing themselves, use the post-fork mode. You may mix & match them for longer computation loops.

EXAMPLE

# First initialize things:
multijob_init

# Then hash a bunch of files in parallel:
for n in {0..20} ; do
        multijob_child_init md5sum data.${n} > data.${n}
done

# Then wait for all the children to finish:
multijob_finish

FUNCTIONS

makeopts_jobs [${MAKEOPTS}]

Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number specified therein. Useful for running non-make tools in parallel too. i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the number as bash normalizes it to [0, 255]. If the flags haven't specified a -j flag, then "1" is shown as that is the default `make` uses. Since there's no way to represent infinity, we return 999 if the user has -j without a number.

multijob_init [${MAKEOPTS}]

Setup the environment for executing code in parallel. You must call this before any other multijob function.

multijob_child_init [--pre|--post] [command to run in background]

This function has two forms. You can use it to execute a simple command in the background (and it takes care of everything else), or you must call this first thing in your forked child process.

The --pre/--post options allow you to select the child generation mode.

# 1st form: pass the command line as arguments:
multijob_child_init ls /dev
# Or if you want to use pre/post fork modes:
multijob_child_init --pre ls /dev
multijob_child_init --post ls /dev

# 2nd form: execute multiple stuff in the background (post fork):
(
multijob_child_init
out=`ls`
if echo "${out}" | grep foo ; then
        echo "YEAH"
fi
) &
multijob_post_fork

# 2nd form: execute multiple stuff in the background (pre fork):
multijob_pre_fork
(
multijob_child_init
out=`ls`
if echo "${out}" | grep foo ; then
        echo "YEAH"
fi
) &

multijob_pre_fork

You must call this in the parent process before forking a child process. If the parallel limit has been hit, it will wait for one child to finish and return its exit status.

multijob_post_fork

You must call this in the parent process after forking a child process. If the parallel limit has been hit, it will wait for one child to finish and return its exit status.

multijob_finish_one

Wait for a single process to exit and return its exit code.

multijob_finish

Wait for all pending processes to exit and return the bitwise or of all their exit codes.

redirect_alloc_fd <var> <file> [redirection]

Find a free fd and redirect the specified file via it. Store the new fd in the specified variable. Useful for the cases where we don't care about the exact fd #.

AUTHORS

Brian Harring <ferringb@gentoo.org>
Mike Frysinger <vapier@gentoo.org>

MAINTAINERS

base-system@gentoo.org

REPORTING BUGS

Please report bugs via http://bugs.gentoo.org/

FILES

/usr/portage/eclass/multiprocessing.eclass

This document was created by man2html, using the manual pages.
Time: 03:25:01 GMT, May 23, 2013