Gentoo Development Guide

MULTIPROCESSING.ECLASS

Section: portage (5)
Updated: Jan 2017
Index Return to Main Contents
 

NAME

multiprocessing.eclass - parallelization with bash (wtf?)  

DESCRIPTION

The multiprocessing eclass contains a suite of functions that allow ebuilds to quickly run things in parallel using shell code.

It has two modes: pre-fork and post-fork. If you don't want to dive into any more nuts & bolts, just use the pre-fork mode. For main threads that mostly spawn children and then wait for them to finish, use the pre-fork mode. For main threads that do a bit of processing themselves, use the post-fork mode. You may mix & match them for longer computation loops.  

EXAMPLE

# First initialize things:
multijob_init

# Then hash a bunch of files in parallel:
for n in {0..20} ; do
        multijob_child_init md5sum data.${n} > data.${n}
done

# Then wait for all the children to finish:
multijob_finish
 

FUNCTIONS

bashpid
Return the process id of the current sub shell. This is to support bash versions older than 4.0 that lack $BASHPID support natively. Simply do: echo ${BASHPID:-$(bashpid)}

Note: Using this func in any other way than the one above is not supported.

get_nproc [${fallback:-1}]
Attempt to figure out the number of processing units available. If the value can not be determined, prints the provided fallback instead. If no fallback is provided, defaults to 1.
makeopts_jobs [${MAKEOPTS}] [${inf:-999}]
Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number specified therein. Useful for running non-make tools in parallel too. i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the number as bash normalizes it to [0, 255]. If the flags haven't specified a -j flag, then "1" is shown as that is the default `make` uses. Since there's no way to represent infinity, we return ${inf} (defaults to 999) if the user has -j without a number.
makeopts_loadavg [${MAKEOPTS}] [${inf:-999}]
Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set for load-average. For make and ninja based builds this will mean new jobs are not only limited by the jobs-value, but also by the current load - which might get excessive due to I/O and not just due to CPU load. Be aware that the returned number might be a floating-point number. Test whether your software supports that. If no limit is specified or --load-average is used without a number, ${inf} (defaults to 999) is returned.
multijob_init [${MAKEOPTS}]
Setup the environment for executing code in parallel. You must call this before any other multijob function.
multijob_child_init [--pre|--post] [command to run in background]
This function has two forms. You can use it to execute a simple command in the background (and it takes care of everything else), or you must call this first thing in your forked child process.

The --pre/--post options allow you to select the child generation mode.

# 1st form: pass the command line as arguments:
multijob_child_init ls /dev
# Or if you want to use pre/post fork modes:
multijob_child_init --pre ls /dev
multijob_child_init --post ls /dev

# 2nd form: execute multiple stuff in the background (post fork):
(
multijob_child_init
out=`ls`
if echo "${out}" | grep foo ; then
        echo "YEAH"
fi
) &
multijob_post_fork

# 2nd form: execute multiple stuff in the background (pre fork):
multijob_pre_fork
(
multijob_child_init
out=`ls`
if echo "${out}" | grep foo ; then
        echo "YEAH"
fi
) &
multijob_pre_fork
You must call this in the parent process before forking a child process. If the parallel limit has been hit, it will wait for one child to finish and return its exit status.
multijob_post_fork
You must call this in the parent process after forking a child process. If the parallel limit has been hit, it will wait for one child to finish and return its exit status.
multijob_finish_one
Wait for a single process to exit and return its exit code.
multijob_finish
Wait for all pending processes to exit and return the bitwise or of all their exit codes.
redirect_alloc_fd <var> <file> [redirection]
Find a free fd and redirect the specified file via it. Store the new fd in the specified variable. Useful for the cases where we don't care about the exact fd #.
 

AUTHORS

Brian Harring <ferringb@gentoo.org>
Mike Frysinger <vapier@gentoo.org>
 

MAINTAINERS

base-system@gentoo.org
 

REPORTING BUGS

Please report bugs via http://bugs.gentoo.org/  

FILES

/usr/portage/eclass/multiprocessing.eclass  

SEE ALSO

ebuild(5)
https://gitweb.gentoo.org/repo/gentoo.git/log/eclass/multiprocessing.eclass


 

Index

NAME
DESCRIPTION
EXAMPLE
FUNCTIONS
AUTHORS
MAINTAINERS
REPORTING BUGS
FILES
SEE ALSO

This document was created by man2html, using the manual pages.
Time: 03:25:03 GMT, January 17, 2017