Marco.org

Warning to non-geek readers: You probably want to skip this post.

I have an 8-core Mac Pro, and I sometimes have a big list of CPU-intensive commands I want to run (image resizing with ImageMagick, video/audio encoding with ffmpeg, etc.), but they’re usually single-threaded or just very bad at using multiple CPU cores.

I assumed that there must be some clever little shell utility to run 8 of these commands at once from a big list until the batch is done, effectively utilizing my 8 cores.

I couldn’t find such a utility, so I wrote it myself. And I’m sharing it with you, because there’s a chance you’ll find it useful, too. (Even if it already exists, at least I got to learn how to use the pthread library and flex my atrophied C muscles.)

Here’s the C source code and documentation. It’s only 57 lines of code (preceded by 22 lines of documentation).

Works at least on OS X 10.5+, CentOS, and Debian. Probably works with any Linux or BSD system. (In theory, it should work with anything with pthreads.)

Usage:

parallelize {thread-count} < big_list_of_commands.txt

Demo:

(for i in 1 2 3 4 5 6 7 8 9 10; do echo "echo 'Delay $i'; sleep $i ; \
echo 'Done $i'" ; done) | parallelize 4

Let me know if you find bugs (especially parallelism bug potential), or if you just find it useful and want to say hi.