You really nailed it. Yes, yes and yes! This is the sort of stupid batch task th... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

mahmud on July 10, 2010 | parent | context | favorite | on: Batch Processing Millions and Millions of Images

You really nailed it. Yes, yes and yes!

This is the sort of stupid batch task that can be done with a shell-script using all the machines in the office after hours. Cut out the middle ware, cut down the threads and fancy IPC, and hunker down with good ole minimal number of processes on a bunch of machines.

Heck, they might even see a big win if they use a different "server" memory management algorithm that's more tailored to batch, not-so-responsive applications.

gaius on July 10, 2010 [–]

Barely even a shell script; you could do it in a one-liner. Let's say you had 16000 images and 16 cores,

    $ cat filelist.txt|xargs -n 1000 -P 16 ./myconvertprog

I do "batch" compression like this all the time.

mleonhard on July 11, 2010 | [–]

Thanks for pointing out the '-P' argument for xargs!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact