Building Data Intensive Apps

There’s a right tool and lots of wrong tools for every job. A web server is definitely the wrong tool for long-running processes such as nightly/weekly/monthly batch processes or other processes which take a while to complete.

However, the web is the mode of choice for doing just about everything. So scheduling and viewing results of long running processes is ideal (or at least can be ideal) for a web environment.

So the question is, how do you connect the process that do the heavy-lifting in the background with the web environment.

I typically schedule cron processes to run in the background and leave some type of information, like a “completed_at” field in the database. A null value in this field indicates that it hasn’t been completed and needs to be run. I often handle locking (to ensure that no more than one of each task is running at a time) by touching a file when the process starts and removing it when the process completes.

I’d be interested to hear what languages, tools and techniques you use in similar situations.

Are there any languages, tools and technique that you would definitely NOT use?

Thanks.