A colleague recently asked me about an issue involving crontab commands that seem to disappear:
“Have you had issues where cron doesn't run scripts?”
If you've had these issues, as well, here are a few simple steps you can take to troubleshoot the cron table.
The Cron in a Nutshell
First, here’s a quick review of the cron table syntax. You can set it up using the crontab command.
minute hour day_of_month month weekday command
Using this syntax, a job that runs at 6:30 am would begin with
If it can run any day of the month and every month of the year, you’d put in an asterisk for those two fields:
30 6 * *
Now, for the interesting part: the weekday is specified by a single digit with 0 representing Sunday, 1 representing Monday, and so on through the number 6 for each day of the week. Here’s a job that runs Tuesday through Saturday at 6:30 am:
30 6 * * 2-6
Of course, there’s the actual job itself that needs to run. This is typically a shell script, but it could be a single command or some other executable. It's always a good idea to specify the full path to the command or script.
30 6 * * 2-6 /usr/local/bin/myscript.sh
You can redirect the standard output from the script to a file:
30 6 * * 2-6 /usr/local/bin/myscript.sh > /tmp/myscript.out
If you want to capture the errors as well, they can go to their own file by using the syntax 2>, just like this:
30 6 * * 2-6 /usr/local/bin/myscript.sh > /tmp/myscript.out 2>/tmp/myscript.err
This works well so far, if you can get your head around the weird syntax.
Now, back to my colleague’s problem with the cron.
You see that the output from the script is going to one file, and any errors are going to a different file, so those two files would be good places to check for the output. But what if the job never got submitted in the first place?
Suppose you put into the cron a script or command that can’t be found, that has permissions, or that has other problems. You may not have any output in the script output file or any log file for the script, but you can still check the output for the cron job itself. You can do this by looking at the cron log file. The cron log file is in /varr/adm/cron/log and is a text file by default on AIX, but this file can get pretty large if you have thousands of jobs that are scheduled regularly.
Here’s the last line of the cron log file:
root : CMD ( /usr/sbin/dumpctrl -k >/dev/null 2>/dev/null ) : PID ( 6226166 ) : Wed Feb 12 10:05:00 2014
Cron Job with pid: 6226166 Successful
As you can see, this shows the user (root), the command, and the process ID (PID), as well as the date the job was submitted. The standard output is getting discarded by being redirected to /dev/null, and the standard error is also being discarded, as you can see from the "2>/dev/null" in the first line.
Let’s Get it Wrong
Here’s a non-existent script that I’ve set up in the crontab:
18 10 * * 3 /usr/local/bin/wrong_script.sh
The cron log file shows that the job failed:
root : CMD ( /usr/local/bin/wrong_script.sh ) : PID ( 7602276 ) : Wed Feb 12 10:18:00 2014
Cron Job with pid: 7602276 Failed
This script could've failed for any number of reasons. There are two commonly seen reasons for a failed script, however; either the script wasn't found, or the permissions weren't correct.
Of course, if the job is submitted successfully and still doesn’t produce the required output, you can try running the command interactively, or do some debugging.
It’s always worth considering alternatives to setting up a job in the crontab, if possible. For example, many applications have quite elaborate job scheduling that will allow you to manage and monitor regular tasks. You may also have access to a job scheduling system.
Another trick that you can try is running the job as an at job. This is simply a one-off job that you can set to run immediately by setting the pipe key to “at now”:
echo “/some_dir/myscript.sh” | at now
You can set the job to run at a later stage, such as “at 6 pm” or “at 10:45 am Tue”. The at command is a good way to troubleshoot scheduled jobs because it doesn’t clutter the cron table when you’re just doing some testing.
However you go about setting up scheduled jobs, it’s worth knowing where to look for the output. It can save you a lot of time in the long run.