Every AIX system administrator knows how important a mksysb backup is. It’s such an essential backup that when you do need to use it, you want the operating system recovery to be quick and easy.
When a customer of mine found that their mksysb backup was “taking ages”, I put on my detective coat and hat, took my magnifying glass, and started my investigation.
How Long is “Ages”?
I could immediately see what the customer meant by "ages." The mksysb backup was taking several hours; for a small root volume group, I would have expected a backup to take about 10 minutes.
My first step was finding out what the mksysb command was actually backing up. By default, the whole of the root volume group is backed up. If rootvg is a bit cluttered, however, there are quite a few ways you can clean it up. I’ve written about cleaning up rootvg in other articles; see the Find Out More section for links to them. I like to keep my rootvg lean and clean, so typically, I won’t have data file systems in it. I much prefer to keep data file systems in their own volume groups, such as datavg.
Next, I checked that the exclude flag was in use. When you call mksysb with a –e, you exclude the files or directories that are listed in /etc/exclude.rootvg. From my experience, the exclude list is worth revising from time to time, especially if there has been a lot of changes on a system from upgrades or new file systems. Sometimes it’s just too tempting to find some spare space in rootvg, and before you know it, the mksysb backup is taking a whole lot longer to run. I have seen database administrators copy the entire production database into a file system on rootvg, and then never go back to clean it up.
Pizza at Midnight
After doing these checks, I ran the mksysb command and after 15 minutes, it had still only backed up 4000 files out of 80,000. I was faced with that perpetual dilemma we face in IT: do you kill the command, hopefully improve something, and start again, or do you just wait? There’s no blanket rule here. However, I do find that over the years, those of us who decided at 5:00 pm to wait around 20 minutes for a slow job to finish will usually end up ordering pizza at midnight.
So I killed off the mksysb backup and ran it again, this time with the –v flag. This puts the mksysb in verbose mode, making mksysb list the files that are being backed up as they go.
As it turned out, the slow mksysb backup had nothing to do with some huge file that took “ages” to back up. Instead, the mksysb was being written across an NFS mount, and there was a problem with the network configuration that the network gurus were looking at. I created the mksysb backup again, this time writing to a local file system with the aim of getting the backup to some other safe place without using the NFS mount. This worked nicely and the backup finished in about 7 minutes.
You may not need to use the verbose flag (–v) every time you create a mksysb, but it’s useful to know that the flag is available when you need it. In a perfect world, you’d never need to recover your operating system from a mksysb backup, but if it's necessary, time is of the essence. It pays to tidy up your rootvg if it’s getting a little cluttered, and it’s also very worthwhile reviewing your mksysb backup options as well as running a restore test. You'll never know when you might need it.