Recently a site was performing a software upgrade over the weekend.  No big deal, everyone thought, because this happens thousands of times across the world every day.

Yours truly was on call to support the site if any issues arose.

Needless to say they called Sunday afternoon after message traffic had gotten stuck in their queues for several hours and queue refreshes had not made a difference.

What I learned about the site was amazing.  They had performed not one, not two, but three software upgrades all at once.  They only backed up the system at the beginning.  They were the first site to go to this release.

Well, this prompted me to draft the following list of rules.

1. Never install release 1.0 of anything.
2. Never be the first site for any release.
3. Always read the release notes and review the installation instructions.
4. Always hold a readiness review meeting to answer questions.
5. Never start an installation or upgrade without a backup of both your database and your system.
6. Never install or upgrade more than two releases at a time.
7. Always backup your databases before your second upgrade.
8. Optionally backup your system before your second upgrade.
9. Always backup both your database and your system after the last release.
10.  Always test the functionality mentioned in the release notes.
11.  Always examine all the log files for errors.
12.  Always hold a party when the smoke clears aka the system is stable.

Now I feel a whole lot better.  Creating this list was quite a catharsis.

Let me know if you agree or have something to add.