Recently a site was performing a software upgrade over the weekend. No big deal, everyone thought, because this happens thousands of times across the world every day.
Yours truly was on call to support the site if any issues arose.
Needless to say they called Sunday afternoon after message traffic had gotten stuck in their queues for several hours and queue refreshes had not made a difference.
What I learned about the site was amazing. They had performed not one, not two, but three software upgrades all at once. They only backed up the system at the beginning. They were the first site to go to this release.
Well, this prompted me to draft the following list of rules.
Never install release 1.0 of anything.
Never be the first site for any release.
Always read the release notes and review the installation instructions.
Always hold a readiness review meeting to answer questions.
Never start an installation or upgrade without a backup of both your database and your system.
Never install or upgrade more than two releases at a time.
Always backup your databases before your second upgrade.
Optionally backup your system before your second upgrade.
Always backup both your database and your system after the last release.
Always test the functionality mentioned in the release notes.
Always examine all the log files for errors.
Always hold a party when the smoke clears aka the system is stable.
Now I feel a whole lot better. Creating this list was quite a catharsis.
Let me know if you agree or have something to add.
There have been products in the software industry that have been a little less than spectacular. For instance, Motif was supposed to be the new holy grail for developing graphical user interfaces (GUIs). (I know I am showing my age.) But it was slow coming up to the screen. Why? Because the Motif designers, decided to load everything into memory at start up. Even rarely used functions.
Now comes middle ware products. And guess what? The designers of those products decided to load everything into memory upon start up. You know what happened? Yep, they were slow at start up. You would think developers would learn from past mistakes.
But in addition, middle ware products were bogged down because the functions stayed in memory. That is because they were designed and tested for middle sized problems. Yes the name for these products was “Middle Sized Ware”. But marketeers decided the name limited the market for sales, so they dropped the “sized” part.
So why don’t software companies load functions into memory “as they are needed”?
So why aren’t software systems designed in case they are used by a thousand users?
It is because software companies are in a hurry and want to keep costs down. So they hire cheap, inexperienced developers.
The solution isto hire a few experienced people (not all) and take some time to design a killer app like Steve Jobs did. Nobody ever said they got stuck with an iPhone.