We do a lot of setup testing at Microsoft. Given the wide distribution of our technologies across a massive matrix of user machine configurations, we can’t afford to skimp on this testing and only cover a few mainline combinations and hope for the best. We put a lot of thought and effort into setup testing.
In this first post of what I think will be a two part series, I’ll focus on setup testing planning. How do you create the test scenarios that go into this type of test plan. In a subsequent post, I’ll dive into execution and verification of the setup test plan. Now the way I approach setup testing is heavily influenced by my experiences at Microsoft, which shouldn’t be a surprise, but I hope there’s some generic points below that can be applied to almost any product.
Let me take a step back and define the purpose of setup testing. Simply, we’re trying to install a product on a machine and verify it works as expected while having no negative impact. That last part is sometimes inadvertently omitted, but you’d be embarrassed if your product’s install broke other products on the user’s machine (unless they were competitors).
Okay, let me take another quick step back (I’ll move forward soon) and describe the motivation for discussing this topic. Up until now, every time I’ve been asked to develop a setup test plan or review one, I’ve primarily relied on my memory to come up with the scenarios. This is idiotic. I’m biting the bullet here and spending a bit of time to document my approach. No more solely relying on you squishy brain.
Moving ahead, the first step in setup test planning is to define the variables that form the different configurations we need to test. We’re thinking about all of the moving parts in the system that could impact whether setup succeeds or fails.
Let me try to define a semi-structured list of variables to consider and questions to answer.
- Operating systems – Which OS’s does your product support?
- Localization – Which languages is your product localized into?
- Globalization – Which OS languages will your product install on?
- Architecture – Does your product run on both 32-bit and 64-bit machines?
- “Dirty” machines – What other bits might installed on a customer’s machine that could interfere with your products setup (e.g., aggressive anti-virus software)? Similarly, make sure you’re not *always* testing on brand new clean machines.
- Screen size – If you' have a UI based installer, run it on a monitor with a low resolution (think netbook). Does the UI scale properly or does it get cropped?
- Disk space – Run the install on a disk that doesn’t have enough free space for the install to succeed.
- Versions – Have you shipped previous versions of your product? Do you plan to ship future versions beyond the current version you’re working on?
- Future versions are an interesting test scenario. You’ll want to ensure you’re not doing something today with your current setup that will cause you setup pain in the future. For example, not writing a registry key which indicates the version of your product installed. You might want a future version of your product to have this information.
When we test our frameworks, we often build a “mock” vNext-Next version (that’s two versions ahead) and do setup testing with it.
- SKU’s – What are the different SKU’s (or flavors) of your product (e.g., basic, pro, ultimate, etc.)?
- Side-by-Side (SxS) – Can your product be installed SxS with other versions of your product? If so, which combinations are supported?
- Different Installers - Is there more than one way to install your product? Maybe you have an interactive UI based installer and an automated command line installer. Or maybe different installers for different OS’s?
- Location of installer – Does it matter if the installer is run locally versus on a remote network share?
- Install location – Is this fixed for the user or can they customize it?
- We’re sometimes nailed by this because all of our test machines have the same setup of two drives and when a customer installs the product on a machine with just one drive, we bomb.
- User accounts – Does your product have to be installed by an administrator level account or will a non-admin account work as well? Can one user account install the product and another user account uninstall it?
- This scenario of one user installing and another user uninstalling is interesting for shared servers where the first user account is deleted at some point.
- Product uninstallation – Don’t forget to test uninstalling the product.
- Order of operations – If SxS configurations (see above) of your product are possible, vary the installation and uninstallation of them. For example, you could [install v1, uninstall v1, install v2] as one configuration or [install v1, install v2, uninstall v1] as a second configuration. Both should produce roughly the same result.
- Redundant operations – Test installing your product, uninstalling and then re-installing it.
- Cancellation – Abort your install and uninstall in the middle of execution via built-in functionality (e.g., typical Cancel button) and via the unexpected (e.g., reboot).
- Blocking – Are there configurations where an install or uninstall operation should be blocked? Maybe installation shouldn’t be possible if there’s a previous version of the product already on the machine, or you can’t uninstall because other products have a dependency on this one. Related to that, if the operation is blocked, is a meaningful message presented to the user instructing them on how to become unblocked?
- Dependent products - Are you assuming that some components must already be on the box?
- I’ve seen my team make assumptions that IIS will always be installed before we try to install ASP.NET because that’s the way our test machines are set up, but there’s no guarantee of this.
- Files in use - Are there files that setup needs to access that might be locked by another process?
- Microsoft products seem to often recommend exiting all other applications before installation begins to help ensure there’s no “file in use” conflicts.
- Right bits - Make sure the build you’re testing is the one you’re shipping. If you’ve done some testing and the bits change, re-evaluate the setup test scenarios you need to re-run.
With the configuration variables defined and the various questions answered, it’s time to crank out the matrix of individual configurations to test. In a follow up post, we’ll continue with setup testing execution and verification.
Oh, one last thought:
If your setup fails, your product is worthless to customers. Take setup testing seriously. Seriously.
Okay, so what did I miss?