Drawn to Test

I’m a visual thinker. When collaborating with another tester, I always go to the whiteboard to draw up the system we’re testing. Pictures help me understand. Just listening to someone for ten minutes while they describe all of the pieces, communication, data flow, etc. in a system causes my brain to melt. I need to write things down to *see* what they’re describing. That’s the way my brain works. I’m probably just getting old.

Anyway, I think this is a useful process so I’ll share it.

Taking a step back, this process helps me because it engages multiple parts of my brain simultaneously. Instead of being a passive listener while someone explains a system, I’m actively engaged. My hands are drawing while my ears are listening. In drawing, I’m focused on identifying the critical pieces of information I’m being told (that’s what I draw) while ignoring extraneous details. This makes me focus on what’s important.

For those involved, drawing on the whiteboard also gets everyone on the same page. If I’m explaining software interactions to you, I’m not certain you’re understanding me. Granted, you might ask for clarification here and there, but maybe not always or you might make incorrect assumptions in your head. Putting our thoughts on the whiteboard helps ensure we’re on the same page because we can see what we’re thinking. If I draw two components and join them with an arrow indicating data flow, you can correct me if I’m wrong.

Drawing also helps identify what we don’t know. In a way, we’re teaching the system to each other while we’re drawing. There’s no fudging of information here. Lots of questions are being asked. When we can’t explain how part of the system works, we’ve identified a gap in our knowledge. Perfect. Now we know that we don’t know. Time to figure it out.

Okay, enough hand waving. Let’s go through an example using NuGet. Never heard of NuGet? No sweat. It's a package management system. Read more about if you’re interested. I’ll wait.

First, a bit of NuGet background to set the stage. If you haven’t read a bit about NuGet, none of this will make sense, but bear with me.

When you add a NuGet package to a Visual Studio project, NuGet adds the files for the package (e.g., a dll) to your project and additionally copies the files to disk at the Visual Studio solution level which is a few physical directories up the hierarchy. It’s like making a backup. But what if you don’t want the copy. Maybe your constantly forgetting to check them into source control, and when your buddy loads up your project on another machine, it fails to build because those files are missing.

Wouldn’t it be nice if NuGet was smart enough to download your packages again. Surprise! NuGet v1.6 has this feature!

When first explained to me, I mentally drew pictures, but when you’re pouring a gallon of knowledge into a shot glass of a brain, knowledge spill out.

I stopped the explanation, picked up a marker, and went to the whiteboard to produce this (yes, my penmanship is laughable):

On the whiteboard we see:

• Web applications (App A and App B) in the upper left corner that are part of the same VS solution.
• A “packages.config” file which records the NuGet packages used by an application.
• In the upper right is the local package cache (PKG CACHE) that’s queried first during the project build phase.
• In the bottom right, is a cloud representing NuGet feeds (NuGet Src). Package bits are downloaded again if they weren’t found in the package cache.

Between each of these are arrows with notes indicating operations. For example, between the web applications and the “packages.config” file we have a “read” operation. That’s Visual Studio looking up the packages used by a project. Next, Visual Studio checks the package cache to see if the package bits are available on-disk. If that fails, we download the package again.

While drawing this system, we came up with questions (see the ‘?’s) regarding functionality and design.

For example, we wondered what would happen if downloading a package required authentication (i.e., a username/pwd)? How would you prompt a user to enter these during a project's build process. Would a blocking prompt pop-up? What if the build was triggered via the command line? There’s no UI shell to work with there. Maybe we need a way to pre-define these credentials?

One question you might be asking is “Where’s the developer design spec?”. Wouldn’t that answer your questions? The answer is the classic “it depends”. We’ve become more “agile” with our development and I now see less formal design documents and specs. We use wiki’s heavily and the level of detail is sometimes low. Let’s sidestep the debate on whether this is good or bad. The bottom line is that in the absence of such documentation, drawing out the system is one way to help you identify what you know and some of what you don’t know.

Long story short, drawing the system under test is helping me raise design questions and identify interesting test scenarios at a low cost.

Still not interested in this approach? Grab a fist full of whiteboard markers, remove the caps and inhale deeply. I’m certain you’ll see my point eventually.

Unhelpful Help

I’m becoming obsessed with error messages. Working with web development frameworks and related tooling, I bump into lots of these, and as I’ve written before, it’s easy for them to go wrong. It’s also easy to brush off their imperfections and simply continue on my way. By arming myself with this checklist, I’m less likely to do that. Here’s my new error message testing checklist:

Does the error message point to the right problem?

This one’s a no-brainer. Simply put, does the error message match the error. This one points nowhere.

Is the error message out-of-date?

We often re-use error messages across multiple versions of a product, but it can become out-of-date. For example, we might have an ASP.NET error message might reference a version of IIS, but that version of IIS might differ than the one shipped with a newer OS.

Is the error message easy to read?

A smaller than usual font might be used to pack a lot of information in a small amount of space. Or maybe there’s a poor choice of formatting. For example, you have a single large paragraph that could easily be broken up into easy to read bullet points.

Is the error message content appropriate for the setting in which it’s encountered?

Stack traces are often helpful, but only in the right context. An ASP.NET error stack trace is often valuable since it tells the user how *their* code triggered an error (e.g., passing a null value). This stack trace, presented in Visual Studio, is a counterexample. Maybe it’s an exception that should have been caught and re-thrown with a user friendly error message. Anyway, you get the idea.

Should the error message include a link to more information?

If you need to display a lot of content and have limited space, you could include a link to a web page that has more details. This has an additional benefit since you can update the information web page without changing any product code. Here’s an example.

Can the error message be easily copied?

In the dialog above, it’s not easy to highlight the link for copying. The trick is to select the dialog and press CTRL+C to get the contents. Probably not obvious to users. Making the link clickable directly from the dialog would have been event better.

Does the error message provide overwhelming detail?

Instead or providing individual error messages for individual error conditions, sometimes a single error message is used for a collection of error conditions. Here’s the error message when an ASP.NET automatic database creation step fails. It covers a number of possible reasons for the failure. I’m not saying this is a bad error message, but it’s a possible example of overwhelming detail.

Is the error message ambiguous?

Looking at the error message below, which operation is it referring to? I might have specified more than one.

Is the error message localizable?

Showing the same string regardless of OS language would be a mistake. It needs to be translated. With the .NET framework, we pull the text of error messages from resource files that are localized into different languages.

Does the error message require additional context to be understood?

This one was triggered only if in a previous dialog the user targeted their local machine for the “deployment action”. Looking at this error message in isolation, the connection isn’t obvious.

Showing an error message to a co-worker unfamiliar with a feature is a great way to catch this.

So there you have it. A set of some additional checks you can use to better asses the quality of your error messages. Now if we could only prevent users from causing errors in the first place…

Oh, Poor QA

Geez Dilbert. Must we perpetuate the stereotype that QA engineers (or testers) are intellectually inferior to developers.

Here’s the strip for 10/28/11:

While I don’t doubt that there are QA jobs that require little brainpower, many folks seem quick to generalize and lump *all* QA jobs into this bucket. I even get a whiff of this when interviewing external candidates for testing positions at Microsoft.

Candidate asks - “How much time will I spend coding?”

I translate this into - “How little time will I have to spend doing inferior QA type activities?”

It makes me ill.

Marketing folks, I’ll let you defend yourselves.

Setup Testing–Execution and Verification

Hopefully you read the previous blog post which set up (no pun intended) the context of this second follow up post. If not, take a few minutes and read it before continuing here.

With your matrix of setup configurations fully defined, it’s time to turn our attention to executing the individual scenarios in an effective and efficient manner.

The first thing to do is check your set of test configurations and double check that there aren’t any in there that are invalid. For example, maybe there are some operations systems you mistakenly assumed your product would support, but it turns out that with this version, you’ve dropped support for some of the older ones.

The next big task is to prioritize the setup configurations. Assuming you have neither infinite time or resources, you need to selectively pick the configurations you can run. Here, having a good sense of your customers’ common configurations help. Hopefully you have some historical data on this or maybe it’s something you can collect it via a survey. Otherwise, you’ll just have to rely on your best guess of the likely configurations. If that’s the case, you might want to enlist a pairwise testing tool.

If you’re not familiar with the concept of pairwise testing (or all-pairs testing), it’s a way to reduce the full combinatorial matrix of your variables down to a smaller set that will likely provide “good” test coverage. It’s a cost-benefit tradeoff that might be worth exploring. Let me provide a quick example.

Imagine that your dealing with three variables for your setup test plan and each have two values:

• Operating system – Windows 7, Windows XP
• Machine architecture – 32-bit and 64-bit
• Product SKU – Basic Edition and Professional edition

A complete combinatorial matrix would produce 2 x 2 x 2 = 8 setup configurations.

A pairwise analysis of these variables would only yield 4 setup configurations, but would ensure that each pair of variables would be covered at least once. So you might get something like:

1. Windows 7, 32-bit, Basic Edition
2. Windows XP, 64-bit, Professional Edition
3. Windows 7, 64-bit, Professional Edition
4. Windows XP, 32-bit, Basic Edition

You can see that “Windows 7, 32-bit, Professional Edition” is not in the list. Breaking this configuration down in variable pairs, you’ll get coverage for “Windows 7, 32-bit” in configuration #1, “Windows 7, Professional Edition” in configuration #3, and “32-bit, Professional Edition” in configuration #4.

Here’s a list of pairwise testing tools to explore. PICT is from Microsoft.

Now, with your setup matrix streamlined and prioritized, you can finally get down to testing. The question though is “who” should do it? In my opinion, this is a great task to be offloaded to less costly resources like vendors or offshore teams. We do this in my group, and it works well, but only because we’re crystal clear and detailed in our communication. Notation is standardized so there’s no confusion on what each setup test definition means in terms of products, install/uninstall order or verification steps. If there’s a lot of back and forth between you and the other resource, you’re not going to save much time or effort.

After running a setup test, you need to verify “success”, but what is this verification process? Are you simply verifying that the setup process ran end-to-end without crashing, or is there more to check?

I hope you answered “there’s more to check”. At a high level, you should at least be evaluating the following:

1. A sanity customer scenario can be completed after the product is installed.
2. If there are other apps that might be affected by the setup, verify they haven’t been broken. We do such checks when testing SxS setups of Visual Studio. An install of the latest version shouldn’t break the previous.
3. If testing uninstall as well, verify you haven’t left any “turds” on the box. This could be files, registry entries, etc. Doing so is just sloppy and might break a future install of a new version of your product. Using tools to take a snapshot of the file system and registry for comparison purposes can help here.
4. Assess the speed of the setup install. Is it reasonable? Is there a UI providing meaningful feedback to the customer so they know it’s proceeding successfully?

Okay, so those are my thoughts on setup testing planning, execution and verification. In closing, I’ll paraphrase my primary message from the previous post. Your product is useless, if your customers can’t successfully install it on their machines. Please don’t screw this up. :-)

Mark

Setup Testing–Planning

We do a lot of setup testing at Microsoft. Given the wide distribution of our technologies across a massive matrix of user machine configurations, we can’t afford to skimp on this testing and only cover a few mainline combinations and hope for the best. We put a lot of thought and effort into setup testing.

In this first post of what I think will be a two part series, I’ll focus on setup testing planning. How do you create the test scenarios that go into this type of test plan. In a subsequent post, I’ll dive into execution and verification of the setup test plan. Now the way I approach setup testing is heavily influenced by my experiences at Microsoft, which shouldn’t be a surprise, but I hope there’s some generic points below that can be applied to almost any product.

Let me take a step back and define the purpose of setup testing. Simply, we’re trying to install a product on a machine and verify it works as expected while having no negative impact. That last part is sometimes inadvertently omitted, but you’d be embarrassed if your product’s install broke other products on the user’s machine (unless they were competitors).

Okay, let me take another quick step back (I’ll move forward soon) and describe the motivation for discussing this topic. Up until now, every time I’ve been asked to develop a setup test plan or review one, I’ve primarily relied on my memory to come up with the scenarios. This is idiotic. I’m biting the bullet here and spending a bit of time to document my approach. No more solely relying on you squishy brain.

Moving ahead, the first step in setup test planning is to define the variables that form the different configurations we need to test. We’re thinking about all of the moving parts in the system that could impact whether setup succeeds or fails.

Let me try to define a semi-structured list of variables to consider and questions to answer.

Machine configuration:

• Operating systems – Which OS’s does your product support?
• Localization – Which languages is your product localized into?
• Globalization – Which OS languages will your product install on?
• Architecture – Does your product run on both 32-bit and 64-bit machines?
• “Dirty” machines – What other bits might installed on a customer’s machine that could interfere with your products setup (e.g., aggressive anti-virus software)? Similarly, make sure you’re not *always* testing on brand new clean machines.
• Screen size – If you' have a UI based installer, run it on a monitor with a low resolution (think netbook). Does the UI scale properly or does it get cropped?
• Disk space – Run the install on a disk that doesn’t have enough free space for the install to succeed.

Product configuration:

• Versions – Have you shipped previous versions of your product? Do you plan to ship future versions beyond the current version you’re working on?
• Future versions are an interesting test scenario. You’ll want to ensure you’re not doing something today with your current setup that will cause you setup pain in the future. For example, not writing a registry key which indicates the version of your product installed. You might want a future version of your product to have this information.
  
  When we test our frameworks, we often build a “mock” vNext-Next version (that’s two versions ahead) and do setup testing with it.
• SKU’s – What are the different SKU’s (or flavors) of your product (e.g., basic, pro, ultimate, etc.)?
• Side-by-Side (SxS) – Can your product be installed SxS with other versions of your product? If so, which combinations are supported?

Install/Uninstall configuration:

• Different Installers - Is there more than one way to install your product? Maybe you have an interactive UI based installer and an automated command line installer. Or maybe different installers for different OS’s?
• Location of installer – Does it matter if the installer is run locally versus on a remote network share?
• Install location – Is this fixed for the user or can they customize it?
• We’re sometimes nailed by this because all of our test machines have the same setup of two drives and when a customer installs the product on a machine with just one drive, we bomb.
• User accounts – Does your product have to be installed by an administrator level account or will a non-admin account work as well? Can one user account install the product and another user account uninstall it?
• This scenario of one user installing and another user uninstalling is interesting for shared servers where the first user account is deleted at some point.
• Product uninstallation – Don’t forget to test uninstalling the product.

Install/Uninstall execution:

• Order of operations – If SxS configurations (see above) of your product are possible, vary the installation and uninstallation of them. For example, you could [install v1, uninstall v1, install v2] as one configuration or [install v1, install v2, uninstall v1] as a second configuration. Both should produce roughly the same result.
• Redundant operations – Test installing your product, uninstalling and then re-installing it.
• Cancellation – Abort your install and uninstall in the middle of execution via built-in functionality (e.g., typical Cancel button) and via the unexpected (e.g., reboot).
• Blocking – Are there configurations where an install or uninstall operation should be blocked? Maybe installation shouldn’t be possible if there’s a previous version of the product already on the machine, or you can’t uninstall because other products have a dependency on this one. Related to that, if the operation is blocked, is a meaningful message presented to the user instructing them on how to become unblocked?

Miscellaneous:

• Dependent products - Are you assuming that some components must already be on the box?
• I’ve seen my team make assumptions that IIS will always be installed before we try to install ASP.NET because that’s the way our test machines are set up, but there’s no guarantee of this.
• Files in use - Are there files that setup needs to access that might be locked by another process?
• Microsoft products seem to often recommend exiting all other applications before installation begins to help ensure there’s no “file in use” conflicts.
• Right bits - Make sure the build you’re testing is the one you’re shipping. If you’ve done some testing and the bits change, re-evaluate the setup test scenarios you need to re-run.

With the configuration variables defined and the various questions answered, it’s time to crank out the matrix of individual configurations to test. In a follow up post, we’ll continue with setup testing execution and verification.

Oh, one last thought:

If your setup fails, your product is worthless to customers. Take setup testing seriously. Seriously.

Okay, so what did I miss?

Mark

Time Wasted

I recently spent a few weeks (off and on) porting the Coding QA website from a proprietary closed web hosting platform to an MVC 3 application hosted on your typical budget web hosting platform. Overall it was a fun experience mixed with lots of pain and suffering (bad database connection string can nuke your site), but in the end, I learned valuable stuff. I also learned a non-technical lesson that I wanted to note down for future reference.

Mark’s Lesson of the Day – “I DON’T KNOW WHEN TO ASK FOR HELP”

Okay, so back to the upgrade of the Coding QA website. The code was written and tested so I “published” it to the web server and gave it a spin.

First check – create a dummy blog post. That required logging in with my super secret admin account. Done. Next, create a blog post, fill in some dummy data and then “Save”.

I got dumped back to my site’s the login screen.

Looking back, it was this point that I started unknowingly proceeding through The Five Stages of Grief.

1. Denial - Uh, no I didn’t just get logged out of the site when trying to create a new blog post. I repeated my steps. Same result. I fulfilled the definition of insanity by repeating the process a third and fourth time.
   
   
2. Anger - WTF is wrong with this web server’s setup!? Everything works fine on my development box. Serves me right for picking the cheapest web hosting plan possible.
   
   
3. Bargaining - Okay, maybe the situation isn’t that bad. After being logged out, if I log in again, it seems like I can sometimes get the “Save” operation to work. Painful, but at least I have a workaround although it might take a few tries.
   
   
4. Depression - Seriously? I’ve got to go through this multiple login/logout crap each time. Why, oh why did I want to move the website? I didn’t have to do this. I’m not even saving any money. I only have myself to blame for this mess. I’m a moron.
   
   
5. Acceptance - Okay, I’m just going to have to live with this. Too much time and effort have gone into porting the site. I can’t back out now. It’s full steam ahead with a steaming POS of a solution.

I then proceeded for hours trying to debug the problem using my vast limited knowledge of ASP.NET.

Okay, what are all of the possible reasons for getting unexpectedly logged out of the site?

• Um, maybe each web request is getting redirected to a different box and that new box doesn’t recognize the authorization cookie issued by the first box.
• Or, maybe there’s a problem with being redirected to the login page when I directly request a restricted page. At one point, I actually convinced myself this was the issue and thought I had a workaround by only logging in directly through the Account login page.
• Ah, there must be a problem with the expiration date set on the authorization cookie after I login. I bet my local web.config setting for this (set at 2 days) isn’t being honored.

No, none of this madness was even close. Nothing worked and I couldn’t get over the fact that I, someone who has worked on ASP.NET for 8+ years, couldn’t solve this stupid logout problem. Geez, if I couldn’t solve this, the rest of the folks using this web hosting company are screwed.

Feeling totally helpless, I made a last ditch effort and sent the web hosting company a one sentence e-mail with a subject line of “I keep getting logged out of my site”.

Their reply – “Your application might be exceeding the 100 MB memory limit”.

Uh, the what limit?

Yep, right there on their hosting plan summary page they call out the 100 MB memory limit for an application. Exceed that and your application is recycled and that logs you out.

I wasted hours trying to solve a problem that an entry level support engineer looked up in two minutes.

I don’t know when to ask for help.

Mark

Effective Integration Testing

If your test team isn’t doing effective integration testing*, you’re making a big mistake. Skipping integration testing is like testing all of the parts of a car individually, but never assembling it and taking it for a drive.

For years I felt our team gave this type of testing minimal attention, and we paid a high price for it – critical bugs found late in the development cycle, high profile demo failures, hotfixes shipped right after the product was released, etc.

Fortunately, our approach today is different. We have at least two team members (I’m one), that spend the majority of their time using Microsoft’s Web Frameworks and Tools (e.g., ASP.NET, Visual Web Developer, etc.) to build real world applications. I call this our “Dedicated Integration Testing” effort.

Let me contrast our team’s current approach with the past.

We always knew integration testing was important because of the “high cost” issues noted previously, but we never did it effectively. Here are some approaches we’ve tried and why I think they’re ineffective:

1. Told the entire team of testers to do integration testing
   • You can’t simply tell someone to “do integration” testing and expect meaningful results. Integration testing requires broad knowledge of all of the pieces in the software system. Your standard tester is probably focused on depth testing a small set of features. They won’t have this broad product knowledge, and it’s not that they couldn’t learn it. They just don’t have time to do it. Granted, some folks that have been with the team for years have been able to develop a bit of product breadth, but I’d argue it’s not enough.
     
2. Schedule the entire product team to spend a few weeks doing integration testing (that’s application building for us)
   • This activity is typically scheduled towards the end of a major product milestone. It’s on the calendar, and we all know it’s coming.  But we’ve had previous schedule delays, and now we need to cut work to get us back on track. Hmm – I wonder what’s first to get whacked?
   • Okay, let’s say we actually get to this work. Based on my past experience, the applications produced are “toy” applications. They’re not meaningful in terms of providing effective integration testing coverage. They don’t cover the interactions of all of the features in the system or they don’t exercise the pieces in a meaningful way. Going back to my car analogy, we’re starting the car but never taking it out of park.
     
3. Form a virtual integration testing team
   • Who do you pick for this? Do you randomly select a handful of folks and tell them to go focus on integration testing scenarios? If so, you’ve likely got the wrong people in this group. You’re probably going to hit the problem of a lack of broad product knowledge, and this is going to cripple your ability to produce meaningful integration test scenarios.
     
     We tried this approach a few years ago. It went poorly. Members of the team took the features they knew (i.e., a subset of what should have been covered) and crammed them together in ways they thought were realistic (i.e., not what our customer’s would do) and called this integration testing. Again, it wasn’t the team members fault. They just didn’t have the right product knowledge or customer understanding.

Here’s why I think we’re doing more effective integration testing today.

First, we don’t rely on any of the above three approaches to cover our integration testing needs. Some of those activities might still happen, but we don’t depend on them to feel confident that we’ve done sufficient integration testing.

Second, there’s two of us dedicated to integration testing, and by dedicated, I meant that we’re not distracted by most traditional tester activities. For example:

1. We don’t own individual features.
2. We don’t write test automation.
3. We don’t work on test infrastructure.
4. We don’t sit in product triage meetings.
5. We don’t manage other team members.

Third, we’re committed to integration testing. Come review time, we’re graded against these efforts. This means we’re unlikely to sucked into other activities not related to integration testing.

Forth, we spend a lot of time learning. We attend brown bags, we do pair test sessions, we participate in customer SDR’s (software design reviews), we watch training videos, we read blogs, etc. This is how we build a broad product understanding and a deep understanding of our customers.

So this is the approach our team is taking today. It took a long time to get here, and it wasn’t a trivial commitment of resources for the team to make. I thank our management team for allowing us to experiment with this approach, and I hope our customers appreciate the effort as well.

Mark

*Some folks might call what I’m describing as “systems testing”. That’s fine. The terminology isn’t critical for this post.