Managing Product Development

I have a Stickyminds column up, Does Exploratory Testing Have A Place On Agile Teams? The column arose out of an email conversation I had with Jon Bach. Please leave comments there.

I like Roy’s analogy about the difference between unit and integration testing: Unit testing vs. Integration Testing : The Restaurant Analogy.

Michael Hunter, the Braidy Tester, and one of the smartest testers I know, interviewed me on his DDJ blog. To be honest, I emailed him and asked if he’d like to interview me I had things to say about testing. The interview is here. You can’t leave comments there, so please leave them here.

I taught a one-day workshop at StarEast yesterday with Esther. I was astonished at the number of test managers who think testing is a service.

Effective testing is not a service. Effective testing is an integral part of development. When people–especially senior management–consider testing a service, there are inevitable consequences:

• Testers multitask between several projects, learning none of them in detail and only cursorily testing any of them. It’s hard to see the value in that kind of testing.
• Few managers make the decision about what project is most important, so ordering projects by value or risk doesn’t happen until the project is in test.
• Testers don’t work with developers, so defect reports look more like blaming the developers rather than feedback to the developers.
• Because testing is a service, project managers and developers tend to throw the product over the wall to test. Instead of collaboration, the project is in an us vs. them dynamic. Too few developers make the extra effort to find all their issues before the testers do. Why should they? There’s nothing in it for them.

This perception of testing as a service is a misunderstanding of the dynamics of software development.

When you contrast testing as a part of the development team, developers are much more likely to form partnerships with testers, to clean up their code as much as possible, and to exhibit professional pride in their work. They take defect reports as feedback.

If you’re a project manager don’t treat testing as a service. Make it an integral part of development.

Labels: project management, testing

In Architects Must Write Code, several architects responded that I was too prescriptive (I’m summarizing their comments). Maybe. But I don’t think so.

I’m in a nice hotel, where things just don’t work completely right. Yes, the hotel is clean (that’s the big thing with me). The hotel upgraded me to a suite with an oval bathtub. Clearly, people of normal height and weight had not tried to shower here–the tub is about 9 inches (a hand-width) from the toilet. There’s no grab bar to balance on one foot while stepping into or out of the tub/shower. I tried to draw you a picture here.

I travel a lot and stay in lots of hotels. This problem is similar to many problems I encounter in hotels: outlets too far away from where I want to use them, chairs too high, sinks too high, and my favorite, too-high shower heads. Most of the time, the shower head is too far away for me to adjust. I’m short (5 feet tall), but if I can’t reach the shower head, it’s set for a 6-foot plus person. If the people who designed the hotel rooms had tried them at all, they would realize they were creating difficult situations for a significant percentage of the users.

So, it’s possible that architects don’t need to code and participate in a project. But I don’t know of other effective techniques to test the design. Testing design with a thought experiment is insufficient. Testing design by using it provides much more feedback.

However you arrange your project, think about how to test the design–the design in-the-large, and all the little pieces of design-in-the-small. Certainly, test-driven development is great for testing design-in-the-small. And if you have a whole project that’s willing to try test-driven design for overall architecture, that’s fabulous. But if you don’t or you can’t see how, at least think about how to build in testing of the design, especially design-in-the-large as you proceed through the project. Then you won’t end up with a bathtub practically in the toilet, requiring people to balance while stepping in and out of the shower.

Not every product has smoke tests (a series of tests you can run after each build to make sure the product works well enough to continue development and testing). Smoke tests provide early feedback to developers about their work. So, for the last several years, I’ve been suggesting to my clients that as they develop a feature, they include one short automated test for that feature in the smoke test. This test allows the developers to know their changes didn’t break the product.

Most of the time, that suggestion works. Some groups have found that just the discussion of what they wanted to insert into a smoke test was a great way to discuss the exceptions and make sure they’d handled the exceptions. And building a smoke test this way seemed relatively painless.

But one group has had trouble with this suggestion. A few of their developers are quite happy to create lots of automated tests per feature. And, their automated tests carefully tiptoe around the feature, avoiding anything that anyone would remotely call testing of the feature. The result is they have a large automated smoke test that tells them nothing. So for them, this is not a useful practice–as it stands.

I’ve suggested that each developer limit the number of smoke tests he or she develops to one per use case. Each developer gets to choose, and presents that choice, along with the smoke test to the rest of the group during a walkthrough.

This isn’t optimal, but the people in the group are becoming accustomed to receiving feedback about their work. With the way they’d been implementing “smoke” tests, they prevented themselves from seeing any feedback about their work. This technique might help.

So consider creating one automated test per feature as you implement. It might be a useful practice. But if people are preventing themselves from seeing feedback on their code, consider some other practice.

I have several accounts with a credit card company: two cards and a merchant account. They don’t want the expense of printing monthly statements for the merchant account, so they sent me a letter to enroll my merchant account in online statements to avoid the paper charge.

In my default browser (Safari), I attempt to log in. I can’t (Problem #1). In fact, I end up in browser limbo, with some message about too many redirect (Problem #2). I try to log in in Explorer. I can’t. It tells the me password is wrong (Problem #3). I call the 800 number three times before I get someone who tells me I need a new user id. I ask why. The nice lady says, “We keep our businesses separate, so you need a user id.” I say, “But (and I’m starting to rave here), the web UI looks the same. I have tabs at the top of each page that allow me to move back and forth. If you’ve integrated the web page, why not integrate the user ids???” (I’m definitly speaking loudly, because I’m so angry with the stupidity, waste of my time, and lack of documentation.) Poor lady just keeps saying they keep their business separate (Problem #4).

These are serious problems. I’m not a UI designer — I’m a user. This makes me an expert — in using computer systems, not designing them. The problems:

1. If a user attempts to log in and can’t, catch the error. Safari shouldn’t have tried to deal with this error, the web site app should have. Catching an unable to log in error is basic programming. How could they have missed this?
2. Why do you care what browser your users use? The world is full of browsers. That’s why we have Java. Accomodate all the browsers. Sure it means that you have to write code more carefully and test on many platforms. Is it worth losing any customers because your developers were too lazy to write good code?
3. Make sure the error text matches the error. There was nothing wrong with my password; they wanted a new user id. The problem is the text on the page. The text says “New user? Create an id.” But I’m not a new user. I manage my credit cards online every month. I’m an experienced user. The instructions were inadequate, as well as the error text.
4. If you have a consistent look and feel to the site, and you’re trying to seel multiple services to one customer, don’t make the customer have multiple ids, one for each service. I’m sure that having multiple ids makes it easy for this company to track revenue by division. But it makes it much harder for me to manage my user ids and passwords. Who is the customer here?

To add insult to injury, I can’t write this in an email and send them feedback. (Problem #5) I can call (and spend more time on hold? No thank you), but I can’t write. So, I’ve now made a public example of a stupid website. Great. When I make mistakes, I want people to tell me, not write about it somewhere public. Gotta wonder why these people don’t care.

Ok, tirade about stupid UIs off. Back to work.

I’m reviewing my students’ updated plans for their projects. One team originally wanted full unit testing on the code as it was created, but added (my paraphrase) “if the project is late, some unit testing will be acceptable.” I responded that the farther behind the project was, the more review and testing is required.

Here’s why. Projects get behind because the people don’t understand something. The more they don’t understand, the harder it is to create the product, and the more defects they introduce. This is not because the people are bad — good or bad development skills have nothing to do with it. This is a function of not clearly seeing what the system is supposed to be. If you find your project is behind, consider these options to help people see the system clearly:

• Hold peer reviews on every work artifact. I prefer reviews, not walkthroughs, but anything to put at least one more pair of eyes on the product is good.
• Inspect every defect fix. Yes, I do mean formal inspection, where someone reads the fix out loud.
• Full unit testing on every piece of code. The developer will understand his or her code when trying to test it
• NIghtly builds (more often if it makes sense). The faster the feedback to the developers about potential problems in their code, the faster you can decide what to do about the problem.
• Implement by slice. Instead of implementing the components of a system, implement just enough of a component to complete a given feature or use case. In my opinion, Big-Design-Up-Front doesnt’ work very well because the developers and architects don’t actually understand what they’re doing. This is not a competence thing :-); it’s an artifact of us as an industry not being able to fully articulate the requirements.

Some of you may be shaking your heads, thinking, “No, this will put the project farther behind.” Nope, it won’t. Instead of the illusion of progress, you’ll see the real progress. Reviews of healthy code speeds up the project in my experience. Reviews of sick and tired code reveals the work still remaining.

The problem is you won’t catch up. But you won’t lose any more time — not without knowing you’re losing time.

When I do assessments, invariably someone says to me, “JR, how long should the testing really take? It takes our testers (4/5/6/30 insert your number of choice here) weeks, and we need it to take less time. Why can’t it take less time and how can we tell what’s going on so we know how much testing we need?”

This is, of course, an “it depends” answer. I’ll try to untangle this.

If you do test-driven development, then there is rarely a lot of extra testing. When I coach teams who are transitioning to test-driven development, I recommend they plan for as much as one iteration at the end for final system test and customer acceptance test. Once the team has more experience with test-driven development, they can plan better. I have found a need for some amount of customer acceptance testing at the end. And that time varies by project and how involved the customers were all along.

If you’re using an agile lifecycle even without test-driven development, I recommend starting with one iteration’s length of final system testing. I am assuming that the developers are fixing problems and refactoring as necessary during an iteration — real agile development, not code-and-fix masquerading as agile.

If you’re using any of:

• an incremental lifecycle such as staged delivery where you plan for interim releases (even if the releases aren’t to the customers)
• an iterative lifecycle such as spiral or evolutionary delivery
• a serial lifecycle such as phase gate or waterfall

I’d expect as much testing time as development time — but it doesn’t have to come all at the end as final system test as it looks like in the pictures. Any proactive work you do, such as reviews, inspections, working in pairs, unit testing, integration testing, building every night with a smoke test, fixing problems as early as they are detected, can all decrease the duration of final system test. If you’re the project manager, ask the developers what steps they are taking to reduce the number of defects they create. If you’re the functional manager, work with the project manager and the developers to create a set of proactive defect-prevention practices that make sense for your project.

If you’re the test manager, recognize that final system test includes several steps: testing the product, finding defects, and verifying defects. Your first step is to separate these tasks when you estimate the duration of final system test.

One question you should be able to answer is: How long will one cycle of “complete” testing take? We all know we can’t do complete testing, so your version of complete is the tests you planned to run and any other exploratory tests or other tests you need to run in one cycle of testing to provide enough information about the product under test. I realize that’s vague and depends on each project. I don’t know how to be more explicit, because this is a project-by-project estimate. If you work with good developers, the cycle time can decrease a bit from the first cycle to the last — because the testers know better how to set up the tests and the product has fewer defects, allowing the testing to proceed faster.

Once you know how long a cycle of testing takes, estimate how long it will take the developers to fix the problems. I use this data: the number of problems found per cycle in the last project, my gut feel for how many more/less we should find per cycle in this project, bug-tracking system data telling me the average cost to fix a defect pre-release. If I knew that the first cycle in the last release found 200 problems, and it took the developers half a person-day each to fix problems and I have 10 developers, I estimate 10 working days to fix problems. That’s a long time. And yes, I was on a project where that’s what it took. It was agonizing — we thought we’d never be done fixing problems.

Now you know how long a cycle takes, how long it will take for fixes after the first cycle. How many cycles of testing do you plan for? When I set up projects, I tend to use some proactive defect detection and prevention techniques, so I generally plan on three cycles of testing. In a casual conversation with Cem Kaner a number of years ago, he said he planned on eight cycles of testing. For one project where I was a contract test manager, we had almost 30 cycles of testing. I can’t tell you how many cycles of testing you’ll need because that depends on the product’s complexity, how good your tests are, and how many defects the product starts with.

Here’s what I have noticed from my work with multiple organizations. The groups who want to decrease testing time tend to perform the least proactive work reducing the overall number of defects, and they typically perform primarily manual black box testing at the end of a project. I understand their desires, but they’ve set up their lifecycle and processes to produce exactly the wrong result.

The best guess for an estimate I have is to estimate the number of cycles you’ll need for testing, the duration of one cycle, the time it takes for developers to fix problems between cycles. Add up the testing plus fixing/cycle and multiply by the number of cycles you think you’ll need, and you’ll have an estimate of the testing time needed. BTW, when I do this, I never give a single number, I always give time per cycle, my estimate of the number of cycles, and my estimate of defect-fixing and verification time. That way I can show the costs of not performing proactive defect prevention in a nice way. And I can show my uncertainty in my estimation.

If you want to reduce testing time, create a low-defect product, test all the way through with a variety of test techniques, and use iterations, so you know at the end of 2-4 weeks where you stand with defects. The lower the number of defects and the more sophisticated your tests, the faster your testing time will be.

Take a look at these links:

1. Tricks of the Trade, thanks to Dave Liebriech. When I was a tester, I read the code (this tip is far down on the list.) I’m not sure the developers appreciated my questions, but if I didn’t understand something, I asked.

Here’s a tip I learned for people who need to facilitate meetings:

Always take blue painter’s tape to a meeting. You can use it to hang flip charts on any wall, or to organize index cards or stickies together, without fear that the adhesive will stick to the walls. You have to be more careful with paper, but the glue on the back of painter’s tape is less sticky than other tape.

2. Outsourcing? Consider the Source. I was a “quacko” (tester) with Tom Parmenter a long time ago. I vividly remember telling one manager or team that they could choose to release software that couldn’t be installed, but I wasn’t going to sit on the phones explaining the magic required to make the install work. We used an incremental/iterative lifecycle, organized as phase-gates. It worked (unless someone tried to finesse the exit criteria for a phase), but only in our culture. Culture matters, as well as process if you’re going to consider outsourcing.