Managing Product Development

There’s a fascinating email thread started by David Anderson about What would Agile Auditing Look Like?. Part of the discussion stems from what the definition of an audit is. Audits are about compliance to a defined process. Do we need audits? Sure, for some projects. I would very much like to know that any project in a regulated industry has a defined process and that the project stuck to that process, unless they could show why (with data) a change improved the process. And, if an organization is going to claim a particular CMMI level, or some other “official” process adherence, the only way to really know that is with an audit. But most projects don’t need audits; they need assessments.

Assessments (at least, when I do them) are about looking at what’s working–and what’s not working. If part of the (either defined or implicit) process isn’t working, it’s my job as an assessor to learn about the root cause of the problem, and to suggest alternatives for the people in the organization to consider. When I do assessments where an organization has a defined process, I’ll use that process to see if the projects perform any work like the process. But it’s my job to understand the system of development, not look for compliance to the process. I’ve discovered the causes of problems in projects that an audit would have missed. For example, the variety of architectural pictures of the same system for one multi-site organization was the cause of much of the disagreement and much of the confusion for one large project. Their process didn’t say to have an architectural picture; the developers and architects thought they could use one. One (and only one) would have been great And the reason they didn’t have only one picture went back to the management team, not the project team.

An audit may have discovered this problem. But audits are about compliance, which means they tend to blame people. (The best auditors do not blame their project teams.) So when I do assessments, I ask questions such as “What situation or assumptions would have made this a reasonable reaction/decision/whatever on the part of this person?” That helps me see the system of the project, not look for compliance.

If you haven’t read the classic Places to Intervene in a System, take a little time and do so. My colleague Don Gray has written an afterword that relates the article to software product development.

I’m realizing that the way I do assessments uses these interventions. I have to think more about how to write that, so that will be a later post.

When I do assessments, invariably someone says to me, “JR, how long should the testing really take? It takes our testers (4/5/6/30 insert your number of choice here) weeks, and we need it to take less time. Why can’t it take less time and how can we tell what’s going on so we know how much testing we need?”

This is, of course, an “it depends” answer. I’ll try to untangle this.

If you do test-driven development, then there is rarely a lot of extra testing. When I coach teams who are transitioning to test-driven development, I recommend they plan for as much as one iteration at the end for final system test and customer acceptance test. Once the team has more experience with test-driven development, they can plan better. I have found a need for some amount of customer acceptance testing at the end. And that time varies by project and how involved the customers were all along.

If you’re using an agile lifecycle even without test-driven development, I recommend starting with one iteration’s length of final system testing. I am assuming that the developers are fixing problems and refactoring as necessary during an iteration — real agile development, not code-and-fix masquerading as agile.

If you’re using any of:

• an incremental lifecycle such as staged delivery where you plan for interim releases (even if the releases aren’t to the customers)
• an iterative lifecycle such as spiral or evolutionary delivery
• a serial lifecycle such as phase gate or waterfall

I’d expect as much testing time as development time — but it doesn’t have to come all at the end as final system test as it looks like in the pictures. Any proactive work you do, such as reviews, inspections, working in pairs, unit testing, integration testing, building every night with a smoke test, fixing problems as early as they are detected, can all decrease the duration of final system test. If you’re the project manager, ask the developers what steps they are taking to reduce the number of defects they create. If you’re the functional manager, work with the project manager and the developers to create a set of proactive defect-prevention practices that make sense for your project.

If you’re the test manager, recognize that final system test includes several steps: testing the product, finding defects, and verifying defects. Your first step is to separate these tasks when you estimate the duration of final system test.

One question you should be able to answer is: How long will one cycle of “complete” testing take? We all know we can’t do complete testing, so your version of complete is the tests you planned to run and any other exploratory tests or other tests you need to run in one cycle of testing to provide enough information about the product under test. I realize that’s vague and depends on each project. I don’t know how to be more explicit, because this is a project-by-project estimate. If you work with good developers, the cycle time can decrease a bit from the first cycle to the last — because the testers know better how to set up the tests and the product has fewer defects, allowing the testing to proceed faster.

Once you know how long a cycle of testing takes, estimate how long it will take the developers to fix the problems. I use this data: the number of problems found per cycle in the last project, my gut feel for how many more/less we should find per cycle in this project, bug-tracking system data telling me the average cost to fix a defect pre-release. If I knew that the first cycle in the last release found 200 problems, and it took the developers half a person-day each to fix problems and I have 10 developers, I estimate 10 working days to fix problems. That’s a long time. And yes, I was on a project where that’s what it took. It was agonizing — we thought we’d never be done fixing problems.

Now you know how long a cycle takes, how long it will take for fixes after the first cycle. How many cycles of testing do you plan for? When I set up projects, I tend to use some proactive defect detection and prevention techniques, so I generally plan on three cycles of testing. In a casual conversation with Cem Kaner a number of years ago, he said he planned on eight cycles of testing. For one project where I was a contract test manager, we had almost 30 cycles of testing. I can’t tell you how many cycles of testing you’ll need because that depends on the product’s complexity, how good your tests are, and how many defects the product starts with.

Here’s what I have noticed from my work with multiple organizations. The groups who want to decrease testing time tend to perform the least proactive work reducing the overall number of defects, and they typically perform primarily manual black box testing at the end of a project. I understand their desires, but they’ve set up their lifecycle and processes to produce exactly the wrong result.

The best guess for an estimate I have is to estimate the number of cycles you’ll need for testing, the duration of one cycle, the time it takes for developers to fix problems between cycles. Add up the testing plus fixing/cycle and multiply by the number of cycles you think you’ll need, and you’ll have an estimate of the testing time needed. BTW, when I do this, I never give a single number, I always give time per cycle, my estimate of the number of cycles, and my estimate of defect-fixing and verification time. That way I can show the costs of not performing proactive defect prevention in a nice way. And I can show my uncertainty in my estimation.

If you want to reduce testing time, create a low-defect product, test all the way through with a variety of test techniques, and use iterations, so you know at the end of 2-4 weeks where you stand with defects. The lower the number of defects and the more sophisticated your tests, the faster your testing time will be.

In the last few weeks, I’ve received several questions about how to assess the productivity and effectiveness of testers. I’m concerned about this, because a tester’s effectiveness doesn’t just depend on the quality of the tester’s work, it depends on the quality of the work product the tester tests (as well as the schedule and the work environment).

I’ve used an assessment technique to qualitatively look at the testers and what they can do. This technique is appropriate when you want to develop a hiring strategy, or when you want to see if you’re testing the product in the most effective ways. This technique is NOT good for comparing every person in your group. (Comparing every person in your group says each person is interchangeable and we know they aren’t.)

1. Define the types of activities you need performed in your group. Since this is an example for a test manager, these are some of the potential activities:

• Technical experts to test behind-the-scenes interactions
• Requirements experts who understand how the requirements translates into design
• Exploratory testers
• Step-by-step testers
• Expert users
• Automated regression testing
• Unit testers (in development?)
• Integration testers (in development?)

2. Now assess each person on how well they perform these tasks. I use an Excel radar chart (kiviat diagram) to do so. That helps you assess functional skills. Here’s what a chart looks like for a fictional group. The chart range is 0-5, where 0 is no experience and 5 is highly experienced and applies the experience. Yes, this is a judgment call by the manager.

Now, it’s time to look at the rest of the person:

3. Define the non-technical qualities, preferences, and skills that are necessary for job success in your job. Assess the person on how well they do that. Here are some examples of non-technical qualities, preferences, and skills for testers:

• Ability to write clear defect reports
• Ability to advocate for defects
• Communicate across the organization in writing
• Organizational ability (planning their work or coordinating the testing or whatever this means in your context)

Assess each person on how well they do that. Make another chart.

4. Define the rest of the technical skills: domain expertise (problem-space and solution-space), tools/technology, industry expertise.

Assess each person on how well they do that. Make yet another chart.

Now you have four pictures of your group, one in each of four dimensions. Look for overlaps and what you value more. This technique helps you see where you have gaps and where you might need to fill in the gaps.

Productivity is a group problem in software. Efficiency is irrelevant unless the person is on the critical path. Effectiveness is sometimes more about how someone uses influence and/or negotiation rather than their technical abilities. If you’re going to use an assessment technique, decide what’s important to you, and decide what “effective” or “productive” means in your context.