Managing Product Development

In the last few weeks, several people have asked me how much rework is normal. Well, if you’re working in a test-driven development environment, you probably have very little rework. My estimates for the few real test-driven projects I’ve seen is that they spend about 10-15% of their time on rework (finding problems and fixing them after the technical staff thought they were done with the particular feature. (This percentage may be this high because the technical staff were learning how to work in an agile way.) They spend lots of time on iterating through the requirements with the customer, which I don’t count as rework.

But in the last few assessments I’ve done, and from other recent emails, my experience is that rework accounts for somewhere between 75% and 80% of a “normal” project’s time. That’s a huge drain from the project. The rework impedes forward progress. I know of these ways to reduce rework:

• See how little you can do, and deliver that much as quickly as possible. The technique I use most often is to break the pieces into groups of requirements/features and then perform iterations within whatever lifecycle the people are used to.
• Use peer reviews, walkthroughs, inspections, anything that forces more than one pair of eyes on the code.
• Pair the testers with developers, or somehow get the testers to test the developers’ code as soon as the product is built. Early feedback from testers can help developers prevent inserting more defects into the code.

All these techniques help developers focus on some small amount of work, and look for problems early in that work.

I wish I could say there was some reasonable amount of rework, but I don’t know how to define that number. Any rework is a problem that you could have caught before. The question you manage is: how much rework should you prevent, and how much rework (time wasted) is ok? (For research-y projects, the answer is lots of rework is ok. For get-the-product-out-the-door projects, any rework is not ok. ) You’re the only one who can answer that question for your project.

I’ve been at the STAR conference this week, and Cem Kaner’s keynote talk yesterday discussed the idea of extended random regression testing — take all your programmatic tests, and run them in random sequences for a long time. You’ll find defects you cannot find just running the tests by themselves. Here’s the logic behind this technique:

• The systems we develop and test today are more frequently many thousands or millions of lines of code, rather than only single-digit thousands of lines of code.
• You can’t adequately (and note, that’s not fully, that just adequately) test such a system with only manual testing, you need programmatic testing to adequately cover a large system.
• Once you have created programmatic tests, you can string the tests together in long sequences, and find problems that don’t show up when you test with each test on its own.

I used this technique (unknowingly) when I tested library calls for an application many years ago. I was surprised at what I did find. I expected to find functional errors in the library code. But I didn’t. Each library call was successful. But the program that ran the calls crashed. I had unknowingly perturbed memory leaks, memory corruption (uninitialized counters and pointers), things I had (naively) not expected to find. (You need testers who can write small programs to perform this kind of testing. If none of your testers have this capability, you may have second class testers. )

I can’t imagine a large system that wouldn’t benefit from using this technique. (Please tell me the circumstances under which you think this technique is not useful.) Even if you’re working in a test-driven environment, developing tests before developing code, you can still benefit from this technique as your system grows.

In this issue of Better Software, I have the featured article, No More Second Class Testers! and Frank Patrick has a great article, “Promises and Prescriptions, How the Theory of Constraints can help cure common project ailments.” I can’t give you a URL to Frank’s article, but maybe in a month or so he’ll post the article on his web site, and you can see it then. Here are some gems (all quoted from the article):

• “Get rid of task due dates. The only dates that count are those promised outside the project.”
• “At the project task level, don’t allow yourself to be bullied into multi-tasking behaviors.”
• “Plan for the need to plan along the way.”

It’s a great article. I hope you read it.

In the last few weeks, I’ve received several questions about how to assess the productivity and effectiveness of testers. I’m concerned about this, because a tester’s effectiveness doesn’t just depend on the quality of the tester’s work, it depends on the quality of the work product the tester tests (as well as the schedule and the work environment).

I’ve used an assessment technique to qualitatively look at the testers and what they can do. This technique is appropriate when you want to develop a hiring strategy, or when you want to see if you’re testing the product in the most effective ways. This technique is NOT good for comparing every person in your group. (Comparing every person in your group says each person is interchangeable and we know they aren’t.)

1. Define the types of activities you need performed in your group. Since this is an example for a test manager, these are some of the potential activities:

• Technical experts to test behind-the-scenes interactions
• Requirements experts who understand how the requirements translates into design
• Exploratory testers
• Step-by-step testers
• Expert users
• Automated regression testing
• Unit testers (in development?)
• Integration testers (in development?)

2. Now assess each person on how well they perform these tasks. I use an Excel radar chart (kiviat diagram) to do so. That helps you assess functional skills. Here’s what a chart looks like for a fictional group. The chart range is 0-5, where 0 is no experience and 5 is highly experienced and applies the experience. Yes, this is a judgment call by the manager.

Now, it’s time to look at the rest of the person:

3. Define the non-technical qualities, preferences, and skills that are necessary for job success in your job. Assess the person on how well they do that. Here are some examples of non-technical qualities, preferences, and skills for testers:

• Ability to write clear defect reports
• Ability to advocate for defects
• Communicate across the organization in writing
• Organizational ability (planning their work or coordinating the testing or whatever this means in your context)

Assess each person on how well they do that. Make another chart.

4. Define the rest of the technical skills: domain expertise (problem-space and solution-space), tools/technology, industry expertise.

Assess each person on how well they do that. Make yet another chart.

Now you have four pictures of your group, one in each of four dimensions. Look for overlaps and what you value more. This technique helps you see where you have gaps and where you might need to fill in the gaps.

Productivity is a group problem in software. Efficiency is irrelevant unless the person is on the critical path. Effectiveness is sometimes more about how someone uses influence and/or negotiation rather than their technical abilities. If you’re going to use an assessment technique, decide what’s important to you, and decide what “effective” or “productive” means in your context.