Managing Product Development

Both David Maister, in Compensation Systems, and George Dinwiddie, in Agile Compensation, have useful comments about compensation systems. There’s something implicit in both pieces, that the criteria to move from position to position (as well as from one salary to another) have to be explicit. For years, I called this expertise criteria.

You develop expertise criteria by defining what’s critical about a position. If you’ve done a hiring strategy and a job analysis for each level of the jobs that report to you, you can develop expertise criteria. Take your 5-7 essential skills (most of these are non-technical), and rank them across the top of a table. Do NOT include number of years of experience or the educational level. Those two things are for HR, not for promoting or compensating people.

Down the side, list the levels. If you have developers and testers, and they similar essential skills (likely in an Agile team), list the levels of developers and testers together. That is developer level 1 and tester level1 are on the same line. Now fill in the matrix. (I know, this is hard work.)

When I did this for my engineering group many years ago, these were the essential skills across the top: Technical knowledge and methods, Initiative, Independence, Responsibility/Judgment, Scope, Complexity. Down the side, I listed positions from associate engineer, engineer, senior engineer, principal engineer/manager, consulting engineer/group manager, senior consulting engineer, director, vp.

I found it relatively easy to fill in the matrix for the first 4 levels. It was much harder for the last 3 levels. But without expertise criteria, I don’t see how you can have an entire conversation about compensation. You get stuck in “did you do this practice” as in George’s post, rather than “did you better the project or the organization?” (George argues against the “did you do this practice” thinking.)

For me, the expertise criteria gives a level of transparency that I think David Maister is trying to get to, but doesn’t actually say. (He does say the system needs to be fair.)

Many of my clients do not have expertise criteria charts. If you’re a manager, consider making one just for your group, and see if your performance evaluations and compensation discussions are easier. I bet they will be.

Links you might want to read:

Hiring Strategy #1: More People for Similar Work. (There are 7 hiring strategy posts. I will categorize all of them at some point.) Avoid Shot-in-the-Dark Job Analysis. There are more job analysis posts. I gotta get to this categorization

I taught my Pragmatic Project Management workshop in Israel last week. I was talking about defect charts and what they mean and how I use them. (No, I don’t include priority or severity data on defect charts; just # opened and closed by week and # remaining open each week.

One of the participants explained that he also tracks # Fixed Private Defects. What are those? Fixed defects in a developer’s sandbox, not checked into the mainline. Why are they not checked into the mainline? Because the organization does staged integration of several highly complex sub-projects, each of which has interdependencies with the other. And the build takes over a day.

This participant’s defect data is not about defects. This data is about the cost of the build and possibly the product’s architecture. When you can’t build the whole product every day (this organization builds a couple of times a week), you create a bottleneck of fixes waiting to be integrated into the build. You slow down development–dramatically at the end of a project, badly enough during the project–to keep the build going.

Let’s run a few numbers and see what it costs them to continue staged integration. Let’s assume it costs just one person-day to create a fix. (That’s optimistic, but not unrealistic.) Now, assume that for three months of a project, they have a weekly average of 50 “private-fixes” waiting to be integrated. (I’m assuming the 50 here, I have no data. But I think it’s not far off reality.)

To keep staged integration going, they spend 50 person-days every week for 12 weeks. (Remember, this is a large project, really a program.) That’s 600 person-days. I wonder if they could spend 600 person-days and fix the build so that they could build every day, without the need for staged integration. I don’t know, but I suspect they could.

Watch for what your data is telling you. You might think you’re measuring one thing, but you’re really measuring something else. In this case, the defect data is not about defects; it’s about the cost of an inadequate build system. What else are your defects telling you about your system?

Labels: measurement, project management

Jason Yip’s Hail Mary software development talks about what happens when you defer all the finishing to the end of the project. In the dashboard chapter of the book, I have a sidebar called “How Can We Have No Completed Work?” which talks about exactly the same thing.

The more serial your lifecycle, or the more you implement by architecture instead of by feature, the less completed work you have. Since I don’t count partially completed work, it’s possible to have 0 progress until the very end where things finally start coming together.

One of the assignments I gave my class this semester was to provide me a status report for the projects. Several of the teams decided to give me % complete measurements. I asked how they knew it was 80% complete. One team actually told me they’d done 80% of the work so far.

I told them to track the rest of the effort. I’m sure they fell or will fall into the 90% done schedule game.

Percent complete makes no sense. Features are done or not done. You can count done features and see how far along you are. You can’t reliably count any percentage done.

Labels: measurement

I taught my project dashboard workshop today. One of the things most people want to measure is progress towards project completion. But you can’t measure project completion progress unless you have completed features: developed, integrated, and tested features. A completed feature is done enough for someone to use.

Implementing by architecture leaves all the feature integration up in the air until the end of the project, so if you must implement by architecture, you just can’t measure completion until the end. That’s too risky for me. But if you implement by feature, you can start measuring completion as soon as you’ve implemented one feature.

I’ve encountered a number of projects where people didn’t know the context of their work. As developers, they were working on the thing they had to develop or fix today. They might remember what they had done yesterday, but there was no sense that they knew what they needed to do tomorrow, or that they were working on something that’s part of a whole. (I wrote an article about this years ago, Of Crazy Numbers and Release Criteria) I’m at a client now doing an assessment and I saw some data yesterday that made me realize people assessing projects need trend numbers, to see the data over time, not just snapshots of data.

When I do assessments, I always ask for trend data (about velocity, requirements, defects during the project and defect escape rates, sometimes code). For many of my clients, the data doesn’t exist, and it’s hard to obtain.

Trend data shows you the context of your progress, whether it’s fast or slow. Velocity charts show you how much work you’re accomplishing over time and how much change is in your project. Defect trend charts show you how well you’re dealing with defects (are they overwhelming you or are you keeping pace). Trend data allows you to see the context of a project. Are 15 (or 150 or 1500) open defects good or bad? It’s hard to know without trends. PMs can’t make decisions about how to guide the work until they know the context, not individual pieces of data.

If you’re managing project work, take a look at your project. Do you have trend data or daily (or weekly or monthly) snapshots? What would you have to do start gathering trend data? And what would the trend data show you?

I’m writing my project management book. I have no idea how far along I am. (Wait, I promise to explain.) When I write, I have several phases: the exploratory phase, where I write articles, the write-it-down phase, where I write the whole thing down (in chunks, of course), and the editing phase. I’m in the write-it-down phase right now. In this phase, I need hours at a time to blast away at a chapter until there’s little enough of it left so I can write it in spurts. (I hope you understood that.)

I was talking with my editor this morning about when I would be done with the first draft, the write-it-down part. I explained that I had a slim chance of being done by mid-October, and a stronger chance of being done at Thanksgiving. Here’s the graph of what that looks like:

Note that there’s a 50% chance I’ll be done with this phase, the write-it-down phase, by mid October. I have a bunch of writing time between now and then for the book. I have some other writing to do, some web site updates, etc. , but most of my time is available for writing the book. If I miss mid-October, I have no writing time until almost the middle of November. That’s why if I’m close in mid-October, I might be able to finish in mid-November. But if I miss mid-October because I’m not close, the mid-November date is quite risky. I have a bunch of writing time from mid-November through early December, but I expect to be traveling again and teaching, which changes my writing time.

This is an example of probabilistic scheduling. I don’t even have a 100% completion date (which you might have to have for one of your projects), because I will be replanning in mid-October, no matter what. Any 100% date I give now will be wrong, wrong, wrong. It will either be too optimistic or too pessimistic, and I won’t know which one until October.

It doesn’t matter what kind of lifecyle you use, the further out the dates are, the less you know. You can use probabilistic scheduling to help you, the project team, and your sponsors see the risk in the schedule.

I met a manager recently who relayed his technique for making sure his testers stayed focused on their jobs. “Our defect-tracking system logs people off after 30 minutes of idle time. If they’re logged off, I know they’re not working.”

This was a new one for me. I’ve heard of counting lines of code. I’ve heard of counting cars in the parking lot at 7pm. I’ve heard of counting number of defects found or fixed (depending on whether you’re a tester or developer). All of which are incredibly stupid measures. But this is the first time I heard a manager use a necessary tool to determine who was working.

I can think of lots of reasons a person might not be actively working in the tool. I played the bounce-back game with a developer many years ago (”it’s a defect”… “No it’s not” … “yes it is” … “No it’s not”…) until I finally decided that what the developer was really saying was “I can’t see the damn defect, so it’s up to you to make it obvious to me.” Took me 4 hours to create the simple test case (I’d already written up the complex test case). In that time, I referred to the defect on my screen, brought up other apps, made notes, tried a few things on my machine, done research in the lab, talked to a few people, and finally, 4 hours later, wrote the 5 steps into the defect report.

I was working the entire time. And if my manager had used licenses as a way to log me out, I’d have had to change context to get back to the defect I’d been working on before the system logged me out. I’m sure it would have taken me longer.

When I asked him why he allowed the system to log off the testers, he explained that they didn’t have enough licenses for all the developers and all the testers to have the defect tracking system open at all times. How much was a license? $1500. Each licenses costs 3 person-days. How many person-days do you think they’ve wasted because someone couldn’t get a license or someone was logged off? A prime example of being penny-wise and pound-foolish.

If you need to keep the costs of development down, use agile lifecycles, or at least agile techniques. If the developers test code before they check code in, you might not even need a defect-tracking system. (Index cards will do.) But don’t keep the cost of development down by not providing all your technical staff with the tools they need. One manager’s salary (and there’s almost always a surfeit of managers in organizations like this) will pay for all the tools you need.

To track work, track completed, deliverable work (features/requirements implemented, tested, built, and able to be delivered to a customer). There is no other substitute.

I gave a talk entitled “Predicting Project Completion” at the Central Mass chapter of the PMI last night. I had some suggestions about techniques to generate and discuss schedule estimates. Then, to practice a little, I asked the audience to become participants and practice a simulation. The simulation is to first estimate how long it will take to sort two decks of cards (I only allow three minutes for this part), and then to do the sorting, comparing the estimated time with the actual time.

As I debriefed the simulation, some of the people said, “Three minutes is not long enough to plan this project. Some people disagreed, “Three minutes is plenty of time to plan the project–because we can’t fully plan, so we can plan enough in three minutes to get started.”

I don’t know how much time is enough time to plan your projects. It depends on where your risks are: in the people not knowing the subject domain; in the pre-determined costs; in the whether you have enough people assigned when you need them; whether the feature set is known or will evolve; whether anyone knows how good the project needs to be. You probably have more risks on your projects.

But I am sure that for a projects of more than one person and more than one week in duration, three minutes is not enough planning. Our projects last night had actual durations of about 2.5 minutes to 7 minutes. Taking more than three minutes to plan would have been nuts–we could have spent more time planning than working. But I regularly see 6-12 month projects where the PM and technical leads spent less than three minutes planning (by their own admission).

I recommend starting every project with a project charter, and taking an hour or maybe two with the technical leads (or on a small project, with each person) participating in generating the charter. You may not know precisely where you’re headed, but you’ll already have generated some shared goals, which will help you make the decisions as you proceed. Once you’ve spent that hour or two, you can decide if you need more planning at the beginning of the project, or if you can continue to plan as the technical staff proceed, or if you iterate the plan as you proceed (or both).

There is no One Right Way to start a project, but a project charter can help. Taking just enough time to charter the project may help you understand where your project’s risks are, and how to deal with them. And that may be about enough planning.

Student Syndrome occurs when the person with the task waits until the last possible moment to start. Some people spend their entire academic career waiting until the night before a project is due and then starting it, pulling an all-nighter, and getting some (hopefully adequate) grade. Student Syndrome isn’t for me, but I know lots of people who do this.

I use these techniques to avoid Student Syndrome:

• Ask each person to develop inch-pebbles so that person (and the PM if necessary) can track progress.
• Use Estimation Quality Factor to continuously predict the end of their current task (not just the end of the project).
• Ask “What have you completed today?” Just asking can help jiggle people into starting the work.

These techniques work for me too, not just when I manage other people. Just because I don’t wait until the last possible moment doesn’t mean I don’t procrastinate every so often. (In English, that means I procrastinate too

Student Syndrome isn’t the same as being stuck, although if I’m stuck, it can look like I’m procrastinating instead of working on the task. I use a timeout to see if I’m stuck. For any given task, if I can’t make progress on it in about 30 minutes, I ask for help. 30 minutes may be too short or too long for your tasks, so adjust accordingly.

If you’re a PM or functional manager, notice if your staff are waiting until the last possible moment to start. If so, try something to help people start earlier. Late-as-possible starts lead to late projects.

---

The reason I thought of this today was because of voting. I hate waiting in line, so I vote in the middle of the day. Mark is an early bird, so he voted at 7:30 this morning. On an errand, someone told me she might not have time to vote. I asked “Can’t you take 20 minutes right now? There aren’t any lines. You could do it and come back to work later.” (She has the ability to do this.) She replied, “I’ll just wait until the end of the day.” Student Syndrome all over again. If you live in the USA, don’t delay. Go vote.

I’m at PNSQC this week. I gave my metrics talk yesterday, and something occurred to me: in traditional projects, we’re used to measuring what’s been done. In agile projects, we measure what’s left to do. I just realized yesterday that the difference in how we measure makes a difference in how people feel about the project. The more you measure what’s left, the more you can see the end of the iteration or the end of the project. It’s also a lot clearer to see how many more iterations it will take if management decides to add more features.

I’ll be modifying my measurements — even for not-specifically-agile projects — to reflect what’s left to do, not what’s done.