Productivity metrics for development team

We have been thinking of some number based targets for our development team. The idea is to show productivity gains to the client over a period of time. Though there needs to be a lot of thought and brain hacking which has to done before we come up with these kinds of metrics, at present, only 2 metrics come to my mind right away.

We can have a quality metric which can be something like an average no of SIT defects attributed to code/design per total number of SIT test cases. Since we have already collected these data over the past 2 years, over many projects executed under the program, we have some idea of this figure and what we can target to achieve in the future. 

The other obvious metrics which can be given as a target is the defect turnaround time. This figure also is available for the past data and we can give a target value for the development leads for this metrics. But when I think hard, this metric is not fool proof. For example, we can track the total defect closure time from the time a defect was opened till it was finally closed. But the time spent in the various state of the defect in between those 2 end states are not tracked. And who should be responsible for it.  If the development team fixes a defect but testing team just sits on it and whiles away time before retesting it, it will look bad on the development team numbers.

Another fact is that both these metrics are lag indicator of the project quality. Neither is a lead indicator. Ideally, I would like to have some lead indicators which can be used to take preventive/ in course corrective actions.

Estimate Quality Metrics

While doing some Google search on metrics, I have stumbled upon a metrics called Estimate Quality Factor, also known as EQF. Now, as a first guess, I thought that this might have something to do with the confidence on the estimate. I was partly right. Though I was thinking in terms of the effort and how confident was one on the effort, the EQF actually is on the delivery date, i.e. the schedule. It is a really simple idea. One needs to plot the estimated end date for any phase of delivery against the calendar date. For instance, if you estimate that the Tech Spec phase of a project will be completed by say 20^th May, you plot 20^th May in the y-axis with today’s date on x-axis. Similarly, whenever the estimated delivery date changes due to any factor, the next estimated end date is plotted. If there are no changes also, one can just plot the estimated end date with respect to the calendar date on a weekly basis. For example, say next week, we estimate that the Tech Specs will be completed by say 23^rd May. So, we plot that. The idea is to see if all the dates plotted form a kind of straight line parallel to x-axis, which will signify that the deviation from planned delivery date is not much. The more there are ups and downs in the delivery date, means that the EQF was not good. I think that the EQF can also be plotted for the effort estimate also. And an effort estimate with a high confidence will have a line parallel to the x-axis.

If estimates keep changing too often, which is the case mostly, the line will never be parallel to the x-axis. Again, only if there is a process of estimation built into the development process and the estimates are recorded somewhere, pulling out this data on a weekly basis should not be much of a challenge.

Now coming to the part where I need to decide if this metric will be of use to any one? I am sure that we will not be able to get a parallel line to x-axis in real life projects since almost all the projects will have change requests coming up in the course of the project, which will change the estimate effort and may be the end dates also. Only in a scenario where there are no change requests, we can probably use this metric to prove the quality of estimates.

Instead of doing all these stuff, a simple metric like the effort variance should be able to tell me how good the estimate was.

Thoughts on measuring productivity

Now that we need to raise the bar, how do we do it? We really need to show some improvement in our process and pass the benefits back to the customer. This thing was there for sometime in my mind on how we actually measure the productivity of the project team and then benchmark it and try to improve further upon that. Now I need to qualify that my project teams consists of development and testing teams. We know that there are some metrics like test case executed per hour which can be used as a testing team productivity metric. Agree though that all of them are not without bias. For instance, some test cases may be very simple and straight forward to test which means more of similar types can be executed in a period of time while some may be pretty complicated ones, requiring specific scenarios to be created and data to be available to test, which in turn will take longer time to set-up and test. But when we are talking about a testing team’s productivity, we can assume that there will be both types of cases and finally at the end of the project we come up with an average test execution productivity figure. Comparing the same across various projects tested over a period of time will give us a fair indication of how the testing team is performing. With a defect tracking and time tracking mechanism that we have in place, we are now able to pull out these figures from those systems and are now working on the benchmarking part for testing teams.

But now the difficult part that I have been grappling with. How do we measure the development team’s productivity? One measure of this may be LOC. But this is not a unbiased one and in maintenance and enhancement type of projects, this actually doesnot mean much, coz most of the development time is spent in analyzing and modifying the existing code, rather than creating new programs. Also, the defect fixing turnaround time can be a measure, but this will be only for the SIT support phase. I am still to figure out some productivity metric for developers for the Analysis phase, Design Phase and Coding phase.

Raising the bar

As part of my regular job, I keep interacting with our clients. I have been fortunate enough to work closely with some people from the client organization and learn how they think an IT service provider should work. In the engagement that we are in, we have pretty established processes for most of the things and the delivery record has by far been quite impressive. However, the manager from the clients’ side feels that apart from consistent delivery, the vendor should also do something more. So, the other day, I asked what more can we do to keep them happy. He asked me what kind of investment you are making in the engagement. This set me thinking. We have in fact not done any worthwhile investment apart from seeing to it that the engagement runs smoothly. He was in a way correct. There can be various kinds of investments to get a better ROI from the engagement. For example, we could provide them a consultant from say our Delivery Quality Assurance wing, who can study our processes and propose improvement. The consultant can even study their processes and give recommendations. 

It is the value we bring to the table which matters. When the client was not happy with their earlier vendor, they came to us and we have proved them our mettle with our fine delivery mechanism and technical thoroughness. After 2 years into the engagement, the client has now got used to the fact that deliveries will be smooth. So now the bar has to be raised.

Quantifying value add by a PM

I was discussing the issue of value addition with my delivery manager. My point was while someone is managing a project, his / her main objective is to successfully complete the project with the minimum amount of issues. So at the end of the day, if one asks the PM, what is the value you have added to the project, the PM can say that I did this and I did that to mitigate this or that risk, I handled this situation in this way and finally the project was completed with minimum issues. But then, can the same be quantified? To this, my delivery manager asked me, what the biggest risk in any project is. As per him, the biggest risk is in loosing the customer, loosing the revenue stream that is established. So if you do anything that will keep the revenue stream intact and keep growing it, that is the value add. Everyone’s contribution may or may not be measurable in dollar terms. Sometimes, what the client pays can be considered to be a measure of the value add; but them it is a great leveler. The client doesnot pay as per individual ability or value add. It is a negotiated amount for a minimum general set of abilities. Even if you do a value add, the client doesnot pay you extra. So, it is very difficult to put a dollar value to it. I agreed to it. Also, another point is that the client appreciates someone when that person really contributes. And though a PM may be handling a team, he can many times act as an individual contributor and add value. Also, in the overall sense, if the team delivers the project successfully, the PM gets his / her share of appreciation. However, this cannot again be quantified. May be some one should come up with a frame work for the value add done by a PM.