How to measure productivity? That’s a question that many in the business, from CEOs to coders to engineers to managers, have in their minds all the time, and Adolfo Cruz, Scio’s very own Project Management Office director discusses metrics, measures, and meanings of productivity.
At the end of the 90s, a methodology called “Personal Software Process”, or PSP, was designed to help developers measure their productivity. You had to take a course, there was a lot of documentation to follow through, and you had to use a timer to measure your progress, stopping it every time you needed a cup of coffee or to go to the bathroom.
The idea was to see how much you accomplished in a day, but in fact, this methodology was entangled too closely with the number of lines you wrote, meaning that you were more productive the more you coded, which is not necessarily true.
But if this is not productivity, what is it?
I define “productivity” as finishing a task in a reasonable time. The word “finishing” here is key because productivity is not starting a lot of things, but seeing a project to completion, right until you get a product. However, how do you define exactly when something is “finished” in software development? What criteria do you have to fulfill to say something is done? If we were building something physical, let’s say a boat, first, you need to build a hull, and this phase ends when it fulfills certain requirements.
And although not all boats are the same (building a freighter or a yacht would look very different), in essence, you have the same process, blueprints, and requirements to do it. Software development doesn’t work that way.
Developing software involves a lot of things. You may see it conceptualized as a “factory”, or a development team working like a well-oiled machine, where you input requirements and get a product at the other end. But in truth, there’s an artisanal quality to developing software; everyone has their approach and style, and progress changes according to the team you are with.
This results in a lively debate where no one agrees on the best way to measure productivity. If you ask a room full of developers how many lines of code they write to see if they are productive, you can get a very heated discussion, because how are you measuring that?
The best code is concise and everyone checking it can understand it, so I don’t know how you can see someone writing 10,000 lines of code and conclude he is more productive than someone achieving the same in 500. Maybe it made more sense at a time with no frameworks to build things faster, when everything was a bit more rudimentary and you coded every single piece of the product, but today you have to write very few things from scratch, with a whole range of tools that let you produce, let’s say, the shell of a website in a minute without coding anything directly.
So imagine if a company starts giving productivity bonuses based on lines of code produced per project. They would end up with developers gaming the system to get the bonus or at least trying to not look worse than their peers by writing less, resulting in bloated, inefficient products because the incentive wasn’t centered on creating something useful.
You have to be very careful when linking rewards to metrics, or else you’ll generate a perverse environment where everybody is just racing to inflate numbers.
The Scio way
I’ve been with Scio for 14 years, and since then, perspectives have changed. With the arrival of Agile Methodologies, we moved on from counting lines of code to seeing how that code comes together, achieving working software whose process of development is not focused on numbers, but on how the product is going to be used.
To give you an idea of this evolution, not long ago the requirements of a project were written from the point of view of the system, so every requirement started with the phrase “The system shall…”: the system shall do X thing, the system shall do Y thing, the system shall do Z thing, etc.
So you ended up with a massive document repeating “The system shall…” for every little thing. Then the Agile Movement centered on the user, with requirements stating “The Administrator can…” or “The Manager can…” because we understood that software is not about a system that “shall” do things, but what people in different roles “can” achieve with the product, resulting in productivity built around how much value we give to the final user, not how much code our devs write.
Coming back to Scio, we see it from the perspective of the stakeholders and clients we are building a product for, and our productivity is measured on the information we get from them, knowing how our teams are doing, how much value they are adding to a project, and what their perception of the whole process is. It’s a more people-oriented approach, far from the days of counting lines of code, and more interested in understanding how a developer is contributing to the goals of our clients.
To that end, we developed some internal tools, like the Team Self-Assessment, based on our prior experiences, consisting of questionnaires about the things we consider important for a team to focus on. For example, there’s an entire section about quality, how they are preventing issues during development, if they are doing Pair Testing practices, or if they are doing code reviews to make sure the code is maintainable and scalable…
Are they giving issues the proper attention? Are they documenting them? The team members have to ask themselves if they are focusing on the right issues, and if they aren’t, it’s something we have to improve. That’s how we try to motivate our teams to share their experiences, practices, and insights into our client’s projects.
It is said that only 35% of software development projects succeed, and I think it has to do with the planning stage of a product. Let’s say I want to complete the A, B, and C steps of a project in six months, based on previous experiences in similar projects. But it ended up taking 8 months instead of 6 because something needed to change, does that 2-month delay mean the project is going wrong?
It happens a lot with start-ups trying to create something completely new. In the course of development, it’s common to see something, a feature or function of the product that changes the client’s perspective, that taps into some potential we haven’t seen before, so the plan has to get reworked to harness that and bring its full potential. In six months, priorities can change.
But if we measure the productivity of the process very rigidly, and then that very same process brings out the value in unexpected places that, nonetheless, force you to rework entire parts of the project, it’s easy to see it as a failure.
The Project Management Institute uses these rigid measures a lot, asking for a specific basis, beginning, and end of every phase of a project, and if you don’t deliver them exactly as planned, then you get a mark against you. In an actual software development environment, that doesn’t happen, because the dynamics of a development cycle can change fast.
Software development works by evolution
The measures you have to use are subjective more often than not. Some industries require strictness, especially when manufacturing something, but in the world of software, and start-ups in specific, I don’t think it’s necessary to be like this to create a successful product.
This is why we back away a little from the technical aspects of a project and focus instead on the business side of things, having conversations with stakeholders and product owners to get them involved, reconciling all these points of view about what the business needs, and how development is.
We take a look at the features planned, check how many people ask for them, how critical they are for the business model to work, and decide how to proceed from there, adding real value by focusing on building those pieces first. Technical aspects are solved later, as you first see what the business needs, and then the technical team starts sketching solutions for the challenge.
Productivity is a question with no definitive answer yet.
Considering all this, forming an exact image of productivity is a question with no definitive answer yet. Every individual has to decide what works, but only in the specific context in which that person is acting, so no one has come up with a universal method to measure productivity in software development, as even the perspective from which you measure can change; seeing it from a client’s perspective is a world apart from a developer’s.
As you discover better routes during development that weren’t foreseen during the planning stage, or maybe because the technical aspect ended up being unfeasible for one reason or another, or the infrastructure cost is too high for your purposes, or any other number of reasons, a lot of what you may define at the beginning of the project will change.
You adapt, which is very different from building furniture or producing food, where it is clear what you are trying to accomplish over and over. But in software, where there’s no single solution for the same problem, it’s difficult to reach a consensus on what you need to do in detail.
However, you want to measure productivity, metrics evolve, and whatever method you use to decide how productive your team or your company is, I think the Agile Methodologies got it right, where it doesn’t matter the number of lines, or the documentation you have, or how pretty your database is, what matters to the client and the final user is having software that works.