Project Management or Why Certain Groups Work and others Most definitely do not Work

Project Management, I will freely admit, is a discipline that I do not entirely understand. It seems like it would be easy enough: collect requirements, keep track of the details of progress made against said goals, identify problems hopefully before they come up, and chart a course through problems when they do arise. And, yet, project management remains mercurial and mysterious. Hell, I think most Project Managers would find it hard pressed to fully articulate what leads to successful project management vs. unsuccessful project management.

Part of the problem, of course, is that so many projects tend to fail. Sure, the project may get finished, but frequently it is past due and over budget. And this keeps happening, over and over and over again. Currently at work my group has a nominal project manager, though he is really assigned to a different group that my group works closely with. So, he's done a fantastic job of being involved with our work and keeping track of progress and problems, but it is not where the core of his attention lies. Because of this we have, as developers, sort of had to adopt the mantle of project managers ourselves.

Keep in mind, we are definitely not project managers, nor do we have much interest in becoming project managers. We want to build things and build them well, and make what we have already built better. This is why we're developers. That said, we've gotten pretty good at keeping track of things, and the last 4 or 5 months have seen us roll out some rather high quality code more or less on time. This is good for us. More recently, however, our team has been split up a little bit, with one of our developers being temporarily reassigned to another project in desperate need of man-power, and our QA guy having to split his time between multiple projects, again, for want of man-power. Our front-end developer went on a much-deserved vacation for 2 weeks. I'm going on vacation for a week in a few days. And all this internal upheaval has revealed something which I think is often overlooked.

When or group is full staffed it looks like this: 3 dedicated application developers (in my case doing PHP/Python web application development), 1 front-end developer/designer, and 1 full time QA guy. Personality wise we are each extremely different from each other. Some of us get into the office at 8 and are out by 5. Others start at 10 or 11 and work till 7 or 8. We check email and IRC at odd times. Only 2 of us work in the same office, the rest are distributed between Mountain View, Davis and Seattle. But, despite this, we work extremely well together.

Basically, we've worked closely together for long enough, that we just know what each other are going to do and, more importantly, we trust each other to do what we say we're going to. Much has been made about different development methodologies (scrum, agile, XP, etc.) but, in my opinion, your choice of methodology doesn't actually matter much. What you need is a small group (no more than 5 or 6, in my opinion) who are in constant communication with each other and focused on getting something done. When you have this your choice of project management tools kind of becomes one of personal taste. I personally find scrum, with it's burndown lists and notes about "velocity" to be unnecessary. But I do find having a 15 minute daily meeting to talk about where we're at and what we're working on next to be invaluable. I find having a list of tasks, either in the form of a burndown list or (preferably) a jira task to be extremely important. (I prefer jira or some other ticketing system to be preferable because I can include the jira number in my code commits. If, on the rare occasion you have to go back and figure out why something was done you can then look up the relevant ticket and get a description of the problem trying to be solved and the entire developmet/qa history. You do not get this with burn down lists.)

So, now that our group has been temporarily fractured what's been the outcome? Stress, mostly. Tasks that others used to handle are now the responsibility of the remaining group members who have even more things to remember and take care of. "Velocity" slows. Quality becomes harder to maintain. Poor decisions are made in the name of expedience.

Fortunately, in this case, I think these will be temporary, and I am hopeful that by the end of March we will "get the band back together," so to speak, and get back to rockin' and rollin' on a new project.

A note about pitfalls. Or, rather, some general observations about common problems encountered. Or, rather, shit I've noticed that keeps going wrong even though it shouldn't. Or, more precise still: why we won't learn from the fucking past.

In every company I've worked with there is a necessary, and at times good and productive, tension between Product (or business, or management) and Development. Generally speaking the Project Manager is caught in the middle and is meant to act as a buffer between those two groups and negotiate a plan that won't drive everyone nuts. In practice this rarely happens. And perhaps this should be elaborated on in a subsequent post, but it is this negotiation that Project Managers most frequently fail at

So, here are two hypotheses that I'm still in the process of testing, but my gut tells me they're correct.

1. Any Project will consume all available time allotted to it
2. In general, the development group does, in fact, provide realistic and accurate time estimates

The first is basically my observation of human nature. If your deadline is sufficiently far enough away it becomes very easy to convince yourself to do something tomorrow rather than today. Even if you don't procrastinate by screwing off (solitaire, mine sweeper, epic tournaments of L4D) you still, as a developer, have an increased willingness to put out fires, do favors for other people, spend some time consulting on a completely unrelated project, etc. A sufficiently far off deadline does not give you the sense of impending doom and failure that a closer one does. In short, distant deadlines are bad. More specifically, a project should probably not think more than a month, maybe 6 weeks, into the future. And, to it's credit, this is something that scrum tries to ameliorate. (Well, scrum fanboys will say scrum aims to solve it, but I think that's just wishful thinking)

The second point is something that Product and many Project Managers don't entirely understand. And part of this is because providing accurate time estimates is actually a skill that you frequently learn the hard way and only comes with experience. It is also a skill that many developers don't spend time learning because they don't think it's important. What is the price of giving an inaccurate time estimate? The immediate cost of being wrong is mostly that Product is annoyed and then you trim back some features. Or you slap "beta" on it. Or whatever.

So, I know what you're thinking: didn't I just say that the development group generally provides realistic and accurate time estimates? Well, I did, and I do think that. And this is what I've seen happen quite a few times. Product will ask "How long will it take to accomplish X?" and 15 engineers will look at it and talk about the component pieces, the amount of work required to make it, how much QA time, what environments to test in, guesstimate what bugs will be found and how long to test, and then factor in the timing for the release process and come up with a date. And the experienced developers know the process and system well enough that they can make some very good guesses about what needs to be done and how long it will take. The inexperienced developers, however, generally don't know these systems/processes well and tend to overestimate their own skill and underestimate the types of problems that are likely to come up. Furthermore, inexperienced developers frequently want to "prove" themselves and are energetic, ambitious and approval-seeking. So, they listen to what Product wants and they try to deliver it, and in the process prove themselves to their peers and get the approval from management.

Unfortunately, most of the time they're just wrong.

Extra unfortunately, Product (or Business or Management) frequently doesn't know a thing about programming, the company's technical infrastructure, or the software release process, nor do they know (or even have a way of judging) the relative experience of individual developers. And so when developer A says 6 weeks and developer B says 4 weeks, Product decides to listen to whomever gives them the answer they want to hear, not the one that most resembles reality.

I can't say I blame them. They're being asked to make a decision with imperfect information. Plus, they have their own agenda. They want to get a new Product Feature out to the public. They don't really care what it looks like under the hood, just provided that it does (more or less) what they want it to. And developers are notoriously bad at interacting with business folks. Developers are logical, rational and, at times, binary in their thinking. Business folks like to negotiate with reality.

It's a wonder anything actually gets done at all, sometimes.

So, are there any ways out of this mess? The obvious solution is to have a strong Project Manager who is good not just at making lists of tasks and tracking progress against them, but also who has enough technical and business acumen to be able to translate the technical reality of software development into a cost/benefit analysis for business folks. Seriously, if you can throw a dollar and cents figure onto different technical decisions then you've given something business folks can understand. Or, barring that, at least come up with a generalizable metaphor which they can hold on to.

Alternatively, we could ask that Product folks learn to trust their developers. But I don't really think that's going to happen. Business folks are not engineers and they never will be. And while I began this post talking about how it took two months for my development group to gel into a well running machine of development awesomeness, forming that kind of relationship between developers and business folks takes much, much longer.

And so, when you find yourself working without a project manager the only advice I can give is this: when you need to make a technical decision that has an initial cost (in terms of time) but will give future dividends you need to express that as a non-technical metaphor. Use the word "investment" a lot. Business folks understand investments. They do not understand understand easily extended code frameworks. Tempering the enthusiasm of less experienced developers would be good, though that's not always possible. But having a consensus is generally more persuasive than not. But, probably the most important thing is that you have to remember that business folks can be negotiated with. Engineers tend to accomplish the task that is presented to them, rather than reframe the project as something that can be negotiated. If Product comes to you and asks you how long it takes to build X, you should tell them. But it would behoove you to also realize that X is comprised of features A...N, and many of those may not be required.

And, finally, be aware that business folks tend to be very imprecise with language, whereas years of coding and just a general rational tendency tends to make engineers very precise with language. So when Product asks how long it will take to build X, there is a very real possibility that what they mean is, "how much of X can we build by this (predetermined but not yet revealed) date?"