Many years ago I asked a developer which was a bigger deal:
- Having an outage
- Missing a deadline
They answered: "oh having an outage is WAY worse". I then asked: "if that's the case, why do you push so hard to hit your deadlines with code you know and I both know is probably not ready?"
They didn't really answer at the time but it eventually dawned on me what's happening:
- odds of being yelled at if you miss a deadline: 100%
- odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%"
Therefore, they are "gambling" rationally by pushing for deadlines vs pushing out good code.
The point I'm getting to is that if the goal is "hit the deadline" vs "deploy code that runs in production for two weeks with no errors", QA is going to be less of a priority.
Couple this with the fact that many firms think of QA as "they have to be cheaper than devs!" and then compensate them accordingly, means that QA people are fighting down both the comp front and the incentives front.
I've seen this happen so many times that I'm not really sure why people are surprised by it any more.
(NOTE: you could say "well then testing should be automated" but you get into a similar argument on who is building and maintaining the testing frameworks).
> - odds of being yelled at if you miss a deadline: 100%
> - odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%"
To reframe this, the odds of a dev having to crunch to hit a deadline they're behind on is 100%, but the odds of any developer catching a support escalation or on-call page from an outage are usually way less, especially on larger teams, because it's rare for every dev to always be on the hook for escalations. That's why things like goalies and pager rotations exist; the perception is that saddling one person with the responsibility occasionally is better than splitting it to everyone all the time. One weekend of abject hell a month is better than four weekends of annoyance.
But when any developer can shirk ownership of an outage, they all effectively do. Even from a support perspective, that doesn't even make me mad — who wouldn't want to sleep in, ignore Slack on weekends, and not feel dread every single time your phone pings with a notification?
On the other hand, teams _never_ let developers off the hook when there's a deadline that might slip. If you don't have something to do, you're pairing off to help someone else who does, or if you can't then you're more likely to be working on the next thing down the pipe so there's not as much deadline pressure, than supporting on stuff (like tests! and docs!) that won't be considered tech debt until someone (probably support!) hits something related to it and calls it out later.
Dedicated QA doesn't lift the outage ownership problem, it helps mitigate it before it happens. But QA teams that deflect outages struggle to provide data-driven reasons for their existence, because they can't track a negative, and credit for _n_ 9s of uptime is always split multiple ways such that nobody gets a big piece. QA winds up forever underappreciated because their wins are so much harder to count, but the times QA causes a deadline to slip are _always always always_ flagged.
Nevermind that outage response pulls engineering resources off hitting deadlines... so that becomes a self-perpetuating cycle...
The best route is to never have deadlines. Just convince sales and marketing of this and you're golden. /s
That (developers only being responsible for a fraction of bad rollouts they personally cause) reminds me of the water dynamic at a lot of apartments I've rented:
For "reasons" the water meters are per building rather than per unit, but the landlords are adamant that residents have to pay their fair share to ensure water isn't wasted. The scheme envisioned to meet those goals is that the total cost of water for a building is averaged out across all units (perhaps normalized by unit size). Looking at the net effect however, using $X of water only costs $X/N because my personal excess is split between the rest of the residents. Consequently, the entire building uses and pays for substantially more water than they would if the meters were more finely distributed.
This is a great description of the dysfunction in development work management. I've always said that deadlines are just made up numbers and the work will get done when it's done. However after leading a team I see how that attitude can lead to a ton of bike shedding instead of prioritizing and shipping features.
This is the classic case of development velocity pitted against operational stability. Entirely different incentives, and when vested into the same role that role is bound to prioritize one incentive over the other. For this reason I think they must be separated at least by body if not by team. They certainly need to separated by different managers.
I tend to think QA is perhaps well situated alongside Ops("DevOps"), and very close to Product + Design.
We were a small and good team developping hardware/software combos.
We ruined a demo to a client by promising some hard to do feature, and during the demo, the said feature had not been well tested for a particular environment.
The debriefing of that failure was memorable. The big boss was yelling at us, saying we should do better, work harder, longer, whatever was required to succeed.
When this calmed down, I only asked one question: when going back to my desk, should I work on this new feature promised to some other customer, or test this old one for any combination of inputs/environments?
The response was: "You do both"
I insisted that I will do both, but which one first?
He responded with some blabla I do not remember, but no response to my question.
To any manager which cannot decide between feature and stability:
If you cannot prioritize, the dev will do it, with whatever information/incentive they have. You may not be happy of the result.
But that's just bad management. It's most of the time better to drop/delay some features, than compromising stability. If the manager doesn't get that, he/she is the problem. And not the software developer.
OTOH, a manager doesn't see outage - they give you an assignment and expect you, the developer, do to a good job. Outages are not because you missed a deadline, but because you didn't do the job that is specified on your job description.
I'm sure you can make analogies. If you get a new kitchen installed, you expect it to be done properly, if it's finished within the day or so they quoted for you but the doors fall off, you won't be happy. You never SPECIFIED that the doors should be firmly attached - you assume they will be, because you trust in the competency of the people installing it.
That's the bad management I'm referring to. I've seen a lot of bad management, but never actually that bad.
If software fails it's a team failure. QA is equally responsible for that, not just developers. Quite often there root of the failure is ambiguous specification (so whoever did that is responsible too).
And in the end the manager is also responsible, because he didn't do his job right (picking suitable people and coaching them, so they can do the job).
- Having an outage
- Missing a deadline
They answered: "oh having an outage is WAY worse". I then asked: "if that's the case, why do you push so hard to hit your deadlines with code you know and I both know is probably not ready?"
They didn't really answer at the time but it eventually dawned on me what's happening:
- odds of being yelled at if you miss a deadline: 100%
- odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%"
Therefore, they are "gambling" rationally by pushing for deadlines vs pushing out good code.
The point I'm getting to is that if the goal is "hit the deadline" vs "deploy code that runs in production for two weeks with no errors", QA is going to be less of a priority.
Couple this with the fact that many firms think of QA as "they have to be cheaper than devs!" and then compensate them accordingly, means that QA people are fighting down both the comp front and the incentives front.
I've seen this happen so many times that I'm not really sure why people are surprised by it any more.
(NOTE: you could say "well then testing should be automated" but you get into a similar argument on who is building and maintaining the testing frameworks).