Crisis culture in software development part 1: diagnosis, prognosis, and progression

In this first of two articles on crisis culture, we describe the building blocks of software project failure, improper remedies’ stoking effect on organizational trouble, failure’s transformation into crisis, and how success despite the odds makes crisis normal, if not permanent.

“Crisis mode is the most expensive way to operate.” —Anyone who’s ever worked in manufacturing

Failure and crisis are not the same thing. In software, failure’s a combined objective and subjective judgment: it doesn’t work, it doesn’t make money, it bricked customer hardware, etc. Failures happen. Crisis, a time of difficulty, can perpetuate or result from failure, but crisis is a mode, and it becomes part of your organization’s character. In software development, crisis often becomes the norm.

In this first of two articles, we’ll describe strategic pitfalls lurking in software development and their suppurating, chronic symptoms. Note that understanding the problems driving crisis culture is almost more important than implementing their solutions—but most common cures make the disease stronger.

70%+ of all software projects fail

According to IAG Consulting circa 2014, 68% of all software projects become either a marginal or complete failure. By 2021, that number had increased to 77%. The primary cause? Poor requirements, of course.

Knowing the cause is one thing for a prospective patient, but the mechanism of failure matters more for its resolution. Many failing projects feel successful, but become runaways, costing far more money and time than expected. Others became parasitic, draining resources and generating little revenue, still in the wild on a desperate hope for some return on investment. Many failures are both.

Software failure paralyzes companies who need a success to increase competitiveness or remain relevant. Paralysis manifests three ways:

• Conservatism. Businesses naturally respond to low returns by decreasing investment.
• Half-measures. Reduced investment robs development teams, leading to applications built entirely on "good enough".
• Unrealistic expectations. Tales of Silicon Valley's shining stars help set wildly unrealistic expectations amongst stakeholders and shareholders.

Although many pundits blame poor or vague requirements solely for that atrocious failure rate, there's no sure-fire approach to building successful software. Perfect requirements can create perfect garbage.

70% failure effectively means that 70% of technology expenditures go up in flames. 70% failure/30% success = trial and error.

It’s true that large, successful tech companies take risks and embrace failure, but they embrace it as a badge of honor for having tried something and a mark of courage of admitting that it didn’t work. Larger tech companies rarely put their going concern’s profitability at risk to try something new. Moreover, they’re extravagantly-well funded and smartly-configured specifically for risk-taking in digital.

Few other businesses are. It's past time to stop emulating tech companies:

Silicon Valley and Big Tech methods work—for them—in their business environment. You probably don’t share that environment. Fortunately, there's no one correct way to create software despite industry orthodoxy. You need to behave differently to create your desired impact more often.

Software development crisis manifests itself in mostly subtle, corrosive ways. Perversely, some organizations don't feel like they're making software unless they're feeling the toxins listed below. Success, at that point, becomes a matter of chance. While not all projects experiencing symptoms are consigned to the 70%+, projects experiencing these (or similar) symptoms must be evaluated, monitored, and traced back to their core issues. Primary and secondary symptoms may include:

Friction

• Angry or intrusive stakeholders
• Blame game
• Broken promises
• Opaque statuses
• Endless meetings
• Skepticism about digital
• Inaccurate estimates and underfunding
• Unreasonable demands
• No rest or observation periods

Hurdles

• Delays and schedule changes
• Budget overruns
• Resource raiding
• Turnover
• Cut corners
• Crippling compromises
• Overstretched or over-idle resources
• Oppressive deadlines
• Missing features

Inflexibility

• Limited extensibility
• Weak integration points
• Poor performance
• Ugly
• Clunky
• High change cost

Absent RoI

• Customer complaints
• Hubris-driven consumer insight
• Confusing preferential data for behavioral data
• Massive fall off in adoption after initial wave
• No plan after launch
• Updates are bugfixes only
• No post-launch analysis
• Sidelined user testing
• No market traction

Recidivism

• Rejection of new methods
• Burning through vendors
• Falling back on mainstays
• Creating "like-for-like" functionality
• "Replace" instead of "rethink"
• Outdated technology
• Dependence on hyper-specific experts

Over our careers, we have encountered organizations with all of these symptoms!

Unfortunately, improper fixes nurture small sparks of failure into raging crisis infernos. The common remedies described below rarely help and accomplish nothing at best, and they exacerbate, mask, or distract from core issues at worst. Should they help, they temporarily relieve the symptom without touching the problem. Many symptoms have no direct remedy, leaving sufferers with only their personal determination to plow forward and tackle the next project. Typical solutions to primary and secondary symptoms include:

Friction

• Deflection
• Increased reporting and planning
• Budget brinkmanship
• Overcommitment
• Meetings about meetings
• Further blacken the black box
• Buy time
• Go harder

Hurdles

• Add labor
• Add process
• Add tools
• Reward performance with more work
• Justify increased expenditure
• Outsource critical systems
• Procrastinate and hope

Inflexibility

• Limp along
• Panic
• Lash out at third parties
• Rush rework
• Rearchitect without regard to technical debt

Absent RoI

• Do-over/throw-away
• Limp along
• Cut budgets
• Decommission without attempting resolution
• Zombify the application
• Report feedback into the void
• Declare the execution a disaster, not the idea
• Abandon vision, enter the cycle of reaction
• Oversimplify and assign blame

Recidivism

• Declare digital pointless
• Meet, debate, repeat
• Cut funding
• Build Leviathan

View each of these actions as holdovers as reflexive, industrial-era holdovers fit for a different context. Software development is not World War-era factory work, but that doesn’t stop us from treating it as such!

Software development teams handle problems every day. On their own, neither the symptoms we've described nor their common solutions inflict acute organizational trauma, particularly when the software in question actually gets delivered. Serious calamity arises once symptoms and treatments fuel each other and spiral out of control—issues swell from simple problems or roadblocks into genuinely worrisome malignancies, ultimately snowballing into a job- or funding-threatening fury.

Like a fire needs oxygen, fuel, and heat to burn, a project issue needs three ingredients to perpetuate itself:

1. Stakeholders watching
2. Money being spent
3. A broken expectation

Once an issue ignites, businesspeople add something to resolve the broken expectation, justify the current expenditure, or tidy up the project's political veneer:

• Tools
• Labor
• Process
• More money
• Binding promises

Unfortunately, additions target symptoms, increase the size of the crisis, and perpetuate the real issues:

• Larger project cost
• More stringent expectations
• Less time
• More staff and onboarding overhead
• Deepened scrutiny
• Raised stakes
• Higher stress

Projects may make it through delivery after this occurs, but more often, another iron barrier (time, budget, functionality) gets crossed during delivery, leading to:

• A broken promise
• Intense scrutiny
• An ultimatum: succeed or else
• The quest for the silver bullet

Bringing the tragedy to its inevitable denouement:

• The silver bullet fails
• Team practices regress
• Project participants make the best of the worst
• The inquisition comes (sometimes)
• Political fallout
• Collateral damage

Then, when things reach a boiling point, organizations repeat the cycle.

Crisis cultures arise from repeated chain reactions.

Any organization that suffers the chain reaction multiple times has essentially been tortured. After every visible rationale for repeated failure has been explained away, there's only one thing left to blame—talent—and many organizations believe they haven't got it, churning through people after they’re out of ideas. At this point, there’s nothing an individual manager or contributor’s personal performance can do to overcome the cycle of failure.

This is why short tenures and frequent layoffs became normal in digital. It’s why excellent, skilled people get fired on a whim. It’s how their attitudes sour. It’s how their motivation evaporates away. And it’s why so many tech people want out.

It gets worse. By their very nature, atmospheres of crisis both breed and necessitate heroism. In software, visible heroes emerge while grinding away at seemingly insurmountable objectives:

Grinding

• Reactive instead of proactive
• Constantly shifting workstreams
• Regular rework
• Endless firefighting
• Very little visible progress, fleeting sense of achievement

Insurmountable

• High pressure
• Moving targets
• Insane deadlines
• Detrimental legacy code
• Limited by existing systems

In this setting, anyone who finishes anything becomes eligible for the hero title. Heroes are born at that moment where a developer (usually a developer) who possesses the requisite expertise cultivates a sufficient sense of ownership, then seizes an unexpected amount of the project's destiny on his or her own volition, demolishing barriers and functioning as the creative generalist needed to glue the project together.

Crisis cultures rely on the heroism, sacrifice, and multi-disciplinary skill of their star performers to produce software.

When that hero generates a project success, he or she positively reinforces the status quo. Woefully ironic given that the status quo was precisely what they overcame to create the success.

As more heroic successes occur, organizations become conditioned to rely on heroes, placing ever larger responsibility for project success upon heroes' shoulders, further stoking the hunger for disposable technical talent, finally leaving the most advanced organizations where they are today:

• Skilled developers viewed as a magic talisman against failure
• Impossible job descriptions
• An arms race driving high turnover, discouraging long-term investment, preferring individual- to team-level talent, isolating heroes further

If you’ve wondered why software development has hardly changed in 15 years, the combination of perpetual crisis and heroic successes ossified it to this point. Many organizations have taken themselves hostage.

Talent alone will not make software projects successful. All the force in the world doesn't matter if it’s misapplied.

Designing, coding, and organizing aren’t the whole job.

Think of success development as a separate project that exists alongside a given software development project. Software teams may build the software, but they also have to build success and insure against failure.

As the boss, you need to help them do it.

In part 2, we’ll outline the steps you need to take to fix crisis culture in software development. If you need immediate assistance, contact us for rapid firefighting!