I've tackled legacy systems my entire career, and there is a certain art to untying the knot of dependencies, procedures, and expectations. Not surprised at all that most projects to modernize end up as smoldering craters rather than functioning systems.
The legacy system trap is built on three pillars:
1. Maintenance costs are always lower than upgrade costs. It's even cheaper to pay for neither. Even if you do spend money aggressively to maintain it; every feature added, operator hired, workaround published, becomes a growing suit of armor protecting the system from replacement.
2. Management incentives are rarely aligned with the planning and sacrifices it takes to unwind technical debt. Everyone wants to cut the ribbon on the shiny new big-data/AI/ML/buzzword project, but no one wants to take responsibility for integrating it with the rusting hulks creaking along actually running day to day operations.
3. The software industry can shift paradigms enough that you end up looking to buy a new/faster horse for your buggy. And nothing fits your use case, or even offers a path out.
All three in combination mean you get businesses spending obscene amounts of money on technology they can't use.
It is quite satisfying to successfully maneuver an organization out of the trap. It's like solving a complex puzzle over months. Where hundreds or thousands of real people will be better off because of your team's effort.
> Not surprised at all that most projects to modernize end up as smoldering craters rather than functioning systems.
This is because people think rewriting is a solution. It very never is. It can help, rarely, but it's never the solution, it just does a good job of hiding what the solution was.
If you can't map out what is really broken in your legacy system and map out a way to refactor it to what you want then a rewrite will always fail.
I knew a legacy system where everything was broken.
There were no proper data model and structures, so every computation involved multiple nested loops, making everything much slower than needed.
Everything was modifiable as if it was public, so it was hard to keep things consistent.
Allocation/GC and latency were through the roof, they tried to add pooling but noone could figure out where objects could be released.
One of the core classes was called Banana (changed name) but actually referred to bananas or to banana trees.
And the list goes on (threading, database usage, still plenty to tell).
Then someone, new to both the domain and the language, stepped up to rewrite everything, which took a bit more than a month.
Ten years later the core of it didn't change much, they are still building on it and it's still smooth.
So sometimes the cost benefit of a rewrite outweighs the risk but I'd still say all those things you pointed out were the problem that needed fixing, rewrite really just fast tracked that no?
It was rewrite "from scratch" (not refactoring), but with the previous team available to answer any question (either about the legacy code or the domain, which they knew well), which was an essential part of the success (I guess a lot of rewrite fail because they are done by new people that didn't take care to learn enough from the old system and the domain to thoroughly understand what must be done).
The legacy code was too bad to make refactoring worth it, but surely not every legacy system is in that bad of a shape.
Management sees the new and shiny buzzwords and proclaims the effort a success, while the users see at best a system that barely works as well as the old one did, and often much worse (missing features, bugs, slowness, etc.) That's how a lot of these modernisations tend to go.
It’s painful to see the people who know all the hotkeys and key sequences on an old green terminal suddenly thrust into a world of mouse hunting and clicking. It makes you wonder if new things are really better.
I worked at a bank, we had this old loan system in a terminal, all the LOs could use it fast, little hiccups but we could fix it.
New web-based UI came in (~2004) and it was super customizable and was supposed to be super fast. But nobody new the new hot keys, or there were none.
But it was customizable! So I spent the next few months learning this new architecture, APIs, SDKs for the crap-ware and deployed for our LOs. I got so free beers.
Then I helped another bank with the same issue in their migration - that second go was like free money.
Do you find it depressing working on crappy old systems for lengths of time? I have done enough maintenance programming that I am sick of it. It would be OK if I had the clout to say we need to refactor this part and that part, but it's always new crap slapped on to of the existing mess, or digging around for most of the day to find the line or two that needs changed to fix a bug. I guess I can just see that it's never going to improve. (Though it does sound like you actually get to improve things more than I do).
Legacy is a special form of good: It means something actually works. Good software outlives its original programmers, great software outlives its original programmers and is still maintainable. You can only get to that level if you make a commitment up front that this software is going to stand the test of time, update your systems, don't get side tracked by fluff and eye candy and concentrate on what matters: the core business logic, at the expense of almost everything else. Do that and you stand a chance of avoiding the legacy trap.
A year of complacency and you have a year of catching up to do and you'll never get that year back unless you want to freeze development.
Absolutely I have seen MGMT wearing "innovation" blinders only to realize nothing fucking was wrong in the first place.
Before declaring something legacy, a fair evaluation of the future business requirement and fitment of current solution is a must.
12 Years back I worked with an org who had an AS400 which they considered legacy. System is still running and serves the business need perfectly well. All plans to replace just wasted money.
Another company decided to implement a new centralized ERP without realizing that company's highly federated businesses which can also be considered a conglomerate of very small independent businesses doesn't need it and won't use it. Million went down the gutter before higher ups realized the sad fact and fired the CIO.
IT world is literred up with sorry tales like these.
No, it's just normal case of "the system is still in production and the only reason you're on ancient hardware is because you're not upgrading", same as if you ran business-critical component on random 486SX assembled by then-CEOs nephew.
AS400 is very much alive and stable platform with new release and adequate support available. It just ain't sexy and that was the main driver for declaring it legacy.
As a systems admin of some 30 years, I have read this over and over, and even seen it firsthand. I think the real root cause of (at least in my case, the ones I have seen) of this problem is management. Leaders do not want to spend now. Leaders do not want to make a progressive plan to keep up with technology and move ahead. It costs money, and is very difficult to talk to upper-upper management about. Telling the boss to tell his boss to spend money on something "that works now" is very difficult. Few want to invest in the future when it costs now and is working now (unless its the stock market).
I think the problem is people think of hardware (and software) as a system, rather than a system-of-systems. It works now, because the exact combination of hardware, software, and peripherals available today works. Things you need to interface with use standards that your hardware is equipped with. As soon as that changes, your system needs to upgrade or change to continue functioning. Backwards compatibility only lasts so long, and is different for each interface (hardware and software).
This is a huge part of the problem. The amount of risk that exists in legacy systems is actually pretty huge because of the amount of interdependencies that exist.
It's the sort of thing that people who operate those systems know but that knowledge doesn't seem to spread beyond that.
Needed to increase the NodeJS version from 0.16 to 4.0 because a new shiny thing needed it. I knew it was a major version change but the dependencies didn't exist for the newer node versions... so rewrite something that isn't yours and maintain it forever... or do nothing.
It should be weighed like any non-trivial investment. Present an estimated yearly cost curve for keeping the system versus overhauling it: two curves. Upper managers will probably relate to that better than techno-speak.
Typically a replacement system has a nearer-term spike in costs for development, but is cheaper in the longer run than keeping the old one, because the old one will need gradually more expensive duct-tape, chicken-wire, and harder-to-find technicians willing to support something obsolete.
One advantage of the older system is that the costs are more predictable, because it probably works "good enough" at the time and will likely just gradually "degenerate". An overhauled system could be great or a grand failure.
An exception is if a vendor stops supporting a key legacy component and there is not enough demand for 3rd-party tweakers to take up the slack. Similar can happen if a security hole is found in a key legacy component.
Companies can't track costs that stem from systemic inefficiencies.
Say your programmers still use CVS for version control and you want to convince management to switch to Git. The CVS server doesn't cost much, a git server wouldn't cost much either. There would be a large cost for changing all your tooling and processes that rely on the CVS model and in training your programmers to use the new system. Why would you do that? How do you quantify the benefits of atomic commits or simple branching?
> There would be a large cost for changing all your tooling and processes that rely on the CVS model and in training your programmers to use the new system.
When git seems to be by far the most used system on the market these days, isn't training any new hires in CVS going to be at least as much of a problem? I used it in my first job in 2003 / 2004, but I can't remember how to use it now.
Do companies with legacy code tend to keep staff around for longer?
That example is too simplistic, it's called legacy systems and not legacy software. If you upgrade your tool-chain you include git.
Really no one is interested in upgrading 'just' git, make a package out of it (new compiler, new CI, new testing, new version controll..fuzz etc) call the project "sustainable nextgen tool-chain...short SuNeTuCha" and sell it to the management like that.
> Present an estimated yearly cost curve for keeping the system versus overhauling it
That's the problem, most organizations can't track this.
At a workplace, we replaced a legacy item with something new without doing this analysis... but after that, the move accelerated when it became aware making new features on "New vs Old" was a 2 hour vs 2 week endeavor. The new system covers 70% of the traffic and will slowly be going all the way to 100% of the traffic over the next 1-2 years.
A thing that I found missing is that legacy systems' influence is much larger than only the systems it directly touches.
Through particular requirements that some of these systems have they might shape policy too!
I primarily deal in security assessments of networks. Guess what happens to your password strength requirements if somewhere deep down in a dark and old server room there is a mainframe that can only handle 8 character lowercase passwords? Assume that this system has been the backbone of your, then hyper-modern, completely integrated IAM system for decades?
...
That's right, security policy mandates 'simple' 8 character passwords... Not ideal.
Shudder. That's horrifying enough to make me want to wall off that whole thing in a VPN which in and of itself has a stronger password, and expose access to it from a proxy.
Well there's this fashion of dumping 50+ IT guys. The systems they worked on will then stagger on. The new guys will throw on patches, but there's no money to really understand the system. The life support just gets more and more expensive.
The smart way to migrate a legacy system is to keep your maintenance crew to retirement age. Once they're approaching retirement bring in some kids to migrate to the shiny new tech and use the old crew to make sure they don't drop stuff.
I wonder if in 100 or so years software development anthropologists (do those exist yet?) will study the effects of ageism in tech using historical hiring/layoff data.
I know that large business systems are complicated, and migrating old data is much much harder. But a persistent part of me thinks... is this really so hard? The IRS core system basically requires a decision tree, a calculator app and several bindersful of business logic. They need to process about 350 million people each year, which is a big but hardly unmanageable Spark job. Isn't this just a question of getting a team of good experienced Silicon Valley engineers to build something that just works?
The Obamacare system is a great example of this. Almost half a billion dollars spent on corporate contractors, then a team of like 10 experienced engineers came in and fixed the damn thing.
I think this comment highlights your inexperience in enterprise it systems. There challenges facing enterprise IT are not due to there being no good engineers - because all of them are working in Silicon Valley.
It is a people problem. Every department, every group, entry business unit is its own little kingdom that is fiercely defended by its workers. They cling on to it because that’s all they have done for the last 20 years. They know that if this system gets modernized, then their job is toast. Their job is vapor ware and they know it.
Having worked with many state and local governments, I see that these entities are never in favor of a business process redesign that simplified things, unless the consultants carve out a special exception for their little Access database or somehow save their pre historic C# ASP.Net application.
Every year absolutely mind bogglingly large sums of money are spent on redeveloping existing applications of that level of complexity. These all have a very predictable ending: utter failure, and the legacy stuff ends up living to fight another day.
It is a special kind of blindness: We think that just because our tools and machines are so much more powerful that rebuilding a complex legacy application is now childsplay.
From a UI perspective you are most likely right. But it's the couple of million lines of business logic that will be your Waterloo. In a nutshell: forget about it. A bit longer: it is possible but only by a very experienced crew who has nothing else to do under competent leadership and who can chart an incremental path between where they are now and where they want to be. In 5 years or so. A big bang redevelopment is DOA. Every time.
To add, what makes the business logic so painful is that you don't define the rules yourself. They are the result of legislation, regulations, lawsuits, contracts, and preferred work processes of very headstrong individuals. The codes tries to reconcile all of that to get to a solution. It's not about a great idea, it's about the laws of physics being written by a committee.
I know that the article and your comments are related to "large" and "high" end software, but I see the same things happening on much smaller/simpler software and I cannot but attribute the issues to "wrong" attitude of the developers, that often are (I mean this in a good way) too smart/forward looking/obsessed by scalability/etc. for their own good (actually for the good of the software).
It seems to me like simplicity (and Occam's Razor) are approaches somehow lost or considered defects intead of values.
Personal example, I have a rather simple DOS (yes DOS) program that alway worked (and still works) just fine, it is Clipper/Dbase based, it does what it is supposed to do, with a single terminal, since 1990 or so.
Since it has financial and tax related connections (invoicing) it cannot (shouldn't) fail.
The business has not basically changed since the '70's, all the programs I could find to replace it now are either:
1) "jokes" not even worth mentioning
2) "worse" semi-legacy (think of MS-Access) crappy programs with terrible UI and layers over layers of "added because a customer asked for it" unneeded "features"
3) complex multifunctional multiuser multiwindow web-enabled real-time dashboard cloud connected <insert here any other buzz-word> based on SQL Server montruosities that actually do the 5 or 6 needed things (and 127 other unneeded ones) through a (slightly) different workflow (which is slower) and with data input masks that take longer to fill (compared to the good ol'DOS one)
I know that the #3 programs can scale with multiple terminals, multiple offices, unlimited number of customers, concurrent access and what not.
Still I don't need anything like that, it is - again since the 70's - a 5,000-10,000 customers per year, 20,000 to 40,000 transactions, 3,000 to 5,000 invoices, it won't become 100,000 or 1,000,000 transactions, ever.
Now I have to replace the good ol' Clipper little thingy, due to some changes in tax laws, and I will need to slow down (slightly) the operations, this is progress.
Something I'm not sure is clear from the article is that not all legacy is bad. The definition of the word 'legacy', essentially, is "something that is handed down [through generations]". You can hand down good things and bad.
Banks often have incredibly old systems because their products don't change. A savings account in 1940 is pretty much the same as a savings account today. They basically have no need for DevOps, because they have no need to provide fast-moving, competitive products, and silos are a natural fit in a world of sensitive financial information. So some really old systems are pretty much fine as-is, as long as the hardware can keep running, and a few over the hill programmers are on call.
Another example of useful legacy is old physical infrastructure. The Frankford Avenue Bridge in Philadelphia is a 73-foot long stone bridge that's 323 years old, and carries a substantial amount of car traffic. But it's also not being maintained like it should be, and the more-than-expected wear over time is slowly eroding it. That's the bad part of legacy systems: when you know they need maintenance and they don't get it, so they rot.
Modern infrastructure can suffer the same fate. Concrete bridges, wooden houses, cars... they all need maintenance. So do modern computer systems, where they need to get regular security patches. So the problem is really just a maintenance problem, and we happen to let older systems go to hell more often than we replace newer ones.
A savings account in 1940 is pretty much the same as a savings account today.
The way customers interact with their bank has changed massively over that period. In the UK traditional banks are losing customers to new upstarts that don't have to bear the cost of all that legacy. They are nimble and much more responsive to their customers.
Software lifetime is an interesting topic. See for example [1], a 1992 paper that surveyed business applications and found an average lifetime of about 9 years, with a maximum of 20 years and a minimum of 2 years. 2 years versus 20 years!
Things don't seem much different today.
On the one hand, HTTP, HTML, and producing HTML server-side in a language like Java or Python and with a SQL database isn't all that different now than it was 20 years ago. There are Java web apps developed in 1999 that are kept up-to-date and work just fine in 2020. On the other hand, there are applications developed with popular technologies a just a few years ago that are hopelessly out-of-date and in need of a rewrite today.
I see it as an end user too. I've been an ING Direct (now Capital One 360) customer for 18 years. The web app that I use in 2020 seems to do the same things as the one in 2002 (though it's slower today), but it seems like it's been completely rewritten at least three times.
Which might be a good thing if the loss of knowledge came from employee turnover. If the architect / author leaves, the visions leaves too. Next step is monkey patching, fast forward a couple years, a decade or two, more turnover, and you end up with a indescribable mess.
In this case the replacement is all about re-gaining the knowledge.
How much do legacy systems affect the success of a business? Are there clear cases where companies with or without legacy systems are doing any better or worse? I feel like both ways seem to work with the right amount of knowledge and leadership.
A lot would be gained by treating software like a pet rather than as a building. Regular brushing and care, occasional picking up of errant poop, vaccines, checkups with the vet, and the knowledge that Fido will only live about 10-15 years.
A salesman will call anything that is not his product "legacy" so the word can get confusing. Also there is an issue when commercial vendors start to overcharge, and their software needs to be replaced with some thing cheaper. This can become a higher priority than replacing COBAL.
The legacy system trap is built on three pillars:
1. Maintenance costs are always lower than upgrade costs. It's even cheaper to pay for neither. Even if you do spend money aggressively to maintain it; every feature added, operator hired, workaround published, becomes a growing suit of armor protecting the system from replacement.
2. Management incentives are rarely aligned with the planning and sacrifices it takes to unwind technical debt. Everyone wants to cut the ribbon on the shiny new big-data/AI/ML/buzzword project, but no one wants to take responsibility for integrating it with the rusting hulks creaking along actually running day to day operations.
3. The software industry can shift paradigms enough that you end up looking to buy a new/faster horse for your buggy. And nothing fits your use case, or even offers a path out.
All three in combination mean you get businesses spending obscene amounts of money on technology they can't use.
It is quite satisfying to successfully maneuver an organization out of the trap. It's like solving a complex puzzle over months. Where hundreds or thousands of real people will be better off because of your team's effort.