The Promises and Perils of LLMs in Building End User Software

It is undeniable that coding agents via large language models have been revolutionary this past year. As technologists, we’ve been grappling with their impact on our craft and the products we deliver, even if the wrestling match is far from over with the current pace of progress. What is plain to see, however, is that our profession is forever changed by the existence of these tools—as is the final software we end up producing with them.

Our transition into a new mode of work has been fast but not without friction, and modes of adoption have run the gamut. Those of us who work in infrastructure have generally been more cautious with increased usage of LLMs, given the more “mission-critical” aspect of the discipline and the complexity of the development tools themselves. But as the harnesses and models for agentic coding have improved, this class of infrastructure developers has seen amazing value accrue to it, precisely because of the rigor and robustness that exists in its ecosystem.

A fantastic example is the Rust programming language, where a draconian borrow-checker and feedback-rich compiler can fuel an LLM to write correct, idiomatic code. This has been so extraordinary that even people who have never written a line of Rust are now relying on LLMs heavily to write Rust programs.

What is happening up the stack for higher-level software and end-user applications points to a very different story. With a lower barrier to entry in development and immeasurable business opportunity, it is a world where performance and the appreciation of the minutiae in computing have largely been set aside to prioritize the delivery of applications. In turn, we have a community that has rapidly adopted LLMs for writing software, with many declaring that coding is dead and that it was just a means to an end.

Indeed, LLMs have thrived in app and web development. Many demos touted by AI labs included web servers or jaw-dropping user interfaces (at first glance, anyway). Deep learning itself has been a field steered by implementations in high level languages like Python, reflecting AI’s understanding of computer programming.

For the most part, LLMs have greatly benefited the lives of app developers. Many end-user experiences are amalgamations of CRUD operations integrating with external APIs. Logically then, the programs we create are well-served by AI to identify and work off of the patterns, engineering standards, and best practices of our codebases.

However, the gung-ho reaction of app-development communities and companies to embrace agentic development has not come without cost to developers and their software. Irresponsible use of LLMs has led to many services shipping exceedingly fast but with immense bloat: high memory consumption, redundant network roundtrips, and poor UX are all symptoms of poorly written applications. Technical pivots are a dime a dozen; when the ever-pleasing analysis of LLMs makes rewrites fast and cheap, work becomes embedded with half-baked ideas resulting in mediocre products.

Fears of being out-shipped by competitors and falling behind the rest of the industry—valid concerns when traditional engineering moats have diminished—have fueled this fire while giving us little chance to reflect on our values and industry-wide best practices.

I have seen few companies take the time and effort to refine their mindset when it comes to LLM use, with Oxide Computer Company being a notable example. I appreciate Oxide being a pioneer in how it has set and shared its stance on LLMs publicly; it is obvious that Bryan Cantrill and company have thought long and hard about how AI impacts all areas of their work, and they have a productive stance on how it should continue to be used moving forward.

But Oxide is focused on infrastructure—real computers in privately-owned clouds. The app-development space has suffered from its lack of rigor and thinking on LLMs as tools, and is in dire need of guidelines. In order to forge a perennial relationship with AI, we need to understand the negative consequences in end-user software and tie those to the values we hold dear; as individuals, as companies, and as an industry.

Rapid, Violent Context Switching

The mental overhead cost of being able to run multiple coding agents at once is not unique or novel to the application developer, and in fact is a cost we must pay for some of the greatest benefits that come with LLMs!

But said cost is magnified to mind-aching levels when building end-user software; it is just too easy to work on many independent features and initiatives at once, especially when the opportunity cost and risk of each task appears so low. I believe this ties back to the fear of not going fast enough, with your competitors shipping before you and leaving you in the dust. Moreover, organizations that have adopted a rapid pace of AI-based development now have implicit expectations of how fast certain tasks should take: in a pre-LLM world, serious planning around a big architectural update or new features could take weeks (or months if your company name starts with “Micro” and ends in “soft”). Now the demand is that such planning should take a fraction of the previous time with the implementation being faster still.

So we continue to offload more and more to AI for the sake of speed, finding ourselves with more and more time sitting on our hands. “Unacceptable!” we yell to ourselves. “This is supposed to be a software factory!” We then proceed to exchange the quiet thinking time about our current work for starting up another prompt and cranking out a brand-new task to completion.

Excuse my facetiousness, but any developer today immediately empathizes with this scenario—we have all been there! If we can do more, then why not do more? We would be irresponsible to not take advantage of such a situation.

And this all leads to the burnout of context-switching. Our minds can only keep juggling between complex tasks for so long that we inevitably shift decision-making and ownership to the LLM, whether we like it or not. The final result produces software we do not have mastery over, which in turn means applications with more volatile functionality, design, and performance. The technical term for this, I believe, is AI Slop.

The great Fabien Sanglard talks about context switching in his fantastic post about adopting LLMs, where he provides no clear remedy except to keep an eye on it:

The greatest difficulty I have encountered is "context switching". Working on multiple projects / independent features allows me to drive multiple agents simultaneously. It is quite a mental gymnastics to keep up. I have seen reports of "mental burnout" and I have also personally experienced increased mental fatigue. This is definitely something to monitor.

The Metastasis of Application Logic

As we hand over decision-making and control of our software to AI, an emergent property manifests itself in the code: the amount of logic constantly metastasizes! LLMs cannot tire (although you may run out of tokens), therefore it will not know when to stop with little human control and judgement.

The challenge becomes evident if you enable auto code reviews from tools like Claude or Codex, which can always find something new to pick on. It’s not about just meeting the user requirements in a plan—it is an inherent characteristic of LLMs to keep producing more content. It can always point something out like a weird, technically plausible edge case that, depending on how you squint at it, is worth a large rewrite that sacrifices long-term maintainability.

Intrinsically, this is not a problem for AI working in a vacuum. It doesn’t have to worry about maintainability as humans do and having to create “mental models” of the software; excellent proof of how LLMs are fantastic tools for analysis of large systems in the first place!

So the software accretes, and developers have less power in managing it themselves. They turn back to LLMs to get a handle on the problem and the cycle repeats itself. But this only happens when you choose to hold yourself at least somewhat accountable for your end-product. What happens when we let go of that responsibility entirely?

Garbage Software

“Lines of code written” has never been a metric worth boasting about, and only those who live on the first peak of the Dunning-Kruger curve decide to do so while looking down at the rest of the world. Sunny Balwani of Theranos infamy proudly proclaimed to have written “one million lines of code” while at Microsoft (he didn’t) and Elon Musk used it as a guideline for which engineers to keep in the traumatic Twitter takeover.

I am not sure what changed with LLM adoption for people to come back to these boasts as if it were different this time around. But unfortunately it is being used as a proxy for how fast you ship and how “AI native” you are. This incentivizes those who don’t know better (and those who obviously should!) to let language models take the wheel and drive them off a cliff.

When all responsibility is forfeited, there is no limit to how bad the software we produce can be. Ironically, it is not even bad in a cosmetic sense—after all, LLMs are fantastic at writing idiomatic code while following standard conventions! It does not look like the work of an inexperienced developer. On the contrary, it is as if a sage and proficient programmer became deeply misguided and wrought a terrible plague on a codebase.

And this is what is so insidious of the garbage that is produced. An application can be dysfunctional, have conflicting rules and permissions, break in ways that are obvious to human operators but not so obvious to a machine. Databases and querying models can amass poor performance with no clear solution in sight. LLMs will keep hammering away at the code with more code as the owners of that software are actively de-skilled with little to do.

This ultimately takes us to a dark place: critical thinking becomes disposable. Developers degrade into fungible operators with AI making all decisions. The expectation of software quality by end users will continue to decline until our craft is known for nothing else than producing the computing equivalent of sewage.

Our Software Reflects Our Values

In order to avoid this tragic fate, I believe it has never been more important to contemplate the values we hold for our work. Our projects and companies will unavoidably have conflicting values, or at the very least certain values weighing more heavily than others.

This is what I appreciate about Oxide’s approach to the use of LLMs. Every stance on how AI impacts each discipline naturally blossoms from a subset of the company’s values, stated in order of importance. It provides practitioners with an operating philosophy that stands the test of time, no matter how good language models get.

What is left is a call to action for other companies—especially those building end-experiences for users—to reflect on their own values, genuinely. We need good answers on knowing how to wield AI as a tool, and our values are a virtuous place to start.