On Complexity and Monoliths

A Realistic Perspective on Microservices and Other Architectures

H.P. Lovecraft, in his short tale The Nameless City, quoted a fictional poet with these words:

> "That is not dead which can eternal lie / And with strange aeons even death may die."

In this context, I’m applying this idea to monoliths rather than to mythical deities lurking beneath the waves, waiting to wreak havoc.

This piece reflects my personal opinions. For a more technical and objective examination of this topic, I recommend the following insightful books:

• Fundamentals of Software Architecture: An Engineering Approach
• Software Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures
• Building Microservices: Designing Fine-Grained Systems

Clarification: Microservices should not be equated strictly with Kubernetes. I consider function-as-a-service (FaaS) a more pure form of microservices. However, I will use the term "microservices" broadly in this article.

Final caution: I will not debate the use of monoliths in scenarios where they are justifiable, such as small projects, prototypes, or specific security contexts. While monoliths can be a valid architectural style, careful analysis of the software's purpose, quality attributes, and the relevant domain is essential.

Reality check: There is no guarantee that progress will occur.

Bad ideas often resurface after a period of dormancy.

One such idea is the monolith. Every idea can appear reasonable from a particular viewpoint. Yes, there are valid reasons to use them occasionally; however, I want to explore the fundamental issues at play:

The challenges associated with “monoliths” extend beyond technical aspects (i.e., being a single deployable unit)—they often stem from inadequate problem decomposition, which leads to issues like poor scoping and communication.

How Did We Arrive Here?

#### Syntactic Prisons and Golden Hammers

Picture a recent graduate or self-taught individual eager to enter the software field. They may know the syntax of one or more programming languages, but can they design effectively? Likely not. Syntax alone does not create software.

> "If I can encapsulate all the code in one package, wouldn’t that be the best choice every time?"

Building architects often lament that seeing a project through—from initial design to completed structure—happens infrequently, resulting in a slow feedback loop and delayed mastery of the craft. Cutting the ribbon on a new building is a luxury many do not experience.

I harbor doubts about the role of software architects—many are absent from larger tech companies—yet I firmly believe in the principles of software architecture and design.

However, the art of software design seems to be fading, even in an era where coding is faster than ever.

> "Software is not bound by physical constraints like buildings; it is limited by imagination, design, and organization. Ultimately, it is constrained by human qualities rather than physical realities." — Martin Fowler, “Who Needs an Architect?”

Software emerges from socio-technical processes. Programming alone does not yield software. Reflecting on (Melvin) Conway’s Law, we find:

> "Any organization that designs a system will produce a design whose structure mirrors the organization’s communication structure."

One could easily draw parallels to microservices. Perhaps the technology isn't the issue? I suspect that many organizations and individuals who are critical of microservices lack the necessary conditions for success. Often, such organizations display characteristics of Taylorist management and produce subpar products due to insufficient customer engagement and reliance on vanity metrics.

Pain and discomfort can lead us to withdraw, freeze, or entrench the very behaviors that brought us here. We often hear cries: “Microservices caused this mess; I want my golden hammer back!”

Given that “we cannot solve problems using the same mindset that created them”—a sentiment often attributed to Albert Einstein—might it not be rational to revert to something “simpler” that “worked”? Let's explore why monoliths might seem appealing today.

Monoliths Are Not the Solution

Kamil Grzybek’s piece on modular monoliths is insightful. However, I contest that a monolith is unnecessary; it only complicates otherwise sound principles of structure and decomposition. You can incorporate these architectural traits into a FaaS setup as well.

The commonality we share—illustrated well by his DDD project—is that with thoughtful software architecture (i.e., modules), we can break down the system into coherent segments. This is advantageous.

Thus: If you are criticizing microservices, you might be missing the fundamental point.

Microservices do not imply that every Lambda (or any FaaS you consider) is entirely self-contained and detached from modular or class-oriented interactions. My own Get-a-Room DDD project and my online DDD book illustrate this well. They are not arbitrarily deployed like satellites; they exist within a context, typically a service, complete with its own interfaces.

Doing this thoughtfully yields independent deployability, a reduced error surface, and an accurate infrastructural representation of your system—without necessitating a monolithic deployment, which Kamil rightly identifies as the hallmark of a "monolithic" system.

Even within a monolithic framework, you might employ asynchronous communication with service/event buses to relay commands and queries to other modules. This is where the irony surfaces: Why is this not more logical and intuitive within a FaaS environment?

A modular monolith remains a single deployable unit, albeit with clear code separation. While that is not inherently bad, we risk conflating infrastructural elements with application code—an undesirable outcome!

Let me clarify a painful truth: Those who struggle with microservices often also fail with their “modern” architecture, communication, and separation of concerns. After all, what are microservices but infrastructural manifestations of these more abstract concepts?

How could a modular monolith—or any monolith for that matter—be simpler or more comprehensible than a clearly defined, independent entity?

Microservices, coupled with logical decomposition and Domain Driven Design, can yield a much more expressive solution with minimal added complexity.

In other words, I question:

> "How can a large, undefined entity be a superior architecture compared to a small, well-defined one?"

I genuinely wonder.

The more measured approach reminds us of the eight fallacies of distributed computing and the various ways network communications can falter: Increasing dependence on the network can make the overall system slower and more fragile. While this is somewhat true, every architecture is essentially “the least worst one,” allowing even extremists to weigh whether the benefits of separation and independent infrastructure outweigh their drawbacks.

High Complexity—The Slow Death

As a young person, I was misled into thinking that software complexity was primarily mathematical or otherwise frightening, which I detested. Much later, I realized this wasn't accurate—the complexity is linguistic and processual. Since most software is enterprise or business-oriented, the key challenge is accurately translating processes into code.

If one believes that “complex code” is an inherent result of “algorithms” and “mysterious black boxes,” let me dispel that notion: Good code should be understandable in its syntax and structure (encompassing both system design and application architecture), with sufficient documentation to clarify processes not readily apparent in the code. Good code is straightforward, predictable, and unexciting.

Without delving too deeply into the distinction between “bad” and “good” code, concepts such as cyclomatic complexity and coupling are immensely useful for understanding software complexity. Throughout my career, I’ve found my background in English and the humanities has proven far more beneficial in navigating semantic and organizational challenges than my technical training.

Spaghetti code is a manifestation of poor programming and reflects a lack of understanding regarding what is actually being modeled. There may be roles for mathematical geniuses and data scientists who use programming in a broad sense, but for the rest of us tasked with building systems, spaghetti code is our greatest adversary.

Words carry weight, and if you fail to recognize that, you should reconsider your career choice. The fact that the classic book Clean Code emphasizes this is disheartening.

The most crucial terms are those related to requirements (or similar concepts).

Many modern organizations expect developers to create something they don't fully grasp, and for engineers to assert they cannot test something because they lack clarity on how it should function—even if they constructed it themselves! This isn't complexity; it’s foolishness.

Due to a lack of understanding around experienced complexity or stupidity—often stemming from incomplete or illogical requirements—many organizations mistakenly believe developers can solve (business) complexity based on any information set, regardless of its quality. Unfortunately, this is a misconception.

Enterprise software can never surpass the quality of the model it represents and the manner in which it is articulated. Cognition, competency, focus, and various other factors will diminish that initial value as soon as the process and requirements interact with reality. Be prepared!

I recently encountered the term “logical debt,” which succinctly encapsulates the issues at hand.

In theory, we seem willing to engage in discussions about essential complexity—the nature of the issue itself—but in practice, this often fails to materialize. Instead, we encounter complication or even accidental complexity, which emerges from our approaches to problem-solving (leading to the creation of additional challenges). Rather than addressing the root problem (which we struggle to define), we allow artificial urgency and other factors to dictate our actions, resulting in both logical and technical debt.

> You might find my article on technical debt enlightening:

From Technical Debt to Technical Health with HealthCheck

Why software fails and how you can practically address it with a six-step plan

betterprogramming.pub

A pivotal question arises: Are we overcomplicating matters? Complication, as opposed to complexity, involves problems with known, predictable solutions. Are microservices inherently more complicated? Indeed, in some respects. However, this is not necessarily true if you leverage FaaS or similar frameworks. Yet, there are clear advantages.

Perhaps the most significant benefit of microservices is their alignment with Gall’s Law:

> "A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system."

Simple, “dumb” components succeed, especially when they have well-defined responsibilities. I contend that a more explicit, semantic, and clarified approach to software creation and deployment is preferable. Microservices facilitate this process.

Due to the coarse-grained nature of a monolithic codebase, monoliths are more prone to accidental complexity. Let’s be honest: complexity isn't merely a minor irritation—it can jeopardize your work, career, and even others' lives. Numerous projects fail due to complexity (including accidental complexity arising from poor planning, communication, and more), making it challenging to choose just a few examples, but here are some notable mentions:

• Windows Vista
• Boeing’s 737 Max disaster
• Tesla’s self-driving feature
• The UK’s National Health Service
• NASA’s Challenger space shuttle disaster
• Knight Capital Group’s “glitch” that cost over $400 million
• The Swedish Öppna skolplattformen (“open school platform”)
• Countless SAP implementations (if sources are to be believed)

Reports suggest that the majority of software projects fail, which seems plausible given various anecdotes and internet findings.

For further insights into complexity, consider Fred Brooks’ seminal paper “No Silver Bullet: Essence and Accident in Software Engineering” (1986) and the Cynefin Framework.

Complexity is not an abstract concept; it is tangible. Just ask anyone who writes code.

In my view, engineering revolves around minimizing unknowns and complexity. If this is not the outcome, then it isn't true engineering. Hiding system details within an undefined black box is a poor strategy.

The monolith embodies a swift regression, yearning for simplicity but fundamentally struggling to manage complexity. Yes, a monolith may have fewer seams; however, seams exist to differentiate and define various needs. Naming is one of the two “hard things” in computer science, and this is still simpler compared to software design: transitioning from chaos to microservices requires extensive naming, separation, and differentiation. While deploying a monolith may seem easier in some respects, each modification can impact everything else. Not ideal.

It is overly simplistic and anti-intellectual to assert statements like “you are not Netflix” or “your system is not complex.” In reality, nearly everything in computing is distributed—you already operate within a distributed system!

Monoliths possess different technical characteristics than microservices, but the challenges associated with microservices surface due to unresolved complexity and unclear mental frameworks. Critiques of microservices may reveal deeper issues within the organization itself.

People and their organizational structures (i.e., teams) are pivotal to the DevOps model. Logically, there is no reason why all microservices couldn't be managed by a single team. The contention arises from the concept of "sharing." Why is it problematic to establish systems with boundaries? Why not ask individuals to take responsibility for different components?

The same applies to those who argue that microservices are challenging or ineffective because co-deployment may be necessary—you’ve entirely missed the crucial task of isolating your service! This isn’t a microservices issue; it’s a personal one!

If you’re reconsidering your stance, here’s a hard truth: unless your organization can adapt to distinct, primarily non-overlapping products, features, or concepts, then yes, microservices might not be easy to manage. Nevertheless, they may be preferable (as a transitional phase) to a chaotic monolith, as they offer benefits that I firmly believe in. It could very well be worth the effort.

The Loss of Language and Its Implications

We tend to overanalyze software while undervaluing everything else surrounding it.

Code exists to address problems. If we cannot articulate the problem and our desired outcome, we should neither propose nor create a solution.

I suspect most mismanagement and process-related issues arise from a failure to clarify the problem and an insufficient regard for changing circumstances.

If, heaven forbid, the code itself becomes the problem, we find ourselves in dire straits, burdened with twice the issues. This situation is bound to occur, as the code “does something” but remains undefined (not in the positive “agile” sense).

“Business” will never voice concerns about the initial “unknown/unknowable” problem (which is intangible), only the new one you've inadvertently created while trying to be a “Good Boy” and fulfill your duties.

Zoom out to see the bigger picture.

I believe the failure of our education system to foster critical and rational thinking contributes to this dilemma. Our training programs for BAs, engineers, and other IT roles are lacking. Jim Aho's article is a testament to this.

Business Analyst — one of the most harmful roles in software engineering

When I say Business Analyst (BA), I mean a person whose primary purpose in the company is to translate business needs…

blog.steadycoding.com

I am deeply concerned about the decline in our educational standards. The loss of critical thinking, contextual action, initiative, and even language (poor grammar correlates with poor code) will not enhance our chances of creating high-functioning organizations, products, and solutions to real-world problems.

Big Tech recognizes the importance of these skills, as Irina Stanescu noted on LinkedIn:

> "In my 14-year career in engineering working for Big Tech companies like Google and Uber, no skill has been more vital than writing. I don’t mean coding; I mean English writing—emails, design documents, presentations, feedback, code reviews—you name it."

Like Dante's descent into Hell, we will witness more evidence of this deterioration and its repercussions for software engineering. Prolonged ideas that fail to transition from concept to reality, merely resulting in PowerPoint slides and urgency-driven “task forces” that dissolve without consequence, exemplify this state of affairs.

The acceptance of individuals who can barely fulfill their roles leading organizations is a disheartening reality, one that defies quantification.

Let me share two of my own experiences, even if they are not quite as extreme.

#### The “Simple” Mega Task

Years ago, I spoke with a colleague involved in project management who struggled to grasp Kanban. After some reflection, he suggested:

> "Can’t we just make it one task: ‘Finish the thing’?"

#### The Diagram with a Literal Black Box

I once had to label an architecture diagram as a “Donald Duck diagram” because it was intended to illustrate a key feature but resembled little more than a cartoon sketch. It wasn't a proud moment, but sometimes you must choose between candor and politely nodding along to your own demise. Time was of the essence, yet this was the best the “great minds” could produce!

We have since redesigned and operationalized that area, but it serves as a reminder that poor outcomes emerge when the wrong individuals are granted autonomy. It would be refreshing to hear the other side of this narrative amidst all the overly optimistic DX research being conducted.

Note: This is not a defense of ARBs or ITIL, but rather an acknowledgment that some individuals may need to consider a different career path.

Without language and supportive structures, we cannot resolve the complexity dilemma. I have yet to encounter a professional capable of articulating domains, models, boundaries, and technical interfaces who would wish to revert to a monolith (the Amazon Prime case being a notable exception) once they transition to microservices.

Moreover, I rarely see critiques demonstrating a genuine understanding of the professional knowledge outlined in key literature within our field. It is virtually impossible to ascertain the preparations individuals undertook for microservices, but I struggle to believe that all these failures would exist if adequate groundwork had been laid.

The ability to work with microservices and contemporary distributed architectures is fundamentally reliant on applying skills that have always been integral to a software engineer's role. Our industry has suffered due to 10–15 years of ultra-niche job descriptions and short “get an IT job” programs, resulting in individuals of diminished expected competency.

Big tech firms prioritize your social and communication skills through code reviews, system design tasks, and solution discussions. Aren't they? Developing distributed systems is no longer an “offbeat” experience in 2023. All these skills are learnable, but most of them will be acquired outside of direct coding practice. The introverted keyboard warrior was not the archetype we aspired to in software engineering.

Someone must take a stand: Why have our organizations and hiring managers tolerated this decline? The loss of literacy and vital human qualities to mere syntax crunching jeopardizes our prospects for a viable, professionalized trade. Robert C. Martin even asserted that software engineering does not merit being labeled a “profession” in a strict sense. After all, what do we profess? It’s a fair question, considering our collective inability to competently perform the “engineering” aspect in light of our interactions with stakeholders.

Endnote: A Simpler Future

Complexity and the self-help industry share similarities. Ironically, self-help books or programs (even seemingly benign ones like Bullet Journaling) often present a core value or aspiration as desirable, claiming their guidance will help you achieve this end—be it time, money, energy, sleep, fitness, or any other goal.

In our somewhat flock-like and mundane human behaviors, we tend to calendarize and structure this work in accordance with our newfound philosophy or spontaneous ideas. Yet, most self-help strategies ultimately prove ineffective. I believe that, in many instances, self-help authors genuinely wish to offer their frameworks—what worked for them! Maybe even for others. However, psychology is a complex field (in the genuine sense). Whether it works for you often depends on luck, persistence, and personal fit.

Most programs will claim they are easy or something similar to make sense within context. However, they invariably add more tasks—do more of this, do more of that. I have long favored a reductionist approach and find inspiration in Zen Buddhism (though I have never formally followed it).

What I strive for is a principle of less: minimize choices, stick with a single haircut, wear essentially uniform clothing, and become freer as a result.

The personal benefits I gain include conserving energy and being able to allocate it where it truly matters—on my projects and, in terms of hours, significantly enhancing my capacity to engage in activities that I enjoy, which is not the case for many people juggling work, children, and various commitments.

Do not let your enthusiasm for complex problems drown in the merely complicated and undefined.

The answer to reducing complexity lies in moving towards fewer components and more precise definitions of identity (what the “thing” does). Avoid creating what you can obtain off the shelf. Do not build what you do not comprehend, cannot illustrate, discuss, or explain. While a monolith may superficially have fewer components, this is because we conceal them within a black box, lacking transparency.

Take a cue from design literature—let's aim for a broader vocabulary, considering concepts like affordances (Donald Norman) and recognizing how limitations can enhance our work. Clearly define the problem and simplify the solution to address it. The tech industry is too eager for novelty, which often harms us more than it helps; however, microservices are not simply a passing trend; they represent a lasting infrastructural expression of distinct responsibilities.

Lastly: A tombstone is also a monolith. They tend to come with an obituary.