ublo
bogdan's (micro)blog

bogdan » A Few Tips on Architecture and Attention

08:58 pm on Sep 20, 2026 | read the article | tags: ideas, medium

I like thinking about software architectures. I guess you already knew that, if you checked around here lately. I like thinking about large distributed systems and how that can take form in practice: boxes connected with arrows, queues, databases, caches, workers, clusters, and all the nice things that eventually wake somebody up at 3 in the morning.

Almost every job I’ve had forced me to do infrastructure work. Sometimes because I was the only person around, sometimes because I was managing a team and tried to protect them from the operational mess, and lately just doing it under my official job title, MLOps. The problem never changes much: you have something that needs to work and not enough people, time, or patience to operate everything that could theoretically make it work.

There’s a difference between liking something and being good at it. I’ve made my share of bad architectural decisions. But, more importantly, I had to live with them. Now, after all these years, my idea of what makes a good system has changed. I care less than I used to about finding the «best» component, and much more about what happens after the choice is made. After the benchmarks are forgotten and somebody has to upgrade it, recover it, or debug it at 2:00 a.m.

Tip #1: Knowledge is part of the infrastructure!

One of the first large systems I operated was a very popular radio station website receiving hundreds of thousands of requests per minute. I chose Apache HTTP Server. There were plenty of reasons why that was a bad idea at that scale, but Apache had one massive advantage that didn’t appear in any benchmarks: I knew it very well (at least back then). I knew how it failed, where the config files lived, which knobs were dangerous, what the logs meant, and when something broke I already had a full mental list of stupid things to try.

The launch wasn’t without incidents, but most took minutes to resolve rather than hours because very little was surprising. Once things settled down, I evaluated alternatives and eventually replaced Apache with Nginx, which was the better solution. Apache was probably the better first decision. There is a difference.

We usually compare technology as if the people using it don’t exist: throughput, latency, features, cost, GitHub stars. But knowledge is part of the infrastructure too. A component that is worse on paper but whose failure modes you understand can easily be cheaper than a perfect component you don’t know how to recover.

This isn’t at all an excuse to stop learning. There is still a minimum technical bar a component has to pass. Yet after that threshold, I no longer ask only whether a component solves the problem. I ask how expensive it will be for us to understand it when it doesn’t.

Tip #2: Architecture Belongs to the Operator

If I’m working alone, using something I know inside-out has enormous value. In a team, using something only I know becomes a liability.

Let’s say I use a Mac and everybody else uses Linux. I can write beautiful automation around whatever BSD-flavoured utilities happen to be on my machine. It works great until I go on vacation. Maybe rewriting it in JavaScript adds a runtime where a shell script would have been enough, but if everybody on the team can understand and maintain it, JavaScript may actually be the simpler system.

Architecture doesn’t belong to the architect. It belongs to whoever has to maintain it after the architect gets bored, promoted, fired, or simply goes offline for a week.

A system with five dull components understood by ten people can be much safer than a system with three brilliant components understood by one. This doesn’t mean choosing the lowest common denominator forever, but novelty has a cost. Introduce a new language, database, deployment model, and orchestrator in the same project and you may have built a training program disguised as software architecture.

Tip #3: Choose Small Contracts over Large Features

Years ago I worked on the foundation for dozens of news websites. The obvious choices were systems like Joomla or Drupal, which could do almost everything. That was exactly the problem: they could do almost everything, and brought their own abstractions, plugin systems, conventions, and assumptions about how the rest of the application should work.

Instead, I picked a much smaller CMS. It couldn’t do much, but it had clean boundaries around what it did, which meant we could attach other systems for streaming, encoding, publishing, and whatever strange requirement appeared next. That small core eventually became the basis for more than 60 projects because it didn’t insist on owning the entire architecture.

There’s this temptation to choose the component that covers the largest possible surface of the future. It sounds responsible. Except next year’s problems have an annoying habit of being different from the ones we imagined (remember LLMs?), while we pay for all the hypothetical features today in configuration complexity, upgrade pain, and coupling.

No worries. I don’t hunt for simple implementations as PostgreSQL and Linux are anything but simple. I hunt for small contracts and small operational surfaces.

Tip #4: There’s always the (hidden) cost of Human Attention

Qdrant is a good recent example. There are vector databases with better performance for particular workloads and products with more features. What I like about Qdrant would make a data scientist less excited: I can run it locally without reproducing a small data center, clustering is understandable, snapshots are straightforward, and recovery doesn’t require learning five other systems first.

Every component creates work twice: once when you build the system, and then forever. Upgrades, monitoring, backups, security patches, incidents, onboarding, and there’s also that strange problem that shows up every time there’s a team-building and nobody remembers how to fix.

Automation belongs in the same calculation. A mediocre component with good operational automation can easily be cheaper than a better component that requires humans to recover it.

Compute is expensive because AWS sends you a bill every month, yet human attention is easier to miss because nobody sends a sanity invoice when three engineers spend two days diagnosing an unhappy cluster.

Human attention is part of the infrastructure cost. It just doesn’t appear on the same cloud invoice.

One of my former CEOs used to say that with money, anybody can solve any problem. I don’t think that’s quite true. With money, many capacity problems get solved, but constraints don’t disappear because you have a larger credit card. Sometimes throwing 10x the compute at a rare spike is exactly the right answer.

Money buys margin and time, but it doesn’t automatically buy understanding. Throwing 10x at the problem just because nobody understands why 1x stopped working is something else.

Tip #5: The harder it is to undo, the more time it deserves

I recently watched two teams tackle vector search. One spent months understanding the workload, testing locally, simulating production traffic, and identifying bottlenecks, so a year later their setup has scaled with no major incidents. The other mostly added compute when things became slow. Incidents started almost immediately and eventually affected the release of a new long-sought feature.

Engineers, me included, like product questions because they have concrete answers. Which vector database? Which queue? Which LLM gateway? The boring questions matter more. How much data is there? How often does it change? How bad is stale data? What happens during a ten-minute outage? What’s the actual p99 we need? Who gets called when it breaks?

There is rarely a best component for a category. More often, there is a component that behaves better under a particular collection of constraints.

The other question is how painful a wrong decision will be. My Apache choice was relatively easy to undo because the web server sat behind a clean boundary. Changing the way an entire company represents customer identity is something else.

Architectural effort should be spent in proportion to how difficult a decision is to reverse. Move quickly on reversible choices and do thorough analysis on the ones that will hurt later.

And now, the conclusion

Maybe the useful question is not «What is the best architecture?» but «What is the cheapest architecture we can understand, operate, and change while still satisfying the actual constraints?»

In distributed architectures, cheap doesn’t mean the smallest cloud bill. It means attention: how much of the system somebody has to keep in their head, how many specialists are required when something breaks, how many people can safely change it, and how much new knowledge is required before touching it.

For a large company, some of this can be purchased. There’s a database team, infrastructure, security, SRE, and somebody whose entire job is understanding one particularly angry Kafka cluster. For a team of five, there’s Bogdan. And Bogdan doesn’t feel like working today.

Once something is technically good enough, I increasingly optimize for a different set of things (here’s a nicer list-like set of tips):

1. Prefer things you understand, unless the unfamiliar thing buys you both lunch and dinner.
2. Keep contracts small, so changing your mind doesn’t require changing everything.
3. Automate the things that happen twice, because twice quickly becomes twenty times.
4. Spread the knowledge around, and worry about anyone who’s always too busy to write some docs.
5. Spend analysis on decisions that are painful to reverse, and yes, YOLO everything else.

The goal isn’t to design a perfect architecture from day one, but to build a system you can run safely today, while leaving enough room to change your mind tomorrow.

Because eventually, the most expensive architecture isn’t the one with the highest cloud bill. It’s the one you can no longer afford to understand.