ublo
bogdan's (micro)blog

bogdan » Stop Searching by Coincidence: Model Management and Workload Prioritization

07:54 am on Jul 27, 2026 | read the article | tags: medium

Building a production-ready semantic search platform requires more than decoupling the application from the vector database. Once machine learning models enter the runtime path, the system acquires another set of operational constraints. Model servers are computationally expensive, can be slow to scale, and often have memory requirements large enough to affect the rest of the platform unless inference capacity is managed deliberately.

In the previous article, I explained how Kubernetes became the portability substrate for the platform and how tenant-aware routing and backend-specific adapters decoupled the storage layer. The same design philosophy needs to extend to the machine learning components because model artifacts are large, startup behavior is expensive, and inference requests do not all operate under the same latency requirements.

This part of the series examines the changing embedding model landscape, the trade-offs behind packaging model artifacts inside container images, and the need to treat search and document indexing as distinct workload classes.

The Moving Target of Embedding Models

Once I accepted that the vector database had to remain replaceable, it became difficult to justify coupling the platform permanently to one embedding model. Model coupling is potentially even more restrictive because the embedding ecosystem is evolving faster than the storage layer.

New architectures continue to appear, benchmark positions change, multilingual retrieval improves, context lengths increase, vector dimensions shift, sparse retrieval techniques evolve, and models become more efficient on available hardware. These changes can affect retrieval quality, infrastructure cost, and the shape of the vectors stored in the database.

There is also no reason to expect one model to dominate every retrieval workload. A product catalogue, a technical documentation corpus, and a multilingual editorial website represent different search problems. A model that performs well on average across a public benchmark may still be the wrong operational choice for a specific corpus, language, latency target, or hardware profile.

This is why I follow benchmarks such as the MTEB leaderboard without treating them as a static purchasing table. Their value is not limited to identifying which model currently ranks near the top. They also show how quickly the landscape changes and how strongly retrieval quality depends on the characteristics of the task.

For that reason, the architecture treats the model server as another routing target. The gateway can send an inference request to a selected model deployment and receive a normalized response that the rest of the search system understands. Snowflake Arctic is currently central to my setup because the platform needs both dense and sparse representations for hybrid retrieval, but the surrounding architecture does not assume that it will remain the preferred model indefinitely.

Model routing creates an infrastructure problem that differs from storage routing. A vector database is stateful because it stores a large amount of derived retrieval data. A model server may be stateless from the perspective of user data, but its model artifact can be several gigabytes in size. This has direct consequences for startup behavior and horizontal autoscaling.

A conventional stateless API container can usually start relatively quickly because its image contains a small application binary and a predictable set of dependencies. Kubernetes schedules the pod, starts the process, waits for the readiness probe, and begins routing traffic once the application is ready.

A model-serving pod may spend much longer in the startup path. When the container downloads its model from remote object storage after startup, the pod can already be scheduled while remaining unable to answer requests. The effective scale-out time now includes downloading the artifact, extracting it, loading the model into memory, and potentially initializing an accelerator.

As a result, the workloads that benefit most from burst capacity can also be among the slowest to scale. Adding a replica does not immediately add usable inference capacity if every new pod must first retrieve and load a large external artifact.

My current approach is to package the model artifact directly inside the container image. This differs from the more conventional separation between application releases and large data artifacts. Keeping the two separate usually makes sense because it avoids rebuilding a large image whenever the model changes and allows model storage to evolve independently from the serving code.

For this platform, however, predictable startup and node-local caching are more valuable than maintaining that separation.

Once a Kubernetes node has pulled the model image, additional pods using the same image can reuse the cached layers rather than downloading the model again from an external location. The container registry therefore participates in model distribution, while Kubernetes nodes provide a local cache for the complete serving artifact.

The resulting image is large, but startup behavior on an existing node becomes more predictable. A pod can begin loading the model immediately from the local image rather than depending on the availability and transfer speed of a separate artifact service.

Packaging the model with the inference code also improves reproducibility. A specific image tag identifies both the serving implementation and the exact model artifact expected by that implementation. Rolling back a deployment restores both together, avoiding situations where an older application release retrieves a newer model revision from a mutable external location.

This approach has visible costs. A fresh node still has to pull the full image at least once. Large images increase registry bandwidth requirements, and every model release requires a new image build. Initial distribution to new nodes may remain slow, particularly when several nodes need the artifact at the same time.

I am comfortable with those costs because they are explicit and because node-local caching improves the more common scaling path in which additional pods start on nodes that already contain the image.

Faster pod startup reduces one source of delay, but it does not by itself solve inference capacity management. Search and indexing still compete for the same model resources unless the architecture treats them as workloads with different operational contracts.

Search and Indexing Have Different Operational Requirements

A semantic search platform should not treat every inference request as equivalent because search and indexing differ in latency sensitivity, throughput requirements, and acceptable completion time.

A search query is part of an interactive request. A person enters text into a search interface, the request reaches the gateway, and the system must generate an embedding before retrieval can begin. Time spent waiting for inference capacity contributes directly to the latency experienced by the user.

Document ingestion has a different completion requirement. A new product, article, or page needs to become searchable, but the embedding and indexing work does not normally have to finish before the HTTP request that submitted the document returns.

The architecture can therefore handle the two paths differently. Search is synchronous and latency-sensitive, while indexing is asynchronous and throughput-sensitive. Both depend on the same inference infrastructure, but they should not receive the same scheduling priority.

Suppose a tenant uploads a large product catalogue and thousands of documents enter the embedding pipeline. Shortly afterwards, another user submits a search query. If both request types are placed into the same first-come, first-served stream, the interactive query may wait behind a substantial amount of bulk indexing work.

The platform can still appear healthy under those conditions. The model server may report excellent throughput, CPU or accelerator utilization may remain high, and operational dashboards may show that every available resource is being used efficiently. Those metrics do not account for the fact that latency-sensitive queries are waiting behind work with a much more flexible deadline.

I therefore treat the platform as having priority tiers rather than exposing one undifferentiated inference path. Search requests receive preferential access to model capacity because they belong to an interactive critical path. Indexing requests can tolerate queueing because the ingestion flow is already asynchronous.

This does not make indexing less important. Once the platform accepts a document, that document still needs to become searchable, and the asynchronous path must eventually converge toward the indexed state. The distinction is that the two workloads are governed by different service expectations.

Search has a latency target.
Indexing has a freshness target.

These contracts should be visible in the architecture rather than mixed into one generic workload queue.

Autoscaling remains useful, but it solves a different part of the capacity problem. Model replicas contain large artifacts, consume substantial memory, and may require expensive or scarce hardware. New capacity may take time to become available even after the autoscaler decides that more replicas are required.

Autoscaling changes how much capacity the system will have. Prioritization determines how the capacity that already exists is allocated when search and indexing compete for it. The platform needs both mechanisms because adding future capacity does not protect an interactive request that is waiting now.

These requirements also clarify the role of the API gateway. Once the system has multiple vector database targets, replaceable model deployments, tenant-specific configuration, and several request priority classes, clients should not be responsible for selecting the correct combination themselves.

The architecture needs a coordination layer that understands the authenticated tenant, resolves the appropriate model and storage targets, and distinguishes between synchronous search and asynchronous ingestion. The API gateway therefore operates as more than a reverse proxy. It becomes a tenant-aware workload coordinator that centralizes placement, routing, authentication, and priority decisions.

In the final part of this architecture series, I will examine that coordination layer in detail. I will explain why I implemented it in Go, how it separates the synchronous search path from asynchronous document ingestion, how NATS carries accepted work toward eventual indexing, and how retries and dead-letter handling prevent processing failures from becoming silently lost documents.