12GB ML container is cold start tax

If you bake the weights of your multi-giga model into a fat CUDA image, you’ll pay a pull tax on every GPU scale-out. Lazy loading isn’t the cure you think it is. Removes weight from the image.

you COPY Adding 16 GB weight to a CUDA image causes all autoscaled GPU nodes to re-pull those bytes before a single token is provided. That is the cold start tax.
Delayed pull snapshotters such as SOCI deferred mode, stargz, and nydus are useful when a container operates on only a small portion of its bytes at startup. ML inference is different. They often read almost the entire image before serving it.
SOCI Parallel Pull mode is a faster full download path for workloads that require the entire image immediately. Do not confuse it with lazy loading.
The structural fix is to take the model weights from the image and stream them from S3, EFS, or another model store at startup.
Slim the remaining runtime images with multi-stage builds and a minimal base to measure your own cold start path.

Scaling out at 2am that didn’t help anything.

A burst of traffic hits the inference service. HPA starts, Karpenter provisions a new GPU node, and the scheduler places vLLM pods on that node.

Then everyone is waiting.

The node is placed there and pulls the container image that bakes the 8B parameter model into layers. 16 gigabytes of weight and CUDA runtime are sent over the network before the process starts. Several minutes passed while the image was unzipped and CUDA was initialized, and the pod did not process any requests.

The traffic spike that triggered the scale-out is already halfway through.

This is why it is a tax and not a one-time expense. It will be repeated.

All new GPU nodes in that scale-out event will re-pull the same fat layer. Scaling from 4 pods to 12 pods across 3 new nodes pays three more parallel weight-pull penalties for the same registry when latency is most important.

The instinct is to reach for lazy loading: “Just stream the images on demand. Don’t grab everything up front.”

When it comes to ML inference, that instinct is often wrong.

People who take snapshots lazily are really smart.

Seekable OCI (SOCI) is a good example. The mechanism is simple. Create a table of contents for each layer, mount the image through a lazy file system, and fetch the compressed span only when your workload reads the corresponding file.

This design is good when the container starts up while accessing only a small portion of its bytes.

A web application may start immediately after reading its configuration, some libraries, and a small runtime path. Lazy loading reduces startup delays since most image bytes are not needed immediately.

ML inference works differently.

The vLLM server does not dump images. Before responding to the first prompt, read the CUDA library, initialize the runtime, and load the model weights into GPU memory.

If the model weights are in the image, there is no small “boot subset”. The boot path is the model.

Lazy loading does not resolve the stall. This relocates the stall from the image pull time to the first inference time. Concurrent fetches can make the initial request feel even worse, as the read path becomes dependent on on-demand remote fetches.

I traded the visible pull time delay for the invisible initial request delay, and called that an optimization.

SOCI has two modes. Do not confuse.

This is one of the most common cold start mistakes.

There are two different ideas.

SOCI lazy loading: fetch-on-read, FUSE-based. Useful if you want to work with only a small portion of the image at startup.

SOCI parallel pull mode: Use concurrent pulls and parallel unpacks to perform faster, complete downloads.

The second is important for AI and ML because inference often requires the entire image at once.

SOCI Parallel Pull does not pretend that a workload can avoid reading most bytes. It just downloads the entire image faster.

While this is useful, the download of the weights originally baked into the image is still optimized.

The bigger lever is to stop putting weights there.

remove weight from image

The fastest weight layers to pull are those that are not in the image.

A better production pattern is to keep the image lean and use an early step, sidecar, CSI volume, or model loader to fetch weights from S3, EFS, EBS, or another model store at runtime.

This is what the two Dockerfiles below actually change.

# ❌ Antipattern: model weights baked into the image FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04 RUN pip install vllm COPY ./llama-3-8b/ /models/llama-3-8b/ CMD ["python", "-m", "vllm.entrypoints.openai.api_server", "--model", "/models/llama-3-8b"] # ✅ Better pattern: image carries runtime only FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04 RUN pip install vllm ENV MODEL_S3=s3://my-bucket/llama-3-8b ENV MODEL_DIR=/models/llama-3-8b ENTRYPOINT ["/bin/sh", "-c", "\ aws s3 sync $MODEL_S3 $MODEL_DIR --only-show-errors && \ exec python -m vllm.entrypoints.openai.api_server --model $MODEL_DIR"]

The first image requires all autoscaled GPU nodes to get the model weights as part of the container image.

In the second image, the container only holds the runtime. Weights are fetched individually and can be cached at the node level.

This changes the operating model.

The model update will be a new S3 prefix rather than a complete image rebuild.

Two pods on the same node can reuse the same local weight cache.

Image distribution is smaller and faster.

Model storage and runtime packaging are separate issues.

simple aws s3 sync This example stages the weights to disk before loading them. Faster paths can use model loaders, CSI drivers, or streaming mechanisms that reduce or avoid double-hops through disk.

The key principles remain the same. Separate model weights from runtime images.

Streamline the remaining runtime

Retrieving weights from S3 or EFS fixes maximum cold start blocks. Bloated runtime images are not fixed.

CUDA development images often include compilers, headers, build tools, and libraries that are required during build rather than runtime. These bytes will continue to be pulled on every new node.

Use two old but effective practices.

Multi-stage build: Compile with the builder image and copy only runtime artifacts to the final image.

Minimal bases or distroless bases: If your workload allows, remove shells, package managers, and unnecessary operating system surface areas.

The final image should contain only what is needed to start the inference server.

Second, optimizations such as SOCI Parallel Pull are even more helpful as they speed up small artifacts rather than huge artifacts.

The right stack is not one silver bullet. It’s a chain:

Removes weight from the image.

Cache weights near nodes.

Streamline the runtime image.

Accelerate the remaining pull.

Measure your actual scale-out path.

honest trade-off

This design is not free.

Moving weights out of the image swaps out self-contained artifacts and runtime network dependencies. Pods may not start if S3 is throttled, the VPC endpoint is configured incorrectly, the credentials are incorrect, or the model store is unavailable.

You also need discipline in IAM boundaries, cache invalidation, model versioning, and clear runbooks in case of download failures.

Fat images have one big advantage. That is, everything you need to run it is packaged into one artifact. This may be required in air-gapped, compliance-focused, or offline environments.

However, for autoscaling GPU fleets under real-world traffic, the tradeoff often points in a different direction. The iterative cost of retrieving model weights during scale-out can dictate the usability of a single large image.

Don’t decide this based on ideology. Please measure.

Get the actual layer size. Time the actual node provisioning pass. Measure model download time, image pull time, CUDA warmup, first token latency, and cache reuse. Next, optimize the parts that actually irritate your users.

takeout

Cold start taxes are recurring increases. Fat model layers are paid again on each new GPU node during scale-out.

Lazy loading is not a universal solution. Useful when startup amounts to several bytes. ML inference often requires a complete model quickly.

SOCI lazy loading and SOCI Parallel Pull are separate tools. One is retrieved on demand. The other speeds up full downloads.

Weights usually need to exist outside the image. Stream from S3, EFS, EBS, or a model store and cache near your nodes.

Streamline the runtime image. Use a multi-stage build and a minimal base to make the rest of the image really lightweight.

Measure your way. Benchmark claims vary based on model size, node type, network, storage, and registry behavior.

Stop asking how to download 16GB faster.

Ask yourself why 16GB is included in the image in the first place.

The fastest layers to pull are those that didn’t exist before.