What's new for serverless LLM usage in 2025?

Serverless computing has changed the way teams deploy and scale machine-learning workloads. With LLMs, the pay-for-what-you-use model is especially powerful: you save GPU spend by ensuring that you only pay for the inference time rather than an entire pod.

Why continue to spend for GPU time when it is just going to sit idle? Not only that, but serverless allows you to scale seamlessly up to spikes in demand with a minimum of fuss. We are leaning hard into serverless and want to share what we have created with you.

Newly Released Features

This release brings faster cold starts, more flexible worker sizing, and clearer visibility into queue depth. These changes reduce latency and make it easier to run larger models without over-provisioning.

Increased 80GB GPUs per Worker

You can now request workers backed by 80GB GPUs. This is ideal for larger models or longer context windows that previously required splitting across multiple workers.

Serverless SGLang Quick

We have added a one-click template for SGLang, making it faster to deploy structured-generation and multi-turn workloads.

Improved Model Selection Screen

The model picker now shows compatible runtimes, expected pricing, and a direct link to the model card, so you spend less time searching and more time building.