Skip to main content

​
Common strategy on setting up deployment

  • GPU type + GPU count = performance per replica
  • Replicas = traffic capacity and availability
Start with the smallest setup that reliably meets baseline needs, then scale deliberately based on real traffic and observability:
1

1 replica set up: Choose a GPU type and gpu_count that meets single-replica performance goals

2

Autoscaling set up: Use replicas to absorb traffic spikes and concurrency

​
Choosing performance setup: GPU types & GPUs per replica

Choosing the right dedicated endpoint configuration depends on your workload’s priorities: latency, throughput, model size, and cost efficiency.

​
GPU type

GPU type determines the performance profile of each replica, including memory capacity, throughput, and cost. In general:
Rule of thumb: Choose the smallest GPU type that supports your model and region. If performance is insufficient, first scale GPU type or GPUs per replica before aggressively increasing replicas.

​
GPUs per replica (gpu_count)

gpu_count defines how many GPUs power a single replica. This primarily affects:
  • Per-request latency
  • Maximum throughput per replica
  • Ability to serve larger or more demanding workloads
Rule of thumb: Start small when you are testing traffic, usage is uncertain, internal deployment, cost control matters. Scale up when requests are latency-sensitive, individual replicas saturate, you need stronger single-endpoint performanceAvoid Over-relying on replicas to solve underpowered replicas. More replicas help concurrency, but they do not fix poor per-request latency caused by insufficient GPU resources.

​
Choosing number of replicas

Replicas determine how much baseline and burst traffic your deployment can handle.

​
Min replicas: guaranteed baseline capacity

Minimum replicas are always allocated while the endpoint is active. Use higher min_replicas when:
  • You need predictable low latency
  • Traffic is steady
  • Cold starts are unacceptable
  • Capacity predictability matters
Use lower min_replicas when:
  • Traffic is irregular
  • You optimize for cost
  • Batch or internal workloads
  • Occasional warm-up is acceptable

​
Max replicas: burst scaling ceiling

Maximum replicas define how far autoscaling can expand if capacity is available. Use higher max_replicas when:
  • Traffic spikes significantly
  • Usage is unpredictable
  • You need burst headroom
Use lower max_replicas when:
  • Workload is stable
  • Budget control matters
  • Capacity is predictable
Rule of thumb:
  • min_replicas - what you always need
  • max_replicas - what you may need during peaks
Note that:
  • Min replicas are guaranteed while active. If you stop the endpoint the reservation will be freed up.
  • Max replicas scaling depends on available capacity and may not always be continuously available after scale-down. Read more in Capacity section