Large Model Service Architecture Design: Autoscaling, Load Balancing, and Failover

Large model service architecture design builds stable services around inference instances, scaling policies, and traffic entry points. It explains the collaborative mechanism between autoscaling and load balancing, helping assess cross-availability-zone highly available inference solutions and failover boundaries.

What Problems Does Large Model Service Architecture Design Need to Solve

Large model service architecture design means planning a service system that can stably provide inference capabilities externally, centered on inference instances, resource scaling, traffic entry points, and exception recovery capabilities. For cloud-based large model inference services, architecture design usually needs to answer three questions at the same time: how access traffic enters, how backend instances adjust as the business changes, and how to reduce the impact when local failures occur.

In this system, autoscaling is used to automatically adjust the amount of computing resources according to policies or business requirements, which means adjusting the number of instances required by the service; load balancing sits between client requests and backend inference instances, and is responsible for providing a unified entry point and distributing traffic; failover focuses on the recovery capabilities of the entry layer, instance layer, and availability zone layer under abnormal conditions.

Scope note: This article focuses on general architectural ideas for cloud-based inference services, including autoscaling, load balancing, cross-availability-zone high availability, and basic fault recovery. It does not cover specific performance metrics, cost formulas, GPU scheduling details, request retry policies, or service mesh implementation.

How Autoscaling Adjusts Inference Resources Based on Business Requirements

Autoscaling is also commonly called Auto Scaling. Its core role is to automatically adjust the amount of computing resources according to configured policies. In the context of large model inference services, it can be understood as follows: when the business requires more or fewer inference instances, the scaling mechanism manages changes in the size of the instance pool.

The Role of Autoscaling in Inference Services

The backend of an inference service usually consists of multiple instances that jointly handle requests. Autoscaling does not directly process individual requests; instead, it is responsible for maintaining the available instance pool.

FocusRole of autoscaling
Number of instancesAdjusts the number of instances required by the business according to policies
Service clusterManages a group of inference service instances through a scaling group
Resource formCan be used for scaling management under different instance forms, such as ECI or ECS type scaling groups
O&M complexityReduces the operational cost of manually adding or removing instances one by one

In existing cloud-based solutions, scaling groups can be used to manage inference service clusters and support adjusting the number of instances such as ECIs. For different instance forms, ECI type scaling groups and ECS type scaling groups can remain consistent in working principle: both are managed around scaling groups, policies, and changes in instance count.

The Relationship Between Scaling Policies and the Instance Pool

The key to autoscaling is not merely “adding machines,” but ensuring that changes in instance count have a clear basis. When designing an inference service, at least the following questions need to be clarified:

  • Which conditions will trigger scale-out or scale-in;
  • How newly added instances enter the service cluster after scale-out;
  • How to avoid affecting instances that are currently handling traffic during scale-in;
  • Whether changes in instance count can be detected by the entry layer in a timely manner.

The last point usually needs to be linked with load balancing. The scaling group is responsible for “how many instances there are,” while load balancing is responsible for “which instances requests are distributed to.”

How Load Balancing Becomes the Unified Traffic Entry Point for Inference Services

Load balancing sits between client requests and backend inference instances, and is the unified entry point through which the inference service provides access externally. Its value lies in the fact that clients do not need to directly perceive changes in each backend instance; instead, they access the service through a unified entry point.

The Position of Load Balancing in the Architecture

A simplified large model inference service can be divided into three layers:

LayerMain responsibilitiesRelationship with other layers
Traffic entry layerReceives external requests and distributes them to backend instancesUsually handled by load balancing
Scaling management layerMaintains the number of instances according to policiesUsually handled by a scaling group
Inference instance layerActually runs the inference service and processes requestsManaged by the scaling group and connected to load balancing

Cloud-based solutions often use scaling groups together with load balancing. When a scaling group adds new instances, the new instances can be included in the unified entry point to handle traffic; when instances are reduced, the entry layer also needs to update the backend instance set accordingly.

The High Availability Value of Managed Load Balancing

Managed load balancing services usually have redundancy and high availability, making them suitable as a stable entry layer. In some cloud-based implementations, if a load balancing component fails, related components can be automatically restarted or replaced, thereby reducing the risk that the entry component itself becomes a single point of failure.

In addition, elastic load balancing can be combined with scaling groups to automatically bind a load balancer to instances in a scaling group, reducing manual operations when newly added instances are connected to the entry layer.

How Autoscaling and Load Balancing Work Together

Autoscaling and load balancing have different responsibilities, but they need to work together in inference services. A scaling group manages the number of instances, while load balancing provides a unified entry point and distributes traffic. Only when the two cooperate can newly added instances truly have the opportunity to handle requests.

Typical Collaboration Process

  1. Set scaling policies: Determine when to adjust the amount of computing resources according to business requirements or preset rules.
  2. Maintain scaling group instances: The scaling group adds or removes inference service instances according to the policies.
  3. Connect to load balancing: Newly added instances enter the backend service set; when instances are reduced, they are removed from the backend set or stop handling traffic.
  4. Traffic enters the inference service: Client requests enter through the unified entry point and are distributed by load balancing to available inference instances.
  5. Continuously adjust the instance pool: As business requirements change, the scaling group continues to maintain the number of instances, while the entry layer continues to distribute requests to available backend instances.

Example: Combining an ECI Scaling Group with ALB

Existing cloud-based solutions provide a way to combine ECI scaling groups with Application Load Balancer ALB to build cross-availability-zone, elastic, highly available large model inference services. In this model:

  • The ECI scaling group is used to manage the inference service cluster and adjust the number of ECIs;
  • ALB serves as the application traffic entry point and directs requests to backend inference instances;
  • Cross-availability-zone deployment helps reduce the impact of an exception in a single availability zone on the overall service;
  • The linkage between the scaling group and load balancing enables changes in instance count to work together with traffic handling capacity.

Viewing the Collaboration Relationship from Architectural Layers

Architecture layerTypical componentsDesign focus
Traffic entry layerApplication Load Balancer, managed load balancing componentsWhether redundancy, high availability, and backend instance access capabilities are available
Scaling management layerScaling groups, scaling policiesWhether the number of instances can be adjusted according to business requirements
Inference instance layerInference service instances, container instances, or cloud server instancesWhether they can be managed uniformly and handle entry traffic
Availability zone deployment layerMulti-availability-zone instance distributionWhether the impact of an exception in a single availability zone is reduced

Implementation Boundaries of Failover in This Architecture

Failover first depends on designs in the architecture that avoid single points of failure, such as cross-availability-zone deployment, redundant managed load balancing components, and backend instance pools that can be managed by scaling groups. For cloud-based inference services, failover should not be understood as a single function; instead, it should be broken down and evaluated separately at the entry layer, instance layer, and availability zone layer.

Automatic Recovery of Load Balancing Components

Managed load balancing components usually have redundancy and high availability. If a load balancing component fails, related components can be automatically restarted or replaced. Such capabilities mainly address the availability of the entry-layer components themselves.

Application-Layer Fault Governance Needs Separate Design

Load balancing and autoscaling are not equivalent to complete application-layer fault tolerance. The following capabilities usually belong to more detailed service design and need to be evaluated separately in specific systems:

CapabilityIs it within the confirmed scope of this articleDescription
Load balancing component redundancyYesManaged load balancing can have redundancy and high availability
Automatic recovery from load balancing component failureYesComponents can be automatically restarted or replaced after a failure
Cross-availability-zone deploymentYesCan reduce the impact of an exception in a single availability zone
Request retriesNoBelongs to application-layer or client-side policies and needs separate design
Circuit breaking and degradationNoBelongs to service governance policies and needs to be determined based on the business
Model instance warm-upNoRelated to model loading and inference cold start; this article does not provide a definitive discussion
Session persistenceNoRelated to specific request patterns and business state, and needs separate evaluation

Key point: Entry-layer high availability, instance pool elasticity, and cross-availability-zone deployment can improve overall resilience, but they cannot replace application-layer design for retries, degradation, warm-up, and state management.

Typical Components of a Cross-Availability-Zone Highly Available Inference Service

A cross-availability-zone highly available inference service usually consists of a load balancer, a scaling group, inference service instances, and multi-availability-zone deployment. Its core goal is: when traffic changes or local infrastructure exceptions occur, the service can still remain available through a unified entry point and a manageable instance pool.

Key Components

ComponentMain role
Application Load BalancerProvides a unified access entry point and directs requests to backend instances
Scaling groupManages the number of inference service instances and supports scale-out and scale-in
Inference service instancesActually run the model inference service and process requests
Cross-availability-zone deploymentReduces the impact of an exception in a single availability zone on the overall service
Managed load balancing componentsImprove entry-layer availability through redundancy and automatic recovery capabilities

Common Requirements and Solutions

User requirementSolutionExpected effect
Significant access traffic fluctuationsUse a scaling group to adjust the number of instances according to policiesThe instance pool can expand or contract as business requirements change
Newly added instances need to handle trafficCombine the scaling group with load balancingNew instances can be included in the backend set of the unified entry point
The entry layer needs to be stable and availableUse managed load balancing with redundancyReduces the impact caused by entry component failures
Concern about an exception in a single availability zoneAdopt cross-availability-zone deploymentReduces the impact of an exception in a single availability zone on the overall service
Want to reduce manual O&MLink elastic load balancing with the scaling groupReduces the operational complexity of connecting instances to and removing them from the entry layer

Checklist for Designing Large Model Inference Services

Before launching or evaluating a large model inference service architecture, you can use the following checklist to review elasticity, entry, and fault recovery capabilities.

Scaling Capability Check

  • Whether scaling policies and the conditions that trigger instance count adjustment have been clearly defined;
  • Whether the correspondence between the amount of computing resources and the number of instances required by the business is clear;
  • Whether a scaling group is used to manage inference service instances uniformly;
  • Whether it has been confirmed that the scaling mechanism under different instance forms can meet the current service form.

Load Balancing Linkage Check

  • After new instances are added, whether they can be correctly connected to the load balancing entry point;
  • When instances are reduced, whether the entry point can be prevented from continuing to distribute requests to unavailable instances;
  • Whether load balancing serves as the unified entry point for client access;
  • Whether instance changes between the scaling group and load balancing can be completed automatically or in a standardized manner.

High Availability and Fault Recovery Check

  • Whether the load balancing component has redundancy and high availability;
  • When a load balancing component fails, whether it has automatic restart or replacement capabilities;
  • Whether cross-availability-zone deployment is needed to reduce the impact of an exception in a single availability zone;
  • Whether the recovery capabilities of the entry layer, instance layer, and availability zone layer have been evaluated separately.

Follow-up Application-Layer Design Check

  • Whether a request retry policy is needed;
  • Whether a circuit breaking and degradation mechanism is needed;
  • Whether there are model instance warm-up or cold start issues;
  • Whether session persistence or state management is needed;
  • Whether these application-layer issues are evaluated separately from autoscaling and load balancing capabilities.

The key to large model service architecture design is not to put all capabilities into one component, but to let scaling groups, load balancing, inference instances, and availability zone deployment each assume clear responsibilities, and to form a verifiable collaboration relationship during instance changes and fault recovery.