Large Model Service Architecture Design: Autoscaling, Load Balancing, and Failover
Large model service architecture design builds stable services around inference instances, scaling policies, and traffic entry points. It explains the collaborative mechanism between autoscaling and load balancing, helping assess cross-availability-zone highly available inference solutions and failover boundaries.
What Problems Does Large Model Service Architecture Design Need to Solve
Large model service architecture design means planning a service system that can stably provide inference capabilities externally, centered on inference instances, resource scaling, traffic entry points, and exception recovery capabilities. For cloud-based large model inference services, architecture design usually needs to answer three questions at the same time: how access traffic enters, how backend instances adjust as the business changes, and how to reduce the impact when local failures occur.
In this system, autoscaling is used to automatically adjust the amount of computing resources according to policies or business requirements, which means adjusting the number of instances required by the service; load balancing sits between client requests and backend inference instances, and is responsible for providing a unified entry point and distributing traffic; failover focuses on the recovery capabilities of the entry layer, instance layer, and availability zone layer under abnormal conditions.
Scope note: This article focuses on general architectural ideas for cloud-based inference services, including autoscaling, load balancing, cross-availability-zone high availability, and basic fault recovery. It does not cover specific performance metrics, cost formulas, GPU scheduling details, request retry policies, or service mesh implementation.
How Autoscaling Adjusts Inference Resources Based on Business Requirements
Autoscaling is also commonly called Auto Scaling. Its core role is to automatically adjust the amount of computing resources according to configured policies. In the context of large model inference services, it can be understood as follows: when the business requires more or fewer inference instances, the scaling mechanism manages changes in the size of the instance pool.
The Role of Autoscaling in Inference Services
The backend of an inference service usually consists of multiple instances that jointly handle requests. Autoscaling does not directly process individual requests; instead, it is responsible for maintaining the available instance pool.
| Focus | Role of autoscaling |
|---|---|
| Number of instances | Adjusts the number of instances required by the business according to policies |
| Service cluster | Manages a group of inference service instances through a scaling group |
| Resource form | Can be used for scaling management under different instance forms, such as ECI or ECS type scaling groups |
| O&M complexity | Reduces the operational cost of manually adding or removing instances one by one |
In existing cloud-based solutions, scaling groups can be used to manage inference service clusters and support adjusting the number of instances such as ECIs. For different instance forms, ECI type scaling groups and ECS type scaling groups can remain consistent in working principle: both are managed around scaling groups, policies, and changes in instance count.
The Relationship Between Scaling Policies and the Instance Pool
The key to autoscaling is not merely “adding machines,” but ensuring that changes in instance count have a clear basis. When designing an inference service, at least the following questions need to be clarified:
- Which conditions will trigger scale-out or scale-in;
- How newly added instances enter the service cluster after scale-out;
- How to avoid affecting instances that are currently handling traffic during scale-in;
- Whether changes in instance count can be detected by the entry layer in a timely manner.
The last point usually needs to be linked with load balancing. The scaling group is responsible for “how many instances there are,” while load balancing is responsible for “which instances requests are distributed to.”
How Load Balancing Becomes the Unified Traffic Entry Point for Inference Services
Load balancing sits between client requests and backend inference instances, and is the unified entry point through which the inference service provides access externally. Its value lies in the fact that clients do not need to directly perceive changes in each backend instance; instead, they access the service through a unified entry point.
The Position of Load Balancing in the Architecture
A simplified large model inference service can be divided into three layers:
| Layer | Main responsibilities | Relationship with other layers |
|---|---|---|
| Traffic entry layer | Receives external requests and distributes them to backend instances | Usually handled by load balancing |
| Scaling management layer | Maintains the number of instances according to policies | Usually handled by a scaling group |
| Inference instance layer | Actually runs the inference service and processes requests | Managed by the scaling group and connected to load balancing |
Cloud-based solutions often use scaling groups together with load balancing. When a scaling group adds new instances, the new instances can be included in the unified entry point to handle traffic; when instances are reduced, the entry layer also needs to update the backend instance set accordingly.
The High Availability Value of Managed Load Balancing
Managed load balancing services usually have redundancy and high availability, making them suitable as a stable entry layer. In some cloud-based implementations, if a load balancing component fails, related components can be automatically restarted or replaced, thereby reducing the risk that the entry component itself becomes a single point of failure.
In addition, elastic load balancing can be combined with scaling groups to automatically bind a load balancer to instances in a scaling group, reducing manual operations when newly added instances are connected to the entry layer.
How Autoscaling and Load Balancing Work Together
Autoscaling and load balancing have different responsibilities, but they need to work together in inference services. A scaling group manages the number of instances, while load balancing provides a unified entry point and distributes traffic. Only when the two cooperate can newly added instances truly have the opportunity to handle requests.
Typical Collaboration Process
- Set scaling policies: Determine when to adjust the amount of computing resources according to business requirements or preset rules.
- Maintain scaling group instances: The scaling group adds or removes inference service instances according to the policies.
- Connect to load balancing: Newly added instances enter the backend service set; when instances are reduced, they are removed from the backend set or stop handling traffic.
- Traffic enters the inference service: Client requests enter through the unified entry point and are distributed by load balancing to available inference instances.
- Continuously adjust the instance pool: As business requirements change, the scaling group continues to maintain the number of instances, while the entry layer continues to distribute requests to available backend instances.
Example: Combining an ECI Scaling Group with ALB
Existing cloud-based solutions provide a way to combine ECI scaling groups with Application Load Balancer ALB to build cross-availability-zone, elastic, highly available large model inference services. In this model:
- The ECI scaling group is used to manage the inference service cluster and adjust the number of ECIs;
- ALB serves as the application traffic entry point and directs requests to backend inference instances;
- Cross-availability-zone deployment helps reduce the impact of an exception in a single availability zone on the overall service;
- The linkage between the scaling group and load balancing enables changes in instance count to work together with traffic handling capacity.
Viewing the Collaboration Relationship from Architectural Layers
| Architecture layer | Typical components | Design focus |
|---|---|---|
| Traffic entry layer | Application Load Balancer, managed load balancing components | Whether redundancy, high availability, and backend instance access capabilities are available |
| Scaling management layer | Scaling groups, scaling policies | Whether the number of instances can be adjusted according to business requirements |
| Inference instance layer | Inference service instances, container instances, or cloud server instances | Whether they can be managed uniformly and handle entry traffic |
| Availability zone deployment layer | Multi-availability-zone instance distribution | Whether the impact of an exception in a single availability zone is reduced |
Implementation Boundaries of Failover in This Architecture
Failover first depends on designs in the architecture that avoid single points of failure, such as cross-availability-zone deployment, redundant managed load balancing components, and backend instance pools that can be managed by scaling groups. For cloud-based inference services, failover should not be understood as a single function; instead, it should be broken down and evaluated separately at the entry layer, instance layer, and availability zone layer.
Automatic Recovery of Load Balancing Components
Managed load balancing components usually have redundancy and high availability. If a load balancing component fails, related components can be automatically restarted or replaced. Such capabilities mainly address the availability of the entry-layer components themselves.
Application-Layer Fault Governance Needs Separate Design
Load balancing and autoscaling are not equivalent to complete application-layer fault tolerance. The following capabilities usually belong to more detailed service design and need to be evaluated separately in specific systems:
| Capability | Is it within the confirmed scope of this article | Description |
|---|---|---|
| Load balancing component redundancy | Yes | Managed load balancing can have redundancy and high availability |
| Automatic recovery from load balancing component failure | Yes | Components can be automatically restarted or replaced after a failure |
| Cross-availability-zone deployment | Yes | Can reduce the impact of an exception in a single availability zone |
| Request retries | No | Belongs to application-layer or client-side policies and needs separate design |
| Circuit breaking and degradation | No | Belongs to service governance policies and needs to be determined based on the business |
| Model instance warm-up | No | Related to model loading and inference cold start; this article does not provide a definitive discussion |
| Session persistence | No | Related to specific request patterns and business state, and needs separate evaluation |
Key point: Entry-layer high availability, instance pool elasticity, and cross-availability-zone deployment can improve overall resilience, but they cannot replace application-layer design for retries, degradation, warm-up, and state management.
Typical Components of a Cross-Availability-Zone Highly Available Inference Service
A cross-availability-zone highly available inference service usually consists of a load balancer, a scaling group, inference service instances, and multi-availability-zone deployment. Its core goal is: when traffic changes or local infrastructure exceptions occur, the service can still remain available through a unified entry point and a manageable instance pool.
Key Components
| Component | Main role |
|---|---|
| Application Load Balancer | Provides a unified access entry point and directs requests to backend instances |
| Scaling group | Manages the number of inference service instances and supports scale-out and scale-in |
| Inference service instances | Actually run the model inference service and process requests |
| Cross-availability-zone deployment | Reduces the impact of an exception in a single availability zone on the overall service |
| Managed load balancing components | Improve entry-layer availability through redundancy and automatic recovery capabilities |
Common Requirements and Solutions
| User requirement | Solution | Expected effect |
|---|---|---|
| Significant access traffic fluctuations | Use a scaling group to adjust the number of instances according to policies | The instance pool can expand or contract as business requirements change |
| Newly added instances need to handle traffic | Combine the scaling group with load balancing | New instances can be included in the backend set of the unified entry point |
| The entry layer needs to be stable and available | Use managed load balancing with redundancy | Reduces the impact caused by entry component failures |
| Concern about an exception in a single availability zone | Adopt cross-availability-zone deployment | Reduces the impact of an exception in a single availability zone on the overall service |
| Want to reduce manual O&M | Link elastic load balancing with the scaling group | Reduces the operational complexity of connecting instances to and removing them from the entry layer |
Checklist for Designing Large Model Inference Services
Before launching or evaluating a large model inference service architecture, you can use the following checklist to review elasticity, entry, and fault recovery capabilities.
Scaling Capability Check
- Whether scaling policies and the conditions that trigger instance count adjustment have been clearly defined;
- Whether the correspondence between the amount of computing resources and the number of instances required by the business is clear;
- Whether a scaling group is used to manage inference service instances uniformly;
- Whether it has been confirmed that the scaling mechanism under different instance forms can meet the current service form.
Load Balancing Linkage Check
- After new instances are added, whether they can be correctly connected to the load balancing entry point;
- When instances are reduced, whether the entry point can be prevented from continuing to distribute requests to unavailable instances;
- Whether load balancing serves as the unified entry point for client access;
- Whether instance changes between the scaling group and load balancing can be completed automatically or in a standardized manner.
High Availability and Fault Recovery Check
- Whether the load balancing component has redundancy and high availability;
- When a load balancing component fails, whether it has automatic restart or replacement capabilities;
- Whether cross-availability-zone deployment is needed to reduce the impact of an exception in a single availability zone;
- Whether the recovery capabilities of the entry layer, instance layer, and availability zone layer have been evaluated separately.
Follow-up Application-Layer Design Check
- Whether a request retry policy is needed;
- Whether a circuit breaking and degradation mechanism is needed;
- Whether there are model instance warm-up or cold start issues;
- Whether session persistence or state management is needed;
- Whether these application-layer issues are evaluated separately from autoscaling and load balancing capabilities.
The key to large model service architecture design is not to put all capabilities into one component, but to let scaling groups, load balancing, inference instances, and availability zone deployment each assume clear responsibilities, and to form a verifiable collaboration relationship during instance changes and fault recovery.