Alauda AI

Install Alauda Build of KServe

Alauda Build of KServe is a cloud-native component built on KServe for serving generative AI models. As an extension of the Alauda AI ecosystem, it specifically optimizes for Large Language Models (LLMs), offering essential features such as inference orchestration, streaming responses, and resource-based auto-scaling for generative workloads.

TOC

PrerequisitesRequired DependenciesOptional DependenciesInstallation NotesDownloading Cluster PluginUploading the Cluster PluginInstalling Alauda Build of KServeEnvoy Gateway ConfigurationEnvoy AI Gateway ConfigurationKServe Gateway ConfigurationGIE(gateway-api-inference-extension) ConfigurationAlauda AI IntegrationUpgrading Alauda Build of KServe

Prerequisites

Before installing Alauda Build of KServe, you need to ensure the following dependencies are installed:

Required Dependencies

DependencyTypeDescription
Alauda build of Envoy Gateway install_envoy_gateway_operatorOperatorProvides the underlying gateway functionality for AI services
Envoy AI GatewayCluster PluginProvides AI-specific gateway capabilities
Alauda Build of LeaderWorkerSetCluster PluginProvides leader-worker set functionality for AI workloads
INFO

Alauda build of Envoy Gateway is natively integrated into ACP 4.2. For environments running earlier versions (including ACP 4.0 and 4.1), please contact Customer Support for compatibility and installation guidance.

Optional Dependencies

DependencyTypeDescription
GIEBuilt-inIntegrated GIE (gateway-api-inference-extension) for enhanced AI capabilities. Can be enabled through the Alauda Build of KServe UI.
Alauda AIOperatorRequired only if you need to use KServe Predictive AI functionality. Can be disabled if you only need LLM Generative AI functionality.

Installation Notes

  1. Required Dependencies: All three required dependencies must be installed before installing Alauda Build of KServe.
  2. GIE Integration: If you want to use GIE, you can enable it during the installation process by selecting the "Integrated GIE" option in the Alauda Build of KServe UI.
  3. Alauda AI Integration: If you don't need KServe Predictive AI functionality and only want to use LLM Generative AI, you can disable the "Integrated With Alauda AI" option during installation.

Downloading Cluster Plugin

INFO

Alauda Build of KServe cluster plugin can be retrieved from Customer Portal.

Please contact Consumer Support for more information.

Uploading the Cluster Plugin

For more information on uploading the cluster plugin, please refer to Uploading Cluster Plugins

Installing Alauda Build of KServe

  1. Go to the Administrator -> Marketplace -> Cluster Plugin page, switch to the target cluster, and then deploy the Alauda Build of KServe Cluster plugin.

  2. In the deployment form, configure the following parameters as needed:

Envoy Gateway Configuration

ParameterDescriptionDefault Value
ServiceAccount NameThe name of the service account used by Envoy Gateway.envoy-gateway
ServiceAccount NamespaceThe namespace where the service account is located.envoy-gateway-system
Create InstanceCreate an Envoy Gateway instance to manage inference traffic with bundled extensions.Enabled
Instance NameThe name of the Envoy Gateway instance to be created.aieg

Envoy AI Gateway Configuration

ParameterDescriptionDefault Value
Service NameThe Kubernetes service name for Envoy AI Gateway.ai-gateway-controller
Port NumberThe port number used by Envoy AI Gateway.1063

KServe Gateway Configuration

ParameterDescriptionDefault Value
EnabledInstall a KServe Gateway Instance for inferenceservices functionality.Enabled
Gateway NameThe name of the KServe Gateway.kserve-ingress-gateway
Gateway NamespaceThe namespace where the KServe Gateway is deployed.kserve
GatewayClassOptional. The custom name for the GatewayClass. If left empty, the system will automatically derive it following the "{Namespace}-{Name}" pattern.(Empty)
Port NumberThe port number used by KServe Gateway.80

GIE(gateway-api-inference-extension) Configuration

ParameterDescriptionDefault Value
BuiltInInstall with the bundled gateway-api-inference-extension v0.5.1 dependencies for enhanced AI capabilities.Enabled

Alauda AI Integration

ParameterDescriptionDefault Value
IntegratedEnable integration with Alauda AI core plugin to reuse existing configurations.Disabled

  1. Click Install to begin the installation process.

  2. Verify result. You can see the status of "Installed" in the UI.

Upgrading Alauda Build of KServe

  1. Upload the new version for package of Alauda Build of KServe plugin to ACP.
  2. Go to the Administrator -> Clusters -> Target Cluster -> Functional Components page, then click the Upgrade button, and you will see the Alauda Build of KServe can be upgraded.