From d45071dcb221c31a3db1a9fa98262026cabf6f87 Mon Sep 17 00:00:00 2001 From: Hadar Cohen Date: Tue, 28 Jul 2026 10:41:28 +0300 Subject: [PATCH 1/2] docs: restructure README into quickstart-style docs layout Replace the mega-README with a short landing page and move lasting content into docs/, so install paths stay in the deploy READMEs. --- AGENTS.md | 3 + PCA_ARO.md | 250 - PCA_Deployment_ARO/README.md | 16 +- .../scripts/post-terraform-fullstack.sh | 2 +- .../testresults_h100.md | 0 README.md | 4146 +---------------- docs/architecture.md | 81 + docs/benchmarks.md | 48 + docs/customization.md | 34 + docs/ide-and-extensions.md | 56 + .../architecture-diagram-landscape.png | Bin .../architecture-diagram-landscape.svg | 0 .../images}/architecture-traffic-flow.svg | 0 .../images}/aws-architectural-view.png | Bin .../images}/aws-architectural-view.svg | 0 .../images}/deployment-view-layered.png | Bin .../images}/deployment-view-layered.svg | 0 ...spaces-ai-code-assistants-architecture.png | Bin 0 -> 91024 bytes .../devspaces-remote-ssh-security-flow.png | Bin 0 -> 85696 bytes .../images}/redhat-components-view.png | Bin .../images}/redhat-components-view.svg | 0 docs/models-and-routing.md | 46 + docs/requirements.md | 46 + 23 files changed, 424 insertions(+), 4304 deletions(-) delete mode 100644 PCA_ARO.md rename testresults_h100.md => PCA_Deployment_ARO/testresults_h100.md (100%) create mode 100644 docs/architecture.md create mode 100644 docs/benchmarks.md create mode 100644 docs/customization.md create mode 100644 docs/ide-and-extensions.md rename {images => docs/images}/architecture-diagram-landscape.png (100%) rename {images => docs/images}/architecture-diagram-landscape.svg (100%) rename {images => docs/images}/architecture-traffic-flow.svg (100%) rename {images => docs/images}/aws-architectural-view.png (100%) rename {images => docs/images}/aws-architectural-view.svg (100%) rename {images => docs/images}/deployment-view-layered.png (100%) rename {images => docs/images}/deployment-view-layered.svg (100%) create mode 100644 docs/images/devspaces-ai-code-assistants-architecture.png create mode 100644 docs/images/devspaces-remote-ssh-security-flow.png rename {images => docs/images}/redhat-components-view.png (100%) rename {images => docs/images}/redhat-components-view.svg (100%) create mode 100644 docs/models-and-routing.md create mode 100644 docs/requirements.md diff --git a/AGENTS.md b/AGENTS.md index 02b9560..4abb6cb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,6 +2,8 @@ Deploy a private, self-hosted AI coding assistant on OpenShift so developers each get their own namespace with an AI-powered IDE — no code leaves the cluster. +User-facing docs: [README.md](README.md) and [docs/](docs/). + ## Inference request path (RHCL + llm-d) ```mermaid @@ -75,6 +77,7 @@ Package lives in `tests/cluster-smoke/` (see its README). Optional Langfuse / OT ## Directory Structure ``` +docs/ # User-facing guides (architecture, requirements, IDE, benchmarks, …) charts/ # Unified Helm charts — single set for ROSA and ARO ├── pca-app-of-apps/ # Root ArgoCD AppProject + child Applications ├── pca-operators/ # Operator Subscriptions (RHOAI, GPU, DevSpaces, NFD, RHCL, …) diff --git a/PCA_ARO.md b/PCA_ARO.md deleted file mode 100644 index f72bcdc..0000000 --- a/PCA_ARO.md +++ /dev/null @@ -1,250 +0,0 @@ -# Private AI Code Assistant — Azure Red Hat OpenShift (ARO) Deployment - -> **Status:** Deployment validated. Performance benchmarks completed — see [Test Results](PCA_Deployment_ARO/testresults.md). - ---- - -## Overview - -This document describes the architecture and deployment of the **Private AI Code Assistant (PCA)** -on **Azure Red Hat OpenShift (ARO)**. The ARO deployment provides enterprise development teams -with a self-hosted, air-gappable AI coding assistant powered by **Qwen3.6-35B-A3B-FP8** — with -no data leaving the customer's Azure environment. - -The deployment uses **Red Hat OpenShift AI 3.3.2** with the **llm-d AI Gateway (GA)**, serving -the model via **vLLM 0.17.1** on an NVIDIA A100 80 GB GPU. - ---- - -## Architecture - -``` -┌─────────────────────────────────────────────────────────────────────────┐ -│ Azure Red Hat OpenShift (ARO) │ -│ Central US — OCP 4.19.24 │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ Developer Tier │ │ -│ │ OpenShift Dev Spaces ── VS Code in Browser │ │ -│ │ Roo Code · Continue · Cline (AI coding extensions) │ │ -│ └────────────────────────┬────────────────────────────────────────┘ │ -│ │ HTTPS (cluster-internal) │ -│ ┌────────────────────────▼────────────────────────────────────────┐ │ -│ │ AI Gateway Tier │ │ -│ │ llm-d Gateway (GA) ── Istio Gateway API │ │ -│ │ OpenAI-compatible API · HTTPRoute → direct Service backend │ │ -│ │ DestinationRule: TLS origination to vLLM (KServe certs) │ │ -│ └────────────────────────┬────────────────────────────────────────┘ │ -│ │ │ -│ ┌────────────────────────▼────────────────────────────────────────┐ │ -│ │ AI Inference Tier │ │ -│ │ vLLM 0.17.1 + KServe (RHOAI 3.3.2) │ │ -│ │ Qwen3.6-35B-A3B-FP8 (MoE — ~3B active params) │ │ -│ │ NVIDIA A100 80 GB · 64K context window · FP8 native │ │ -│ │ Standard_NC24ads_A100_v4 ($3.67/hr) │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌────────────────────────────┐ ┌──────────────────────────────────┐ │ -│ │ Platform Operators │ │ Azure Infrastructure │ │ -│ │ RHOAI 3.3.2 (stable-3.x) │ │ Resource Group · VNet │ │ -│ │ OpenShift GitOps (ArgoCD) │ │ Subnets (no NSG — ARO-managed) │ │ -│ │ NVIDIA GPU Operator v24.9 │ │ managed-csi storage │ │ -│ │ NFD · Service Mesh 3.x │ │ Central US region │ │ -│ │ Serverless · DevSpaces │ └──────────────────────────────────┘ │ -│ │ LeaderWorkerSet · cert-mgr│ │ -│ └────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## Why NVIDIA A100 on Azure? - -Azure does not offer NVIDIA L40S virtual machines (the GPU used on AWS `g6e.2xlarge`). -The closest available GPU for single-node LLM inference on Azure is the A100: - -| GPU | VRAM | FP8 | Azure VM | $/hr (Central US) | -|-----|------|-----|----------|---------------| -| NVIDIA L40S (AWS) | 48 GB GDDR6 | Yes | `g6e.2xlarge` | $2.07 | -| **NVIDIA A100 (Azure)** | **80 GB HBM2** | **Yes** | `Standard_NC24ads_A100_v4` | **$3.67** | -| NVIDIA A10 (Azure) | 24 GB max | No | `NV36ads_A10_v5` | $3.20 | -| NVIDIA H100 NVL (Azure) | 94 GB | Yes | `NC40ads_H100_v5` | $6.98 | - -The A100 is the right choice because: -- **Single-GPU fit:** 80 GB fits Qwen3.6-35B-A3B at FP8 (~30 GB weights) with a 65,536-token - context window and KV cache — comfortably larger than the 32,768-token window on L40S -- **FP8 support:** Native FP8 on Ampere for inference acceleration -- **No tensor parallelism needed:** Single-GPU inference avoids multi-GPU coordination overhead -- **A10 ruled out:** Maximum 24 GB per Azure A10 VM — too small for Qwen3.6-35B-A3B -- **H100 ruled out:** ~2× the cost for incremental throughput gains on a 30B model - ---- - -## Infrastructure - -| Resource | Type | Count | -|----------|------|-------| -| ARO Cluster | OCP 4.19.24 | 1 | -| Master nodes | `Standard_D8s_v5` | 3 | -| Worker nodes | `Standard_D8s_v5` | 3 | -| GPU nodes | `Standard_NC24ads_A100_v4` | 1 | -| Virtual Network | Azure VNet | 1 | -| Subnets | Master + Worker (no NSG) | 2 | -| Storage | managed-csi PVC | 100 Gi | - -**Estimated monthly cost (Central US, pay-as-you-go):** -- ARO cluster fee: ~$0.18/hr (Microsoft managed control plane) -- 3× `Standard_D8s_v5` workers: ~$1.15/hr -- 1× `Standard_NC24ads_A100_v4` GPU node: $3.67/hr -- Storage, networking: ~$50–100/month -- **Total (approximate): ~$4,000–4,500/month** for a 24×7 deployment - -Cost optimisation options: Azure Reserved Instances (1-year) provide ~40% savings on compute. - ---- - -## Deployment - -### Quick Start - -```bash -# 1. Clone the repository -git clone https://github.com/manu-joy/Private_Code_Assistant.git -cd Private_Code_Assistant - -# 2. Install prerequisites: terraform >= 1.4.6, azure-cli >= 2.50, oc >= 4.19, jq - -# 3. Authenticate -az login -az account set --subscription "" - -# 4. Register ARO providers (first time only) -az provider register --namespace Microsoft.RedHatOpenShift --wait -az provider register --namespace Microsoft.Compute --wait - -# 5. Ensure GPU quota: Standard NCADSv4Family (24 vCPU) in your region - -# 6. Configure variables -cd PCA_Deployment_ARO/terraform -cp terraform.tfvars.example terraform.tfvars -# Edit terraform.tfvars — fill in subscription_id, pull_secret, gitops_repo_url - -# 7. Deploy (~45 minutes) -terraform init -terraform apply - -# 8. Install NFD operator (required for GPU detection — see README.md Step 5) - -# 9. Set HuggingFace token -HF_TOKEN_B64=$(echo -n "hf_your_token" | base64) -oc patch secret hf-token -n ai-serving --type='json' \ - -p='[{"op":"replace","path":"/data/token","value":"'"${HF_TOKEN_B64}"'"}]' - -# 10. Apply vLLM 0.17.1 upgrade (required — see README.md Post-Deploy section) -``` - -For complete step-by-step instructions including the **vLLM 0.17.1 upgrade workaround**, -NFD installation, and troubleshooting, see [PCA_Deployment_ARO/README.md](PCA_Deployment_ARO/README.md). - ---- - -## Model Configuration - -| Parameter | Value | Rationale | -|-----------|-------|-----------| -| Model | `Qwen/Qwen3.6-35B-A3B-FP8` | State-of-the-art code reasoning, FP8 quantized (~37 GB) | -| Architecture | `qwen3_5_moe` | MoE with ~3B active params per token | -| GPU | NVIDIA A100 80 GB | Single-GPU fit with headroom | -| max-model-len | 65,536 | Full 64K context window — fits on A100 80 GB at 90% utilization | -| vLLM version | 0.17.1 | Upstream image (RHOAI 3.3.2 ships 0.13.0; qwen3_5_moe requires >= 0.17) | -| GPU memory utilization | 90% | Leaves ~8 GB headroom for activation memory | -| Quantization | FP8 | Native A100 FP8 support | - ---- - -## GitOps Structure - -``` -PCA_Deployment_ARO/ -├── terraform/ # Azure infrastructure (RG, VNet, ARO via az aro create, GPU MachineSet) -├── argocd/ -│ ├── 00-app-of-apps.yaml # Root ArgoCD application + AppProject -│ ├── 01-operators/ # RHOAI 3.3.2, GPU Operator v24.9, Service Mesh, Serverless, -│ │ # DevSpaces, LWS, cert-manager -│ ├── 02-platform-config/ # DataScienceCluster, CheCluster, namespaces, RBAC, HF token -│ ├── 03-ai-serving/ # LLMInferenceService, llm-d Gateway, model cache PVC, TLS cert -│ └── 04-devspaces/ # DevWorkspaces, AI extension configs (Roo Code, Continue, Cline) -└── scripts/ - ├── create-gpu-machineset.sh # Post-cluster A100 node provisioning (Gen2 image) - └── validate.sh # Post-deployment validation -``` - ---- - -## Performance Benchmarks - -Benchmarks were run using [GuideLLM v0.6.0](https://github.com/vllm-project/guidellm) with -a sweep profile (synchronous → throughput → constant rate escalation). - -Full results: [PCA_Deployment_ARO/testresults.md](PCA_Deployment_ARO/testresults.md) - -### Single-User Latency - -| Metric | Short (128/128) | Medium (512/256) | Long (2048/512) | -|--------|-----------------|-------------------|-----------------| -| TTFT | 57 ms | 117 ms | 1,020 ms | -| ITL | 6.8 ms | 6.8 ms | 6.9 ms | -| E2E latency | 0.93 s | 1.86 s | 4.52 s | -| Output tokens/s | 138 | 138 | 139 | - -Inter-token latency is remarkably consistent at ~6.8 ms across all prompt sizes, -translating to ~138 output tokens/second for a single user. - -### Peak Throughput - -| Metric | Short (128/128) | Medium (512/256) | Long (2048/512) | -|--------|-----------------|-------------------|-----------------| -| Output tokens/s | 2,781 | 2,008 | 1,263 | -| Requests/s | 20.7 | 7.6 | 2.4 | - -### Capacity Planning Summary - -| Total Developers | Concurrent (30%) | Output tok/s | Tok/s Per User | ITL (ms) | Experience | -|:----------------:|:-----------------:|:------------:|:--------------:|:--------:|:----------:| -| 10 | 3 | ~270 | ~90 | ~10 | Excellent | -| 20 | 6 | ~420 | ~70 | ~13 | Excellent | -| 50 | 15 | ~700 | ~47 | ~21 | Good | -| 100 | 30 | ~1,000 | ~33 | ~30 | Acceptable | - -For multi-GPU scaling recommendations and detailed capacity planning tables, see -[PCA_Deployment_ARO/testresults.md](PCA_Deployment_ARO/testresults.md). - ---- - -## Key Differences from ROSA Deployment - -| Aspect | ROSA (AWS) | ARO (Azure) | -|--------|-----------|-------------| -| GPU VM | `g6e.2xlarge` (L40S 48 GB) | `Standard_NC24ads_A100_v4` (A100 80 GB) | -| GPU provisioning | RHCS machine pool via Terraform | MachineSet script post-cluster (Gen2 image required) | -| Storage class | `gp3-csi` | `managed-csi` | -| IDP | HTPasswd via RHCS Terraform resource | kubeadmin + manual HTPasswd | -| Cluster creation | Terraform RHCS provider | `az aro create` via `null_resource` (handles SP internally) | -| NSG | AWS Security Groups (managed) | None on subnets (ARO-managed) | -| RHOAI channel | `stable-2.19` | `stable-3.x` (RHOAI 3.3.2) | -| AI Gateway | Technology Preview | **GA** | -| vLLM max-model-len | 32,768 (L40S 48 GB) | **65,536** (A100 80 GB) | -| vLLM version | RHOAI default | **0.17.1** (upstream, qwen3_5_moe support) | -| NFD | Pre-installed | **Manual install required** | - ---- - -## Roadmap - -- [ ] Upgrade to RHOAI 3.4+ when GA — eliminates the vLLM 0.17.1 manual workaround -- [ ] Add architecture diagrams: Azure infrastructure view, AI serving traffic flow -- [ ] Evaluate Azure Reserved Instance pricing for 1-year commitment -- [ ] Test with Gemma 4 Coder when vLLM support is available in RHOAI GA -- [x] Run `guidellm` sweep and populate benchmark table -- [x] Test Roo Code, Continue, and Cline end-to-end within ARO Dev Spaces -- [x] Validate HTPasswd IDP configuration for DevSpaces user onboarding diff --git a/PCA_Deployment_ARO/README.md b/PCA_Deployment_ARO/README.md index 5c48549..8d87dcd 100644 --- a/PCA_Deployment_ARO/README.md +++ b/PCA_Deployment_ARO/README.md @@ -653,7 +653,21 @@ GuideLLM sweep results for Qwen3.6-35B-A3B-FP8 on H100 NVL: | Code Review | 4,096 | 1,024 | 16,133 | 5.59 | 157 | | File Generation | 8,192 | 2,048 | 13,976 | 11.15 | 208 | -Full results: [`testresults_h100.md`](../testresults_h100.md) +Full results: [`testresults_h100.md`](testresults_h100.md) · A100 sweep: [`testresults.md`](testresults.md) · Summary: [`docs/benchmarks.md`](../docs/benchmarks.md) + +--- + +## Key differences from ROSA + +| Aspect | ROSA (AWS) | ARO (Azure) | +|--------|------------|-------------| +| GPU VM | e.g. `g6e` (L40S) via RHCS machine pool | GPU MachineSet (e.g. H100 NVL / A100 family) after cluster create | +| Storage class | `gp3-csi` | `managed-csi` | +| Charts | `charts/` + `values-rosa.yaml` | Same `charts/` + `values-aro.yaml` | +| Cloud values | Terraform sets `gitops.cloud=rosa` | Terraform sets `gitops.cloud=aro` | +| NSG / network | AWS security groups | ARO-managed subnets (no customer NSG on master/worker) | + +Azure GPU choice depends on quota and model size (A100 80 GB vs H100 NVL 94 GB). Prefer a SKU with native FP8 and enough VRAM for your context window; see [docs/benchmarks.md](../docs/benchmarks.md). --- diff --git a/PCA_Deployment_ARO/scripts/post-terraform-fullstack.sh b/PCA_Deployment_ARO/scripts/post-terraform-fullstack.sh index 94097cc..9f517eb 100755 --- a/PCA_Deployment_ARO/scripts/post-terraform-fullstack.sh +++ b/PCA_Deployment_ARO/scripts/post-terraform-fullstack.sh @@ -9,7 +9,7 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" BASE_PATH="${SCRIPT_DIR}/../argocd" -RESULTS_FILE="${SCRIPT_DIR}/../../testresults_h100.md" +RESULTS_FILE="${SCRIPT_DIR}/../testresults_h100.md" RED='\033[0;31m' GREEN='\033[0;32m' diff --git a/testresults_h100.md b/PCA_Deployment_ARO/testresults_h100.md similarity index 100% rename from testresults_h100.md rename to PCA_Deployment_ARO/testresults_h100.md diff --git a/README.md b/README.md index 1058469..d4b42a6 100644 --- a/README.md +++ b/README.md @@ -1,4100 +1,142 @@ # Private AI Code Assistant on Red Hat OpenShift -## Table of Contents - -- [Executive Summary](#executive-summary) -- [Why Private Code Assistants](#why-private-code-assistants) - - [Security Advantages](#security-advantages-over-public-ai-code-assistants) - - [Additional Enterprise Benefits](#additional-enterprise-benefits) -- [Architecture Overview](#architecture-overview) - - [Platform Stack](#platform-stack) - - [High-Level Architecture](#high-level-architecture) - - [Full-Stack Architecture Diagram](#full-stack-architecture-diagram) - - [Inference Traffic Flow](#inference-traffic-flow) - - [Architecture Diagram (SVG — Landscape)](#architecture-diagram-svg--landscape) - - [Deployment View (Layered Architecture)](#deployment-view-layered-architecture) - - [AWS Architectural View](#aws-architectural-view) - - [Red Hat Components View](#red-hat-components-view) -- [Frontend: Developer IDE Environment](#frontend-developer-ide-environment) - - [Web IDE (Browser-Based)](#deployment-option-1-web-ide-browser-based) - - [Remote SSH (Thin Client)](#deployment-option-2-remote-ssh-thin-client-desktop-ide) - - [Current Implementation](#current-implementation) - - [Extension Comparison: Continue vs Cline vs Roo Code](#ai-code-assistant-extension-comparison-continue-vs-cline-vs-roo-code) - - [Configuring vLLM for Roo Code Tool Calling](#configuring-vllm-for-roo-code-tool-calling-qwen3-coder) - - [Production Scaling](#production-scaling-considerations) -- [Backend: Model Hosting and Intelligent Routing](#backend-model-hosting-and-intelligent-routing) - - [Model: Qwen3-Coder-30B-A3B-Instruct](#model-qwen3-coder-30b-a3b-instruct) - - [NVIDIA Deployment (LLMInferenceService)](#deployment-via-llminferenceservice-nvidia) - - [Inferentia Deployment (InferenceService)](#deployment-via-inferenceservice-inferentia) - - [llm-d Intelligent Routing](#llm-d-intelligent-routing) - - [Multi-Accelerator Scaling Phases](#multi-accelerator-scaling-phases) - - [Prefix Caching Compatibility](#prefix-caching-model-compatibility) - - [Accelerator Selection Guide: NVIDIA vs Inferentia](#accelerator-selection-guide-nvidia-gpu-vs-aws-inferentia2) - - [Gateway URL](#gateway-url) -- [Test Results](#test-results) - - [Phase 1: Single NVIDIA GPU Node](#phase-1-single-nvidia-gpu-node) - - [Phase 2: Two NVIDIA Nodes with llm-d](#phase-2-two-nvidia-gpu-nodes-with-llm-d-load-balancing) - - [Phase 3: Heterogeneous Routing](#phase-3-heterogeneous-routing-cuda--inferentia) - - [Inferentia Standalone Performance](#inferentia-standalone-performance-pre-llm-d) - - [Performance Benchmark: NVIDIA L40S](#performance-benchmark-llm-d-on-nvidia-l40s-fp8) - - [GuideLLM Sweep Results](#guidellm-sweep-benchmark-results-april-7-2026) - - [Inferentia Context Window Limits](#inferentia-context-window--maximum-supported-length) - - [Phase 4: Trainium trn1.32xlarge with DRA](#phase-4-trainium-trn132xlarge-with-neuron-dra-driver) - - [Phase 4b: 2-Replica Scaling Investigation](#phase-4b-2-replica-scaling-investigation-april-9-2026) -- [Planned Test Phases](#planned-test-phases) - - [Phase 5: Dual Qwen on inf2.48xlarge](#phase-5-dual-qwen3-coder-30b-on-inf248xlarge) - - [Phase 6: NVIDIA L40S Full Benchmark](#phase-6-nvidia-g6e2xlarge-l40s-full-guidellm-benchmark) - - [Cross-Phase Cost Comparison](#cross-phase-cost-comparison-qwen3-coder-30b-2-instances) - - [Why BF16 Only on Inferentia2](#why-bf16-only-on-inferentia2) -- [ARO Deployment: Qwen3.6-35B-A3B-FP8 on NVIDIA A100](#aro-deployment-qwen36-35b-a3b-fp8-on-nvidia-a100) -- [Step-by-Step Deployment Guide](#step-by-step-deployment-guide) - - [Prerequisites](#prerequisites) - - [Steps 1-15: Full Deployment](#step-1-install-prerequisite-operators) - - [Step 9b: Alternative Model Deployments (Gemma 4, Nemotron 3 Nano)](#step-9b-alternative-model-deployments-gemma-4-nemotron-3-nano) -- [Component Versions](#component-versions) - - [Upgrade Notes (April 21, 2026)](#upgrade-notes-april-21-2026) -- [Model Caching and Startup Optimization](#model-caching-and-startup-optimization-analysis) -- [KV Cache Optimization: FP8 Quantization](#kv-cache-optimization-fp8-quantization) -- [Project Considerations](#project-considerations) -- [Enterprise Customization for Implementation Services](#enterprise-customization-for-implementation-services) - - [Track 1: Customizing for Customer Development Practices](#track-1-customizing-for-customer-development-practices) - - [System Prompt and Context Engineering](#11-system-prompt-and-context-engineering) - - [Per-Project Context via Rules Files](#12-per-project-context-via-rules-files) - - [DevWorkspace Templates per Technology Stack](#13-devworkspace-templates-per-technology-stack) - - [MCP Server Integration with Internal Systems](#14-mcp-server-integration-with-internal-systems) - - [Track 2: SonarQube-Aware Code Generation](#track-2-sonarqube-aware-code-generation) - - [Embedding SonarQube Rule Profiles in System Prompts](#21-embedding-sonarqube-rule-profiles-in-system-prompts) - - [SonarQube Findings as Live AI Context](#22-sonarqube-findings-as-live-ai-context) - - [Pre-Commit and CI Integration](#23-pre-commit-and-ci-integration) - - [Language-Specific SonarQube Customization Matrix](#24-language-specific-sonarqube-customization-matrix) - - [Continuous Improvement Loop](#25-continuous-improvement-loop) -- [Appendix A: Neuron-Scheduler / OVN Annotation Conflict](#appendix-a-neuron-scheduler--ovn-kubernetes-annotation-conflict) -- [Appendix B: OVN Annotation Fix -- Live Tests](#appendix-b-ovn-annotation-fix----live-test-results-april-2-2026) - ---- - -## Executive Summary - -**Brief overview:** An enterprise-grade private code assistant platform built on **OpenShift Dev Spaces** and **privately hosted LLMs** on **ROSA HCP**, so proprietary source code does not need to be sent to external AI services. Developers get IDE-integrated assistance while inference stays inside the organization's network and governance model. - -## Why Private Code Assistants - -### Security Advantages Over Public AI Code Assistants - -- **Source code never leaves the organization's network perimeter** -- inference and workspace workloads stay on infrastructure you operate. -- **No risk of proprietary code being used to train third-party models** -- you are not contributing code to external vendors' training pipelines by default. -- **Full audit trail of AI interactions** -- cluster-level logging, mesh, and platform logging can record who called which endpoints and when (subject to your retention and privacy policies). -- **Compliance with data sovereignty regulations** -- data residency and processing boundaries (GDPR, HIPAA, FedRAMP-aligned controls) can be enforced when workloads and logs remain in approved regions and systems. -- **No dependency on external service availability or SLAs** -- model serving is on your cluster; outages are bounded by your own operations, not a public API's incident. -- **Per-user quotas and access controls** -- OpenShift **RBAC**, namespaces, quotas, and network policies can scope who may run workspaces and call inference endpoints. -- **No risk of prompt injection attacks via shared model context** -- **Enterprise IDP integration** -- LDAP, OIDC, SAML, and cluster SSO tie developer identity to workspace and API access. -- **Model selection under organizational control** -- teams can standardize on models vetted for code quality, safety, and license fit. - -### Additional Enterprise Benefits - -- **Predictable costs** -- fixed infrastructure and capacity planning versus volatile per-token public API bills. -- **Customizable system prompts and guardrails** -- per team or project, aligned with internal policies. -- **Ability to fine-tune models** on internal codebases where policy and law allow. -- **Network isolation between development teams** -- namespaces, policies, and mesh can separate tenants. -- **Integration with internal CI/CD and code review** -- same private network as pipelines, SCM, and review tools. - -## Architecture Overview - -### Platform Stack - -- **Cluster**: ROSA HCP 4.21 on AWS (us-east-2) -- **AI Platform**: Red Hat OpenShift AI (RHOAI) 3.3 -- **IDE Platform**: OpenShift Dev Spaces 3.27 -- **Model Serving**: vLLM via LLMInferenceService (KServe) -- **Intelligent Routing**: llm-d (load-aware, KV-cache-aware, prefix-cache-aware) -- **Gateway**: Service Mesh 3 (Istio) with Gateway API -- **Accelerators**: NVIDIA L40S GPUs (g6e.2xlarge) + AWS Inferentia2 (inf2.24xlarge) - -### High-Level Architecture - -Developers connect to **OpenShift Dev Spaces** using either the **web IDE** or **Remote SSH**. Each **Dev Spaces workspace** runs the IDE and AI extensions (e.g., Continue, Cline). Those extensions call an **OpenAI-compatible HTTP API** exposed by the **llm-d gateway**. The **llm-d Endpoint Picker (EPP)** chooses an optimal **vLLM** replica based on queue depth, KV cache headroom, and prefix-cache affinity. **vLLM** serves the model on **NVIDIA GPU** or **AWS Inferentia2**, depending on deployment. Traffic is protected by **Service Mesh**, **Gateway API**, and cluster **TLS** where configured. - -### Full-Stack Architecture Diagram - -``` -┌─────────────────────────────────────────────────────────────────────────────────────────┐ -│ DEVELOPER │ -│ Browser / VS Code Desktop (Remote SSH) / Cursor / Kiro / CLI │ -└────────────────────────────────┬───────────────────────┬────────────────────────────────┘ - │ HTTPS :443 │ SSH :2022 - ▼ ▼ -┌─────────────────────────────────────────────────────────────────────────────────────────┐ -│ AWS Network Load Balancer (NLB) │ -│ ┌──────────────────────────────────────────┐ ┌───────────────────────────────────┐ │ -│ │ HTTPS :443 → Istio Ingress Gateway │ │ HTTPS :6443 → K8s API (PrivLink) │ │ -│ └────────────────────────────┬─────────────┘ └───────┬───────────────────────────┘ │ -└────────────────────────────────┼───────────────────────┼────────────────────────────────┘ - │ │ -┌────────────────────────────────┼───────────────────────┼────────────────────────────────┐ -│ RED HAT OPENSHIFT (ROSA HCP 4.21 ─ us-east-2) │ -│ ▼ ▼ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ -│ │ DEVELOPER EXPERIENCE LAYER (ns: openshift-devspaces) │ │ -│ │ ┌─────────────────────────────────────┐ ┌─────────────────────────────────────┐ │ │ -│ │ │ OpenShift Dev Spaces 3.27 │ │ Dev Spaces Workspace Pods │ │ │ -│ │ │ ┌──────────────────────────────┐ │ │ ┌──────────────────────────────┐ │ │ │ -│ │ │ │Dashboard (Web UI) │ │ │ │VS Code for the Web │ │ │ │ -│ │ │ │DevWorkspace Controller │ │ │ │┌────────┐ ┌────────┐ │ │ │ │ -│ │ │ │DevWorkspace Operator │ │ │ ││Continue│ │Roo Code│ │ │ │ │ -│ │ │ │HTPasswd IDP · RBAC │ │ │ │└───┬────┘ └───┬────┘ │ │ │ │ -│ │ │ └──────────────────────────────┘ │ │ └──────────────┘ │ │ │ -│ │ │ │ │ OpenAI-compat API ↑ │ │ │ -│ │ └─────────────────────────────────────┘ └─────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ -│ │ AI PLATFORM LAYER (Red Hat OpenShift AI 3.3) │ │ -│ │ ┌───────────────────────────────────────────────────────────────────────────────┐ │ │ -│ │ │ Service Mesh 3 (Istio) + Gateway API │ │ │ -│ │ │ HTTPRoute → llm-d Gateway (Envoy ext_proc filter) · TLS + mTLS │ │ │ -│ │ └──────────────────────────────────────┬────────────────────────────────────────┘ │ │ -│ │ ▼ gRPC ext_proc │ │ -│ │ ┌───────────────────────────────────────────────────────────────────────────────┐ │ │ -│ │ │ llm-d Endpoint Picker (EPP) ─ InferencePool Controller │ │ │ -│ │ │ Scoring: prefix × 0.4 + kv_cache × 0.3 + queue_depth × 0.3 │ │ │ -│ │ └──────────────────────┬──────────────────────┬─────────────────────────────────┘ │ │ -│ │ │ HTTPS (TLS) │ HTTPS (TLS) │ │ -│ │ ▼ ▼ │ │ -│ │ ┌──────────────────────────────────────┐ ┌─────────────────────────────────────┐ │ │ -│ │ │ vLLM Pod ─ CUDA │ │ vLLM Pod ─ Neuron │ │ │ -│ │ │ Qwen3-Coder-30B FP8 · TP=1 │ │ Qwen3-Coder-30B BF16 · TP=8 │ │ │ -│ │ │ max_model_len=32768 │ │ max_model_len=16384 │ │ │ -│ │ │ LLMInferenceService / KServe │ │ InferenceService / KServe │ │ │ -│ │ └──────────────────────────────────────┘ └─────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ -│ │ OPERATOR LAYER │ │ -│ │ ┌───────────────────────────────────────────────────────────────────────────────┐ │ │ -│ │ │ NVIDIA GPU Operator: GPU Device Plugin · DCGM Exporter · Driver Manager │ │ │ -│ │ │ AWS Neuron Operator: Neuron Device Plugin · Neuron Scheduler · Neuron Mo│ │ │ -│ │ │ · KMM (kernel module management) │ │ │ -│ │ │ Node Feature Discovery (NFD) · cert-manager · LeaderWorkerSet │ │ │ -│ │ └───────────────────────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────────────────────┘ │ -└────────────────────────────────┴───────────────────────┴────────────────────────────────┘ - │ │ - ▼ ▼ -┌─────────────────────────────────────────────────────────────────────────────────────────┐ -│ AWS EC2 INFRASTRUCTURE │ -│ ┌────────────────────────────────────────┐ ┌─────────────────────────────────────┐ │ -│ │ g6e.2xlarge (NVIDIA L40S) │ │ inf2.24xlarge (Inferentia2) │ │ -│ │ ┌────────────────────────────────┐ │ │ ┌───────────────────────────┐ │ │ -│ │ │ 1× NVIDIA L40S GPU │ │ │ │ 6× Inferentia2 chips │ │ │ -│ │ │ 48 GB GDDR6X VRAM │ │ │ │ 12 NeuronCores/192GB │ │ │ -│ │ │ FP8/FP16/BF16/CUDA 12 │ │ │ │ BF16 · Neuron 2.28 │ │ │ -│ │ └────────────────────────────────┘ │ │ └───────────────────────────┘ │ │ -│ │ 8vCPU · 32GB RAM · 300GB EBS │ │ 96vCPU · 384GB · 300GB EBS │ │ -│ └────────────────────────────────────────┘ └─────────────────────────────────────┘ │ -│ ┌───────────────────────────────────────────────────────────────────────────────────┐ │ -│ │ Amazon EBS (gp3-csi) ─ PVCs: model-weights · NEFF cache · HF cache │ │ -│ └───────────────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────────────┘ -Request Flow: - Developer → Browser/IDE → AWS NLB (HTTPS) → Service Mesh Gateway → llm-d EPP - → EPP scores all vLLM replicas (queue depth, KV cache, prefix affinity) - → Routes to optimal vLLM pod (NVIDIA GPU or Inferentia2) → Model inference → Response -``` - -**Diagram notes:** - -- The **EPP scoring formula** (`prefix_cache × 0.4 + kv_cache × 0.3 + queue_depth × 0.3`) favors replicas with matching prefix cache hits, then available KV cache headroom, then lowest queue depth. -- **Both accelerator types serve the same `served-model-name`** (`Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`), allowing the EPP to treat them as a single pool. -- **TLS terminates at each vLLM pod** (self-signed certs), not at the gateway -- the gateway performs ext_proc routing on TLS-passthrough connections. -- **Dev Spaces extensions** make HTTPS calls from **inside the workspace pod** to the llm-d Gateway via Kubernetes ClusterIP DNS (`*.svc.cluster.local`). Traffic is routed through the EPP for intelligent endpoint selection, staying entirely cluster-internal with no external egress. - -### Inference Traffic Flow - -How AI code assistant extensions communicate with the LLM — every hop stays inside the cluster. - -![Inference Traffic Flow](images/architecture-traffic-flow.svg) - -The diagram illustrates the four-step request lifecycle: - -1. **Developer workspace** — AI extensions (Continue, Cline, Roo Code) running inside an OpenShift Dev Spaces workspace pod send OpenAI-compatible HTTPS requests to the llm-d Gateway using Kubernetes internal DNS. The DNS resolves to a ClusterIP; traffic never leaves the cluster network. -2. **Gateway (Envoy)** — The Kubernetes Gateway API terminates TLS and applies HTTPRoute rules. For inference endpoints (`/v1/chat/completions`, `/v1/completions`), the Envoy `ext_proc` filter invokes the EPP via gRPC to determine the target backend. -3. **EPP (Endpoint Picker)** — The llm-d Endpoint Picker scores all available model replicas using a weighted algorithm: prefix-cache affinity (0.4), KV-cache headroom (0.3), and queue depth (0.3). It returns the optimal pod endpoint to Envoy. -4. **Model replica (vLLM)** — Envoy routes the request to the selected vLLM pod. The model performs inference and returns the response through the same cluster-internal path. - -**KServe's `LLMInferenceService`** CR creates the entire serving stack declaratively — the model deployment, workload service, InferencePool, EPP instance, and HTTPRoute are all provisioned from a single resource definition, making the architecture fully GitOps-ready. - -### Architecture Diagram (SVG — Landscape) - -> **Tip:** Open each `.svg` file directly in your browser or IDE for full-resolution rendering. - -![Full-Stack Architecture Diagram](images/architecture-diagram-landscape.png) - -### Deployment View (Layered Architecture) - -Four-layer left-to-right flow: **Application/Workload** (Dev Spaces → Service Mesh Gateway → llm-d EPP → vLLM Pods) → **Operator/Driver** (11 operators including RHOAI/KServe, GPU Operator, Neuron Operator, Service Mesh, NFD, cert-manager, KMM, LeaderWorkerSet) → **Platform** (ROSA HCP 4.21) → **Infrastructure** (AWS EC2 + EBS + NVMe). Note: llm-d is not an operator -- it is CRDs and controllers deployed by RHOAI/KServe. - -![Deployment View — Layered Architecture](images/deployment-view-layered.png) - -### AWS Architectural View - -Enterprise private connectivity design: developers on **customer premises** connect via **AWS Direct Connect** (dedicated circuit) through a **Transit Gateway** to the **ROSA VPC**. An internal **NLB** distributes traffic to worker nodes across three Availability Zones. The Kubernetes API is accessible via **AWS PrivateLink**. No Internet Gateway -- source code never traverses the public internet. - -![AWS Architectural View](images/aws-architectural-view.png) - -### Red Hat Components View - -OpenShift-centric view: **external** components (developers, AWS NLB, EC2 nodes, EBS/NVMe storage) sit outside the cluster boundary. **All software components** -- Dev Spaces, Service Mesh Gateway, llm-d EPP, vLLM pods, 11 operators, platform services -- reside inside the OpenShift cluster. Organized by namespace: `openshift-devspaces`, `openshift-ingress`, `llm-d-multi-gpu`, plus operator namespaces. - -![Red Hat Components View](images/redhat-components-view.png) - -## Frontend: Developer IDE Environment - -### Deployment Option 1: Web IDE (Browser-Based) - -The developer opens the **Dev Spaces Dashboard** in a browser. A **Cloud Development Environment (CDE)** is created that includes: - -- **VS Code for the Web** (che-code) -- AI assistant extensions (Continue, Cline, Roo, Kilo, Claude Code CLI, GitHub Copilot CLI) -- Runtimes, source code, and plugins -- all executing **inside OpenShift** - -**Key characteristics:** - -- Most OSS AI assistants are **client-side in the workspace** -- HTTP calls go from the workspace pod to your LLM endpoint, **not** through a vendor's relay, when configured that way. -- **Red Hat AI / platform administrators** control routing: private model endpoint, hybrid, or approved external providers per policy. -- **No desktop install** -- works from any supported browser. -- Compatible with many tools: Kilo, Roo, Codex, CLIs (Claude Code, GitHub Copilot CLI), and others per your image and policy. - -![Dev Spaces AI Code Assistants Architecture](assets/devspaces-ai-code-assistants-architecture.png) - -### Deployment Option 2: Remote SSH (Thin Client Desktop IDE) - -For developers who prefer **native desktop IDEs** (Cursor, GitHub Copilot in VS Code, Kiro, etc.): - -- The **local IDE** is primarily a **UI shell**; heavy lifting can run **server-side** in the Dev Spaces workspace on OpenShift. -- Unlocks **official VS Code Marketplace** extensions (e.g., GitHub Copilot, C# Dev Kit) where your organization permits them. -- The desktop IDE may also connect to **third-party backends** (e.g., Cursor Server) **in addition to** your private model -- subject to your security review. - -**Remote SSH technical flow (Dev Spaces 3.27+):** - -1. **Authentication** -- Dev Spaces SSH integration uses `oc login --web`; the **cluster IDP** is the primary authentication gate. -2. **Tunnel** -- `oc port-forward -n ${namespace} ${workspace-pod} ${local-port}:2022` establishes an encrypted path; access is governed by **RBAC**. -3. **Ephemeral keys** -- A fresh SSH key pair is generated per workspace session and rotates on workspace restart. - -**Security properties:** - -- **SSHD on port 2022** is not exposed via **Route** or **Ingress**; it is reachable through the **authenticated Kubernetes API tunnel**. -- Traffic transits the **Kubernetes API server** (TLS + RBAC). -- SSH algorithms align with **UBI 10** cryptographic standards (FIPS-compatible configurations where required). - -![Dev Spaces Remote SSH Security Flow](assets/devspaces-remote-ssh-security-flow.png) - -### Current Implementation - -Our deployment uses **Option 1 (Web IDE)** with: - -- OpenShift Dev Spaces **3.27** -- **Continue** extension **v1.1.80** -- OpenAI-compatible provider targeting the **llm-d** gateway -- **Cline** extension (`saoudrizwan.claude-dev`) -- **Roo Code** extension (`RooVeterinaryInc.roo-cline`) -- OpenAI-compatible provider, agent-first UX -- **Three test users** (`dev-user1`, `dev-user2`, `dev-user3`) via **HTPasswd** IDP -- **Per-user resource governance** via **LimitRange** and **ResourceQuota** - -### AI Code Assistant Extension Comparison: Continue vs Cline vs Roo Code - -All three extensions are **Apache 2.0 licensed**, support **OpenAI-compatible API endpoints** (e.g., vLLM behind llm-d), and are available on both the **VS Code Marketplace** and **Open VSX** (compatible with Dev Spaces che-code). - - -| Capability | Continue (v1.1.80) | Cline (`saoudrizwan.claude-dev`) | Roo Code (v3.51+, `RooVeterinaryInc.roo-cline`) | -| -------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -| **Primary UX** | Chat, Edit, Agent, **Tab Autocomplete** | Autonomous agent sidebar | Multi-mode agent: **Code, Architect, Ask, Debug, Custom** | -| **Tab autocomplete** | **Yes** (dedicated feature with `tabAutocompleteModel`) | No | No | -| **File read/write** | Yes | Yes | Yes | -| **Terminal execution** | Yes | Yes | Yes (configurable allow/deny lists) | -| **MCP support** | Yes | Yes (MCP marketplace UX) | Yes | -| **Browser automation** | No | Yes | No | -| **Config format** | `~/.continue/config.yaml` (YAML, schema v1) — delivered via ConfigMap mounted at `/home/user/.continue/config.yaml` | Internal globalState (VS Code database) — delivered via `cline-settings-config` ConfigMap | VS Code `settings.json` (`roo-cline.*`) + `provider_profiles.json` — delivered via `roo-code-config` + `roo-code-provider-config` ConfigMaps | -| **Dev Spaces pre-config** | `postStart` writes `config.yaml` — **fully automated** | Extension installed by `postStart`; **manual UI setup required** (4 fields) | ConfigMap → `provider_profiles.json` + `settings.json` — **fully automated** | -| **Self-hosted vLLM setup** | Straightforward: `apiBase` + model in JSON/YAML | OpenAI-compatible base URL in UI | OpenAI-compatible base URL; **requires model to support native tool/function calling** | -| **Enterprise features** | CI/PR checks integration | SSO, policies, observability (enterprise tier) | Profiles, custom modes, skills as slash commands | -| **Installs (Marketplace)** | ~20M+ | ~3.5M+ | ~1.5M+ | -| **Origin** | Independent project (Continue Dev, Inc.) | Independent project (Saoud Rizwan) | **Fork of Cline** (now separate under Roo Code, Inc.) | - - -**Key trade-offs for private LLM deployments:** - -- **Continue** is the best fit when **tab autocomplete** is a priority -- it is the only one of the three with a dedicated autocomplete feature backed by your self-hosted model. Configuration via `config.yaml` is file-based and GitOps-friendly. Works out of the box after workspace startup. -- **Cline** provides the most mature autonomous agent with browser automation. It has the largest community but stores API config in VS Code's internal globalState database, making full pre-configuration impossible via ConfigMaps. Users must configure the API provider once through the Cline Settings UI (30 seconds, settings persist across restarts). -- **Roo Code** offers the richest **modal agent UX** (Architect, Debug, etc.) and configures natively via VS Code `settings.json`, which maps cleanly to Dev Spaces ConfigMaps. Roo Code requires the model to return **native OpenAI-style `tool_calls`** in the API response. This is achieved by configuring vLLM with the correct `--tool-call-parser` and `--reasoning-parser` for your model (see [Configuring vLLM for Roo Code Tool Calling](#configuring-vllm-for-roo-code-tool-calling-qwen3-coder)). - -**Recommendation:** Deploy all three and let developers choose. Continue covers autocomplete, Cline/Roo cover agentic workflows. The Dev Spaces ConfigMap approach (Step 15) supports installing and pre-configuring all three simultaneously. - -### Configuring vLLM for Roo Code Tool Calling (Qwen3-Coder) - -Roo Code's agent modes (Architect, Code, Debug, Orchestrator) depend on the model returning **structured OpenAI-style `tool_calls`** in the API response. Without the correct vLLM configuration, the model generates tool invocations as XML inside the `content` field (e.g., `...`), which Roo Code cannot parse, producing the error: - -> *"The model provided text/reasoning but did not call any of the required tools."* - -**Root Cause:** Qwen3-Coder uses a proprietary XML tool-call format and has a built-in "thinking mode" that emits `...` tokens before acting. Two vLLM flags are required to handle both correctly. - -#### Required vLLM Arguments - -Add these flags to `VLLM_ADDITIONAL_ARGS` in the `LLMInferenceService` spec: - - -| Flag | Value | Purpose | -| -------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `--tool-call-parser` | `qwen3_coder` | Parses Qwen3-Coder's XML tool-call format (``) and maps it to OpenAI `tool_calls` array — only activates when `tools` is present in the request | -| `--reasoning-parser` | `qwen3` | Separates `...` reasoning tokens into the `reasoning_content` field, preventing them from polluting the `content` field and confusing tool-call extraction | - - -> `**--enable-auto-tool-choice` is required**: Roo Code sends `tool_choice: "auto"` in every request that includes tools. vLLM rejects these requests with a 400 error unless `--enable-auto-tool-choice` is set. Without this flag, Roo Code is completely non-functional. -> -> **Side effect — Continue "Invalid tool name"**: With `--enable-auto-tool-choice`, vLLM runs the tool-call parser on ALL responses, including Continue's plain chat requests (which don't send tools). Qwen3-Coder sometimes spontaneously generates `` XML in coding responses; vLLM's parser extracts these as `tool_calls`, and Continue then fails with "Invalid tool name" because the tool names (Roo Code tools) don't match anything Continue knows about. **Fix**: Add a `systemMessage` to Continue's model config explicitly telling the model not to use XML tool-call syntax in its responses (see [Continue extension configuration](#14b-continue-extension-configuration)). This prevents the spontaneous XML generation at the source. - -The full `VLLM_ADDITIONAL_ARGS` for Qwen3-Coder-30B with tool calling and FP8 KV cache: - -``` ---disable-uvicorn-access-log --max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --kv-cache-dtype fp8 -``` - -> **Note:** `gpu-memory-utilization` was reduced from 0.95 to **0.90** because FP8 KV cache doubles the number of allocated cache blocks, leaving insufficient GPU memory for the vLLM sampler warmup (256 dummy requests). At 0.95, the sampler OOMs during `topk_topp` sorting. 0.90 provides sufficient headroom while still delivering significantly more KV cache capacity than BF16 at 0.95. - -#### Patching an Existing Deployment - -```bash -oc patch llminferenceservice qwen3-coder-fp8 -n ai-serving --type=json -p '[ - {"op":"replace","path":"/spec/template/containers/0/env/0/value", - "value":"--disable-uvicorn-access-log --max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --kv-cache-dtype fp8"} -]' -``` - -#### Verification - -Test from inside the cluster (or from any pod in the same namespace): - -```bash -curl -sk https://qwen3-coder-fp8-kserve-workload-svc.ai-serving.svc.cluster.local:8000/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8", - "messages": [ - {"role":"system","content":"You are a coding assistant."}, - {"role":"user","content":"Read the file /tmp/test.txt"} - ], - "tools": [{ - "type": "function", - "function": { - "name": "read_file", - "description": "Read contents of a file", - "parameters": { - "type": "object", - "properties": {"path": {"type":"string"}}, - "required": ["path"] - } - } - }], - "tool_choice": "auto", - "max_tokens": 200 - }' -``` - -**Expected (correct):** `"tool_calls": [{"function": {"name": "read_file", ...}}]` with `"finish_reason": "tool_calls"`. - -**Broken (without flags):** `"tool_calls": []` and `"content": "..."` -- Roo Code cannot parse this. - -#### Critical: Use `openai` Provider (Not `openai-native`) With Streaming Disabled - -Roo Code 3.x offers two OpenAI-compatible provider types: `openai` and `openai-native`. **You must use `openai` with `openAiStreamingEnabled: false`** for tool calling to work correctly with self-hosted vLLM. - - -| Provider | Streaming | Tool calling behavior | -| --------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `openai` with `openAiStreamingEnabled: false` | Non-streaming | **Correct** — sends `tools` natively; vLLM returns complete `tool_calls` array in one response. | -| `openai-native` | **Always streams** (no streaming control) | **Broken** — the `qwen3_coder` streaming parser emits tool-call XML as `content` chunks instead of `tool_calls` deltas. Roo Code never sees structured tool calls. | - - -**Why `openai-native` breaks:** In Roo Code 3.37+, both `openai` and `openai-native` send `tools: [...]` natively in the request body. The only meaningful difference is that `openai-native` always uses streaming while `openai` respects `openAiStreamingEnabled: false`. The vLLM `qwen3_coder` streaming parser has known bugs (tool-call XML emitted as raw content, incorrect `finish_reason`). Non-streaming mode processes the complete response at once and works correctly. - -**Symptom when misconfigured:** Roo Code reports *"You did not use a tool in your previous response"* on every turn, and the `api_conversation_history.json` shows `` XML in the assistant's `content` instead of structured `tool_calls`. - -#### Enterprise Endpoint Architecture: llm-d Gateway with EPP Routing - -All extensions must connect through the **llm-d Gateway** (Kubernetes Gateway API + Envoy), which routes inference requests through the **EPP (Endpoint Picker)** for intelligent load balancing. This is the Red Hat recommended architecture for production llm-d deployments. - - -| Endpoint | Protocol | EPP Routing | Use with Extensions? | -| ----------------------------------------- | -------- | ------------------------ | ------------------------------ | -| `https://llm-d-gateway-...:443/v1` | HTTPS | **Yes** -- InferencePool | **Yes** -- production endpoint | -| `https://...-kserve-workload-svc:8000/v1` | HTTPS | No -- direct to pod | Fallback only | -| `http://...-epp-service:9002/v1` | gRPC | N/A (ext_proc) | **No** -- not HTTP | - - -#### Internal Network Traffic Flow (Cluster-Internal Only) - -All traffic from DevSpaces extensions to the model stays **entirely within the OpenShift cluster**. No inference data ever traverses the public internet or leaves the cluster network. See the [Inference Traffic Flow diagram](#inference-traffic-flow) for a visual representation. - -**Key networking properties:** - -- Extensions use Kubernetes internal DNS (`*.svc.cluster.local`) to reach the llm-d Gateway — DNS resolves to a **ClusterIP** on the pod network, traffic never leaves the cluster -- The llm-d Gateway terminates TLS and invokes the EPP via `ext_proc` for intelligent backend selection before routing to the optimal vLLM pod -- The Gateway has a LoadBalancer service (external ELB) for external clients, but internal extensions use the ClusterIP — the ELB is never involved in IDE-to-model traffic -- End-to-end TLS is available via Service Mesh for pod-to-pod mTLS encryption - -#### Critical: Disable Streaming (vLLM 0.13.x) - -The `qwen3_coder` streaming tool-call parser in vLLM 0.13.0+rhai11 has a **known bug**: when streaming is enabled, tool-call XML is emitted as raw `content` chunks instead of being parsed into `tool_calls` deltas. The `finish_reason` comes back as `"stop"` instead of `"tool_calls"`. - -**Non-streaming requests work correctly** -- tool calls are properly extracted into the `tool_calls` array. - -#### Complete Roo Code Provider Configuration - -The `roo-code-provider-config` ConfigMap in each DevSpaces namespace must contain: - -```json -{ - "currentApiConfigName": "llm-d-qwen3-coder", - "apiConfigs": { - "llm-d-qwen3-coder": { - "id": "llm-d-qwen3-coder", - "apiProvider": "openai", - "openAiBaseUrl": "https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1", - "openAiApiKey": "EMPTY", - "openAiModelId": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8", - "openAiStreamingEnabled": false - } - }, - "modeApiConfigs": { - "code": "llm-d-qwen3-coder", - "architect": "llm-d-qwen3-coder", - "ask": "llm-d-qwen3-coder", - "debug": "llm-d-qwen3-coder", - "orchestrator": "llm-d-qwen3-coder" - } -} -``` - -> **Important — ConfigMap mount path:** The ConfigMap annotation `controller.devfile.io/mount-path` must be set to `/checode/remote/data/User/globalStorage/rooveterinaryinc.roo-cline/settings` (Roo Code 3.x reads from `/checode/remote/data/`, not `/checode/checode-linux-libc/`). The `mount-as: subpath` annotation causes the `provider_profiles.json` key to land at `/provider_profiles.json`. - -> **Startup script:** Because startup commands may run before ConfigMap subpath mounts are processed, the DevWorkspace `install-roo-code` command explicitly copies the config file: -> -> ```bash -> mkdir -p /checode/remote/data/User/globalStorage/rooveterinaryinc.roo-cline/settings -> if [ -f /checode/checode-linux-libc/User/globalStorage/rooveterinaryinc.roo-cline/provider_profiles.json ]; then -> cp /checode/checode-linux-libc/User/globalStorage/rooveterinaryinc.roo-cline/provider_profiles.json \ -> /checode/remote/data/User/globalStorage/rooveterinaryinc.roo-cline/settings/provider_profiles.json -> fi -> ``` - -This setting is a required workaround -- **do not enable streaming for tool calls.** Validated on vLLM 0.14.1+rhai4 (RHOAI 3.4.0-ea.1): the streaming `qwen3_coder` parser still emits raw `` XML as content with `finish_reason: stop` instead of properly structured `tool_calls` deltas with `finish_reason: tool_calls`. This bug persists in the Red Hat `rhai4` patch build even though it is fixed in upstream vLLM 0.14.x. **Non-streaming tool calls work correctly** and Roo Code operates as expected with `openAiNativeStreamingEnabled: false`. Re-test streaming when RHOAI 3.4 GA ships. - -#### Tool-Call Parser Compatibility Matrix - - -| Model | `--tool-call-parser` | `--reasoning-parser` | vLLM Version Required | -| -------------------------------- | -------------------- | -------------------- | ------------------------------- | -| Qwen3-Coder-30B-A3B-Instruct-FP8 | `qwen3_coder` | `qwen3` | 0.13.0+ (RHOAI vLLM) | -| Qwen3-30B-A3B (non-Coder) | `qwen3_xml` | `qwen3` | 0.13.0+ | -| Qwen3.5-35B-A3B-FP8 | `qwen3_coder` | `qwen3` | 0.18+ (community vLLM required) | -| DeepSeek-R1 / V3 | `hermes` | `deepseek_r1` | 0.13.0+ | -| Granite-3.x | `granite` | *(none)* | 0.13.0+ | -| Gemma 4 26B-A4B-it | `gemma4` | `gemma4` | **0.14.1+** (RHOAI 3.4 GA) | -| Nemotron 3 Nano 30B-A3B-FP8 | `nemotron_nano` | *(none)* | **0.14+** (RHOAI 3.4 GA) | - - -#### GPU Deployment Strategy: Avoiding Rollout Deadlocks - -When running a **single GPU replica**, the default Kubernetes `RollingUpdate` strategy causes a deadlock: the new pod cannot schedule because the old pod still holds the GPU, and the old pod won't be terminated until the new pod is ready. The `LLMInferenceService` controller manages the underlying Deployment and continuously reconciles the strategy back to `RollingUpdate`, so patching the Deployment directly to `Recreate` does not persist. - -**Workarounds:** - -1. **Manual rollout** (immediate): Delete the old pod and scale its ReplicaSet to 0 to free the GPU: - ```bash - # Identify old and new ReplicaSets - oc get rs -n ai-serving -l app.kubernetes.io/name=qwen3-coder-fp8 - - # Delete the running pod from the OLD ReplicaSet - oc delete pod -n ai-serving --grace-period=0 --force - - # Scale down the old ReplicaSet - oc scale rs --replicas=0 -n ai-serving - ``` -2. **Scale to zero first** (controlled): Scale replicas to 0 in the LLMInferenceService, apply the config change, then scale back to 1: - ```bash - oc patch llminferenceservice qwen3-coder-fp8 -n ai-serving \ - --type=merge -p '{"spec":{"replicas":0}}' - # Wait for pod termination, then apply config changes, then: - oc patch llminferenceservice qwen3-coder-fp8 -n ai-serving \ - --type=merge -p '{"spec":{"replicas":1}}' - ``` -3. **Multi-GPU pool** (production): Maintain 2+ GPU nodes so rolling updates can proceed normally. This is the recommended approach for production but increases cost. - -### Production Scaling Considerations - -- **Factory URL** pattern for **self-service** workspace creation at **hundreds of users** -- **Kubernetes Image Puller (KIP)** to **pre-cache** IDE and extension images -- **Advanced authorization** via **RBAC** (and organizational policy) -- **Pod anti-affinity** and capacity planning for **high availability** of the Dev Spaces control plane - -## Backend: Model Hosting and Intelligent Routing - -### Model: Qwen3-Coder-30B-A3B-Instruct - -Two deployment variants validated: - - -| Variant | Accelerator | Instance | Quantization | KV Cache Dtype | TP | Prefix Caching | max_model_len | Model ID | -| ------------------- | -------------------- | ------------- | ------------- | ------------------------ | --- | ----------------------------- | ------------- | --------------------------------------- | -| **NVIDIA L40S** | 1x L40S (48 GB VRAM) | g6e.2xlarge | FP8 (~16 GB) | **FP8** (2x compression) | 1 | **Enabled** | 32,768 | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` | -| **AWS Inferentia2** | 8 NeuronCores | inf2.24xlarge | BF16 (~57 GB) | BF16 (default) | 8 | **Disabled** (MoE constraint) | **8,192** | `Qwen/Qwen3-Coder-30B-A3B-Instruct` | - - -### Deployment via LLMInferenceService (NVIDIA) - -The `**LLMInferenceService`** CRD (`serving.kserve.io/v1alpha1`, RHOAI 3.3) provisions the full serving stack in one resource: - -- **vLLM** model server pods with **KServe** RawDeployment integration -- **Inference Scheduler** (**EPP** -- Endpoint Picker) for intelligent routing -- **InferencePool** and **InferenceModel** resources (Gateway API Inference Extension) -- **HTTPRoute** through the **Service Mesh** gateway -- **TLS** certificates for pod-to-pod encryption (automatically generated) - -The EPP runs as a sidecar-like deployment alongside the model pods. It communicates with the Service Mesh gateway via the `ext_proc` (external processing) filter in Envoy. - -### Deployment via InferenceService (Inferentia) - -Inferentia models are deployed using the standard `**InferenceService`** CRD (`serving.kserve.io/v1beta1`) with a custom `**ServingRuntime**` (`vllm-neuron-runtime`), since the `LLMInferenceService` CRD does not support Neuron accelerators. Key differences: - -- Container image: `public.ecr.aws/neuron/pytorch-inference-vllm-neuronx:0.13.0-neuronx-py312-sdk2.28.0-ubuntu24.04` -- Security capabilities: `IPC_LOCK` and `SYS_ADMIN` required -- Scheduler: `neuron-scheduler` (required for multi-core topology-aware allocation) -- Metrics port: `8080` (vs `8000` for CUDA) -- Protocol: HTTP (vs HTTPS for CUDA with KServe-managed certs) - -### llm-d Intelligent Routing - -The **llm-d Endpoint Picker (EPP)** scores replicas using configurable weighted scorers: - - -| Scorer | Weight | Effect | -| ----------------------------- | ------ | --------------------------------------------------------------- | -| `queue-scorer` | 2 | Prefer pods with shorter request queues | -| `kv-cache-utilization-scorer` | 2 | Prefer pods with more available KV cache headroom | -| `prefix-cache-scorer` | 3 | Prefer pods that already hold the request's **prefix** in cache | - - -The prefix-cache scorer carries the **highest weight**. When multiple developers share a common **system prompt** (e.g., Continue/Cline default prompts) or similar file context, the EPP **co-locates** their requests on the same replica to maximize **KV cache hits** and reduce **time-to-first-token (TTFT)** -- a significant advantage over round-robin for code assistant workloads. - -### Multi-Accelerator Scaling Phases - - -| Phase | Infrastructure | Description | Hourly Cost | Status | -| ----- | --------------------------------- | --------------------------------------------------------------------------- | ----------- | ---------------------------- | -| **1** | 1x g6e.2xlarge | Single L40S GPU, FP8 model, llm-d gateway | $2.24 | **Verified** | -| **2** | 2x g6e.2xlarge | Two L40S GPUs, load-aware + prefix-cache routing | $4.48 | **Verified** | -| **3** | 2x g6e.2xlarge + 1x inf2.24xlarge | Heterogeneous CUDA + Neuron routing | $9.18 | **Verified** (April 7, 2026) | -| **4** | 1x inf2.48xlarge | Dual Qwen3-Coder-30B instances (TP=8 × 2, 16 of 24 cores) | $12.98 | **Planned** | -| **5** | 2x g6e.2xlarge | Two L40S GPUs (48 GB VRAM each), FP8, llm-d EPP — re-baseline with GuideLLM | $4.48 | **Planned** | - - -**Phase 3: Heterogeneous llm-d Requirements** - -Routing a single model across NVIDIA GPU and AWS Inferentia2 backends through a unified llm-d gateway requires aligning several dimensions that differ between the two accelerator stacks: - - -| Requirement | NVIDIA L40S (LLMInferenceService) | AWS Inferentia2 (InferenceService) | Alignment Needed | -| -------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------ | -| **Model name** | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` | `qwen3-coder-neuron` | Must use the **same served model name** for the EPP InferenceModel | -| **Protocol** | HTTPS (KServe auto-TLS) | HTTP (no TLS) | Enable vLLM native TLS via `--ssl-certfile`/`--ssl-keyfile` on Inferentia pods | -| **Metrics endpoint** | `https://pod:8000/metrics` | `http://pod:8000/metrics` | Both HTTPS after enabling vLLM native TLS on Inferentia | -| **Namespace** | `llm-d-gpu` | `llm-serving` | Deploy both in same namespace; use `nodeSelector` for machine pool targeting | -| **PVC / Storage** | Model on EBS PVC via KServe storage initializer (`kserve-provision-location` volume override) | Model on shared EBS PVC (`model-storage`) | Separate PVCs per namespace; no cross-namespace PVC sharing | -| **SELinux** | Managed by KServe (unique MCS per pod) | Requires explicit `seLinuxOptions.level` for shared PVC | Pods sharing a PVC must set matching `seLinuxOptions.level` (e.g., `s0:c0,c0`) | -| **Prefix caching** | Enabled (FP8 on CUDA) | Disabled (MoE on NxD constraint) | EPP prefix-cache scorer will always prefer GPU replicas for cache-warm prompts | -| **Quantization** | FP8 (8-bit) | BF16 (16-bit) | Output quality may differ slightly; acceptable for code assistant use | - - -**Heterogeneous Deployment: Same-Namespace Native TLS (Verified April 8, 2026)** - -> **All model pods must run in a single namespace.** This is an architectural requirement of llm-d, not a preference. The `InferencePool` label selector is **namespace-scoped** — the EPP can only discover and route to pods within its own namespace. Cross-namespace deployments require a proxy, which breaks EPP scoring (KV-cache utilization, queue depth) and adds operational complexity with zero benefit. During testing, the proxy approach showed 10-21% throughput degradation and timeout errors at high concurrency due to broken EPP metrics visibility. - -**Why same-namespace matters for llm-d:** - -1. **EPP discovery is namespace-scoped**: The `InferencePool` uses Kubernetes label selectors to discover backend pods. These selectors only match pods within the same namespace. Pods in other namespaces are invisible to the EPP. -2. **EPP scoring requires direct vLLM metrics**: The EPP scrapes each pod's `/metrics` endpoint to make routing decisions based on KV-cache utilization and queue depth. A proxy between the EPP and vLLM returns the proxy's metrics, not the model server's, rendering intelligent routing useless. -3. **TLS consistency**: All pods behind the same `InferencePool` must serve HTTPS on the same port with consistent TLS. Sharing the TLS secret created by `LLMInferenceService` is straightforward within one namespace. -4. **Simplified operations**: One namespace means one set of RBAC, one `InferencePool`, one EPP router, and unified `oc get pods` visibility across all backends. - -**Implementation details:** - -1. **TLS on all endpoints**: vLLM natively supports HTTPS via `--ssl-certfile` and `--ssl-keyfile`. The Inferentia pods reuse the self-signed TLS secret created by the `LLMInferenceService` (e.g., `qwen3-coder-fp8-kserve-self-signed-certs`), mounted at `/etc/certs`. Readiness/liveness probes use `scheme: HTTPS`. -2. **Node targeting via `nodeSelector` + taints/tolerations**: Both accelerator types coexist in one namespace, scheduled to the correct hardware via: - - NVIDIA pods: `nodeSelector: { node.kubernetes.io/instance-type: g6e.2xlarge }` + toleration for `nvidia.com/gpu=present:NoSchedule` - - Inferentia pods: `nodeSelector: { node.kubernetes.io/instance-type: inf2.24xlarge }` + `schedulerName: neuron-scheduler` + toleration for `aws.amazon.com/neuroncore=present:NoSchedule` - **Machine pool taints** prevent non-accelerator workloads from consuming CPU/memory on expensive GPU and Inferentia nodes. Each accelerator machine pool should be created with a taint: - - GPU pool: `--taints="nvidia.com/gpu=present:NoSchedule"` - - Inferentia pool: `--taints="aws.amazon.com/neuroncore=present:NoSchedule"` - Model deployments must include matching `tolerations` in their pod spec. The `LLMInferenceService` CR supports tolerations in its `spec.template`; standalone Inferentia `Deployments` add them directly to `spec.template.spec.tolerations`. Without these taints, OpenShift platform pods (dashboards, controllers, operators) will fill GPU node CPU and block model scheduling. - Each accelerator type uses its **own PVC** (NVIDIA uses KServe storage initializer; Inferentia uses an EBS-backed PVC). SELinux `seLinuxOptions.level` is only needed for Inferentia pods sharing a PVC. -3. **Model name alignment**: Both backends must use the **same** `--served-model-name` value so the EPP treats them as interchangeable endpoints: - - | Component | NVIDIA (LLMInferenceService) | Inferentia (Deployment) | - | --------------------- | ---------------------------------------------- | -------------------------------------------------- | - | `--served-model-name` | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` (auto) | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` (explicit) | - | Actual model | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` | `Qwen/Qwen3-Coder-30B-A3B-Instruct` (BF16) | - | InferencePool labels | Auto-applied by LLMInferenceService | Manually applied to match pool selector | - -4. **InferencePool label matching**: The Inferentia Deployment must carry labels matching the `InferencePool` selector created by the `LLMInferenceService`: - - `app.kubernetes.io/name: qwen3-coder-fp8` - - `app.kubernetes.io/part-of: llminferenceservice` - - `app.kubernetes.io/component: llminferenceservice-workload` - - `kserve.io/component: workload` - - `llm-d.ai/role: both` - -**Verified routing behavior (April 8, 2026):** - -- EPP auto-discovers all pods (2 NVIDIA + 1 Inferentia) via InferencePool label selector -- Under 10 concurrent requests: 4 routed to Inferentia, 6 to NVIDIA (natural EPP load balancing) -- At low concurrency, EPP prefers NVIDIA pods (faster response, prefix cache support) -- Inferentia pod serves as overflow capacity when NVIDIA pods are busy -- **Zero errors** across all concurrency levels in GuideLLM sweep (7,781 requests) - -Full step-by-step deployment procedure is in `private_code_assistant_TEST_PLAN.md` Phase D. - -**neuron-scheduler / OVN-Kubernetes annotation race (resolved):** - -On ROSA HCP with OVN-Kubernetes CNI, the AWS `neuron-scheduler` writes device-allocation annotations that **overwrite** the OVN `k8s.ovn.org/pod-networks` annotation, preventing pod networking. This is resolved via a **MutatingAdmissionWebhook** that preserves the OVN annotation (see Appendix B). Long-term fix: Neuron DRA driver (see Appendix A). - -### Prefix Caching: Model Compatibility - -Prefix caching is a **platform requirement** for full **llm-d** benefit; support depends on **model architecture** and **runtime**: - - -| Model Type | CUDA | Neuron (Inferentia2) | -| -------------------------------------------- | ----------------------- | ------------------------------------------- | -| Dense (e.g., Llama-3.x, CodeLlama, Qwen3-8B) | Prefix caching: **YES** | Prefix caching: **YES** | -| MoE (e.g., Qwen3-Coder-30B-A3B) | Prefix caching: **YES** | Prefix caching: **NO** (NxD MoE constraint) | - - -**Recommendation:** Prefer **dense** models on **Inferentia** when **prefix-cache-aware** routing is critical. **MoE** on **CUDA** gets the full llm-d feature set. Prefix caching remains a core platform requirement for all supported non-MoE models. - -### Accelerator Selection Guide: NVIDIA GPU vs AWS Inferentia2 - -Choosing the right accelerator for a given model depends on the model architecture, required features, context length needs, quantization format, and cost constraints. This guide captures the trade-offs validated during this project. - -#### Decision Matrix - - -| Factor | NVIDIA L40S (CUDA) | AWS Inferentia2 (Neuron) | Recommendation | -| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | -| **MoE models** (e.g., Qwen3-Coder-30B-A3B, Mixtral) | Full support. FP8 quantization, prefix caching, TP=1 possible. | Supported but **no prefix caching** (NxD MoE constraint), BF16 only, requires TP=8+. | **NVIDIA preferred** — MoE models get the full llm-d feature set on CUDA | -| **Dense models** (e.g., Llama-3.x, CodeLlama, Qwen3-8B) | Full support. FP8/FP16/BF16, prefix caching, flexible TP. | Full support. Prefix caching **YES**, BF16, Neuron NEFF compilation required. | **Either** — dense models work well on both; Inferentia is cost-competitive for dense models with prefix caching | -| **FP8 quantization** | Native hardware support (FP8 Tensor Cores on Ada Lovelace+). 2x memory savings → TP=1 for 30B models. | **Not supported** — NeuronCore-v2 has no FP8 ALU. BF16 only. | **NVIDIA required** for FP8. Inferentia forces BF16 (2x memory → 2x TP → higher instance cost) | -| **Prefix caching** | Always available (CUDA + vLLM). Full llm-d EPP prefix-cache scoring. | Available for **dense models only**. Disabled for MoE models on NxD. | **NVIDIA preferred** when prefix caching is critical across all model types | -| **Context window** | Up to 32,768 tokens on L40S (48 GB VRAM, FP8, TP=1). | **8,192 tokens max** on inf2.24xlarge (128 GB HBM, BF16, TP=8). 16,384 on inf2.48xlarge. | **NVIDIA preferred** for long-context workloads (32k+). Inferentia limited by NEFF activation buffers. | -| **Cold-start time** | ~3-5 min with PVC cache (GPU load only; download skipped). ~7-13 min cold. | **30-45 min** (download + Neuron NEFF compilation). ~10s on warm PVC cache. | **NVIDIA preferred** for environments requiring fast scaling. Both benefit from PVC cache strategy. | -| **Single-user latency** | TTFT: ~62ms, ITL: ~10ms (FP8, 512 tokens). | TTFT: ~480ms, ITL: ~57ms (BF16, 512 tokens). | **NVIDIA preferred** (~8x faster ITL) for latency-sensitive interactive coding | -| **Peak throughput** | ~1,690 tok/s per L40S. | ~137 tok/s per inf2.24xlarge (Qwen3-Coder MoE). | **NVIDIA preferred** for throughput. Inferentia is effective as overflow capacity behind llm-d. | -| **Cost per instance** | $2.24/hr (g6e.2xlarge). | $6.49/hr (inf2.24xlarge). | **NVIDIA 2.9x cheaper** for equivalent model on this architecture | -| **Cost per 1M output tokens** | ~$0.18 (at peak throughput). | ~$13.16 (at peak throughput for MoE). | **NVIDIA 73x cheaper** per token for MoE. Gap narrows significantly for dense models. | -| **Scheduling complexity** | Standard Kubernetes scheduler. No annotation conflicts. | Requires `neuron-scheduler` + MutatingAdmissionWebhook to fix OVN annotation race on ROSA HCP. | **NVIDIA simpler** operationally. Inferentia requires additional workarounds on ROSA. | -| **Model format flexibility** | FP8, FP16, BF16, INT8, GPTQ, AWQ, GGUF via vLLM. | **BF16 only** via NxD-Inference. Limited INT8 for select operations. | **NVIDIA preferred** for quantization flexibility | - - -#### When to Choose Inferentia2 - -Despite NVIDIA's advantages in most dimensions, Inferentia2 is the right choice in specific scenarios: - -1. **Dense model cost optimization**: For dense models (not MoE), Inferentia2 supports prefix caching and the per-token cost gap narrows. On dense models like Llama-3-8B, Inferentia2 achieves comparable throughput to L40S at lower cost per NeuronCore-hour. -2. **Overflow / burst capacity**: In a heterogeneous llm-d deployment, Inferentia pods absorb overflow traffic when NVIDIA pods are saturated. The EPP automatically routes requests away from busy NVIDIA pods to available Inferentia endpoints. -3. **AWS commitment / reserved capacity**: Organizations with existing Inferentia reserved instances or AWS Neuron investment can leverage those commitments for inference workloads that don't require FP8 or long context. -4. **Future-proofing for Trainium2/Inferentia3**: AWS Trainium2 adds FP8 hardware support. Early Inferentia2 deployment builds operational expertise with Neuron tooling that transfers to next-gen silicon. - -#### When to Choose NVIDIA GPU - -NVIDIA GPUs are preferred for: - -1. **MoE models** — prefix caching, FP8 quantization, and lower TP requirements make NVIDIA the clear choice for Mixture-of-Experts architectures. -2. **Low-latency interactive coding** — 10ms ITL vs 57ms ITL matters for real-time code completion and streaming chat responses. -3. **Long context windows** — 32,768 tokens on L40S vs 8,192 on inf2.24xlarge. Code review, large file analysis, and multi-file context require NVIDIA. -4. **Fast scaling / ephemeral environments** — 2-5 min cold start vs 30-45 min compilation. NVIDIA is the only option when rapid elasticity is needed. -5. **Cost efficiency** — for the Qwen3-Coder-30B MoE model specifically, NVIDIA L40S is 73x cheaper per output token at peak throughput. - -#### Summary Recommendation for This Project - -For the **Qwen3-Coder-30B-A3B (MoE)** model used in this private code assistant platform: - -- **Primary fleet: NVIDIA L40S (g6e.2xlarge)** — FP8, prefix caching, 32k context, 10ms ITL, $0.18/1M tokens -- **Overflow capacity: Inferentia2 (inf2.24xlarge)** — BF16, no prefix caching, 8k context, 57ms ITL, useful for absorbing traffic spikes behind llm-d -- **If switching to a dense model** (e.g., Qwen3-8B, CodeLlama-34B): re-evaluate Inferentia2 as a primary fleet option — prefix caching support and narrower throughput gap make it competitive - -### Gateway URL - -``` -https://.us-east-2.elb.amazonaws.com/llm-d-gpu/qwen3-coder-fp8 -``` - ---- - -## Test Results - -### Phase 1: Single NVIDIA GPU Node - -**Configuration:** 1x g6e.2xlarge, `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`, TP=1, prefix caching enabled, `max-model-len=32768`. - - -| Test | Result | -| ------------------------------------- | ---------------------------------------------------------------------------------------------- | -| `GET /v1/models` | Model info returned; `max_model_len: 32768` | -| `POST /v1/chat/completions` | ~1.3 s for short completions (single-turn, ~50 output tokens) | -| `POST /v1/completions` | ~1.8 s for code completions | -| Prefix cache (repeated system prompt) | `prefix_cache_queries_total`: 18 -> 141; `prefix_cache_hits_total`: 0 -> 64 (**45% hit rate**) | -| Continue extension (Dev Spaces) | Model appears in dropdown, chat completions work through llm-d gateway | -| Cline extension (Dev Spaces) | Connected to llm-d endpoint, code generation functional | - - -### Phase 2: Two NVIDIA GPU Nodes with llm-d Load Balancing - -**Configuration:** 2x g6e.2xlarge (2 replicas), same model and settings as Phase 1. - - -| Test | Result | -| ------------------------------- | ------------------------------------------------------------------------------------------------------ | -| EPP pod discovery | Both vLLM pods discovered; metrics refresher active | -| Load distribution (20 requests) | Pod 1: 14 requests, Pod 2: 6 requests (EPP weighted, not round-robin) | -| Prefix-cache-aware routing | 6 requests with **identical system prompt** -> **all routed to Pod 1** (which had the prefix cached) | -| Pod 1 prefix cache hits | 64 -> 224 (160 new hits from the 6 same-prompt requests) | -| Pod 2 prefix cache hits | 0 new hits for same-prompt requests -- **correct EPP behavior** | -| KV cache distribution | Pod 1: ~12% utilization, Pod 2: ~4% utilization (proportional to load) | -| Multi-user Dev Spaces | Three test users (`dev-user1`, `dev-user2`, `dev-user3`) configured, workspaces point to llm-d gateway | - - -### Phase 3: Heterogeneous Routing (CUDA + Inferentia) - -**Status: VERIFIED** (April 8, 2026) -- Same-namespace heterogeneous llm-d routing verified across 2 NVIDIA L40S + 1 Inferentia2 endpoints with native TLS. - -**Verified capabilities:** - -- Qwen3-Coder-30B on Inferentia2 (8 NeuronCores, TP=8) serves inference via standalone Deployment with matching InferencePool labels -- Neuron NEFF compilation takes 30-45 min (cold cache) or ~10s (PVC cache hit); model loading takes ~15 min; requires **200Gi memory limit** during compilation -- All caches (HF_HOME, NEURON_COMPILE_CACHE_URL) on EBS-backed PVC for persistence across restarts -- All 3 pods serve the same `--served-model-name` (`Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`) over HTTPS -- **llm-d EPP auto-discovers all 3 pods** via InferencePool label selector (same namespace) -- EPP routes 4/10 concurrent requests to Inferentia, 6/10 to NVIDIA (natural load balancing) -- MutatingAdmissionWebhook resolves the OVN annotation race (see Appendix B) - -**Default method for heterogeneous GPU model deployments:** - -1. Create a single namespace (e.g., `llm-d-multi-gpu`) for all model backends -2. Deploy NVIDIA models via `LLMInferenceService` — this creates the `InferencePool`, EPP router, TLS secrets -3. Deploy Inferentia models as a standard Kubernetes `Deployment` with: - - Labels matching the `InferencePool` selector (see Phase 3 requirements table) - - `--served-model-name` matching the NVIDIA model name - - `--ssl-certfile`/`--ssl-keyfile` using the same TLS secret from step 2 - - `nodeSelector` for the Inferentia instance type + `schedulerName: neuron-scheduler` - - EBS PVC for model and NEFF cache, memory limits ≥200Gi -4. The EPP automatically discovers all pods via label selector — no additional configuration needed -5. Update Dev Spaces extension ConfigMaps to point to the workload service in the new namespace - -**Architecture diagram (verified):** - -``` -Client → llm-d EPP → ┬─ NVIDIA Pod 1 (g6e, FP8, HTTPS:8000) ← same namespace - ├─ NVIDIA Pod 2 (g6e, FP8, HTTPS:8000) ← same namespace - └─ Inferentia Pod (inf2, BF16, HTTPS:8000) ← same namespace, native TLS -``` - -### Inferentia Standalone Performance (Pre-llm-d) - -For detailed GuideLLM benchmark results on Inferentia2, see `inferentia_vllm_test_results.md`: - -- **Single-user**: ~24-25 output tokens/sec, 609-632 ms TTFT, 39 ms ITL (inf2.48xlarge, TP=16) -- **Max throughput**: ~278 output tok/s aggregate at 32 concurrent requests -- **NeuronCore utilization**: 16 of 24 cores usable on inf2.48xlarge (33% waste due to TP divisibility) - -### Performance Benchmark: llm-d on NVIDIA L40S (FP8) - -**Date:** April 2, 2026 -**Environment:** 2x g6e.2xlarge (NVIDIA L40S, 48 GB VRAM each) via `LLMInferenceService` with llm-d EPP routing -**Model:** `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` (MoE, 30B params, FP8 quantized, TP=1) -**Settings:** prefix caching enabled, `max-model-len=32768` -**Path:** Client -> AWS ELB (HTTPS) -> Service Mesh 3 Gateway -> llm-d EPP -> vLLM pod - -#### Test 1: Single-User Streaming Latency - -5 runs per configuration, measuring Time to First Token (TTFT), Inter-Token Latency (ITL), and output throughput. - - -| Config (in/out) | Avg TTFT (ms) | Avg ITL (ms) | Output tok/s | Avg Output Tokens | -| --------------- | ------------- | ------------ | ------------ | ----------------- | -| 128 / 128 | 284 | 10.3 | 80.7 | 128 | -| 128 / 256 | 278 | 10.3 | 87.9 | 256 | -| 512 / 256 | 269 | 10.3 | 87.7 | 256 | -| 512 / 512 | 267 | 10.4 | 92.0 | 512 | -| 1024 / 512 | 284 | 10.4 | 91.7 | 512 | - - -**Key findings:** - -- **TTFT is consistently low** at 265-335 ms across all prompt sizes (128-1024 tokens), indicating fast prefill even through the full gateway stack. -- **ITL is stable at ~10.3-10.4 ms** regardless of prompt or output size, showing predictable decode performance. -- **Single-user throughput scales** from 80.7 tok/s (short outputs) to 92.0 tok/s (long outputs) as the decode phase amortizes the prefill overhead. - -#### Test 2: Concurrent Load (Aggregate Throughput) - -Multiple concurrent requests (512 prompt tokens, 128 output tokens), distributed across 2 vLLM replicas via llm-d EPP. - - -| Concurrency | Avg Latency (s) | P50 Latency (s) | P95 Latency (s) | Aggregate tok/s | Req/s | -| ----------- | --------------- | --------------- | --------------- | --------------- | ----- | -| 1 | 2.097 | 2.092 | 2.192 | 60.7 | 0.47 | -| 2 | 2.487 | 2.482 | 2.573 | 100.1 | 0.78 | -| 4 | 2.851 | 2.788 | 3.034 | 176.0 | 1.37 | -| 8 | 3.358 | 3.337 | 3.658 | 294.8 | 2.30 | -| 16 | 4.032 | 4.032 | 4.440 | 482.2 | 3.77 | - - -**Key findings:** - -- **Near-linear throughput scaling** up to 8 concurrent requests: from 60.7 tok/s (1 user) to 294.8 tok/s (8 users), a 4.9x increase. -- **At 16 concurrent**, aggregate throughput reaches **482.2 tok/s** (7.9x vs single-user), with latency increasing moderately to ~4s (1.9x the single-user latency). -- **P95 latencies remain bounded**: even at 16 concurrent, P95 is 4.440s (only 10% above P50), indicating stable tail latency. -- **llm-d EPP effectively distributes load** across both GPU replicas, as evidenced by the near-linear throughput scaling and bounded latency growth. - -#### Comparison: NVIDIA L40S (FP8) vs Inferentia2 (BF16) - - -| Metric | NVIDIA L40S (g6e.2xlarge, FP8) | Inferentia2 (inf2.48xlarge, BF16) | -| ----------------------------- | ------------------------------ | ------------------------------------- | -| **TTFT (single-user)** | 265-284 ms | 609-632 ms | -| **ITL** | 10.3-10.4 ms | 39.3-39.7 ms | -| **Single-user tok/s** | 80.7-92.0 | 24-25 | -| **Peak aggregate tok/s** | 482.2 (2x L40S, 16 concurrent) | 278 (1x inf2.48xlarge, 32 concurrent) | -| **Instance cost** | $1.12/hr per g6e.2xlarge | $6.94/hr per inf2.48xlarge | -| **Cost per 1M output tokens** | ~$0.65 (at 16 concurrent) | ~$6.93 (at peak throughput) | -| **Model precision** | FP8 (8-bit quantized) | BF16 (16-bit) | -| **Tensor Parallelism** | 1 (single GPU) | 16 (16 NeuronCores) | - - -> **Note:** The comparison is between different precision formats (FP8 vs BF16) and instance types. FP8 quantization on L40S provides a substantial performance advantage at significantly lower cost. The Inferentia2 benchmarks were run on a standalone KServe route (not through llm-d), so the gateway overhead is not included in the Inferentia numbers. - -#### Per-Accelerator Performance Summary (Capacity Planning) - -For multi-developer capacity planning, the key metric is **peak aggregate throughput per accelerator unit** at realistic concurrency levels. - - -| Metric | L40S (1x GPU, g6e.2xlarge) | inf2.24xlarge (8 cores, TP=8) | inf2.48xlarge (16 cores, TP=16) | -| ------------------------------- | -------------------------------- | ----------------------------- | ------------------------------- | -| **Single-user TTFT** | 265-284 ms | 495-523 ms | 609-632 ms | -| **Single-user ITL** | 10.3-10.4 ms | 40.9-41.0 ms | 39.3-39.7 ms | -| **Single-user output tok/s** | 80.7-92.0 | 22.7-24.2 | 23.2-25.1 | -| **Peak aggregate output tok/s** | ~241 (est. per GPU at 8 conc.) | 154.2 (8.5 concurrent) | 278.1 (32 concurrent) | -| **Instance cost (on-demand)** | $1.12/hr | $6.49/hr | $12.98/hr | -| **Cost per 1M output tokens** | ~$1.29 (est. per GPU at 8 conc.) | ~$11.69 (at peak) | ~$12.97 (at peak) | -| **Quantization** | FP8 (8-bit) | BF16 (16-bit) | BF16 (16-bit) | -| **Tensor Parallelism** | 1 | 8 | 16 | -| **Prefix caching** | Enabled | Disabled (MoE) | Disabled (MoE) | -| **Max model length** | 32,768 tokens | 8,192 tokens | 16,384 tokens | - - -> **Per-GPU throughput derivation:** The NVIDIA benchmark used 2x L40S GPUs via llm-d EPP. At 16 concurrent requests, aggregate throughput was 482.2 tok/s across 2 GPUs. Per-GPU maximum is estimated at ~241 tok/s. At 8 concurrent (within the near-linear scaling range), aggregate was 294.8 tok/s, giving ~147 tok/s per GPU. - -**Developer capacity estimates** (assuming average code assistant interaction = 512 prompt + 256 output tokens): - - -| Accelerator | Single-User tok/s | Concurrent Developers (interactive) | Concurrent Developers (batch) | -| ---------------- | ----------------- | ----------------------------------- | ----------------------------- | -| 1x L40S GPU | 87.7 | ~8-10 (responsive, <500ms TTFT) | ~16+ (bounded latency) | -| 1x inf2.24xlarge | 24.1 | ~4-5 (responsive, <1s TTFT) | ~8 (at peak throughput) | -| 1x inf2.48xlarge | 25.1 | ~4-5 (responsive, <1s TTFT) | ~16-32 (at peak throughput) | - - -> The L40S provides **3.6x better single-user throughput** and **~10x better cost-per-token** than Inferentia2 for this MoE model with FP8 quantization. Inferentia2's strength is with dense models and scenarios where FP8 quantization is not available, or where BF16 precision is required. - ---- - -### GuideLLM Sweep Benchmark Results (April 7, 2026) - -**Date:** April 7, 2026 -**Tool:** GuideLLM 0.5.4, sweep mode (10 strategies per test: synchronous, throughput, 8x constant at increasing rates) -**Duration:** 60 seconds per strategy (90s for 2048/1024 tests) -**Cluster:** ROSA HCP `rosa-pca`, OpenShift 4.21.7, OVN-Kubernetes - -#### NVIDIA L40S (FP8) — 2x g6e.2xlarge via llm-d EPP - -**Model:** `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` (FP8, TP=1, max-model-len=32768) -**Path:** GuideLLM pod → HTTPS → llm-d workload service → 2 vLLM pods - - -| Test | Prompt/Out | TTFT (ms) Mdn | TTFT (ms) p95 | ITL (ms) Mdn | ITL (ms) p95 | Req Latency (s) Mdn | Sync tok/s (out) | Peak Throughput tok/s (out) | -| ---- | ---------- | ------------- | ------------- | ------------ | ------------ | ------------------- | ---------------- | --------------------------- | -| 1 | 128/128 | 37.2 | 38.2 | 10.2 | 10.3 | 1.3 | 95.2 | 6,317 (throughput) | -| 2 | 128/256 | 36.3 | 37.6 | 10.2 | 10.3 | 2.7 | 95.2 | 6,317 (throughput) | -| 3 | 256/256 | 44.5 | 45.5 | 10.4 | 10.5 | 2.7 | 95.1 | 4,991 (throughput) | -| 4 | 512/512 | 62.1 | 64.5 | 10.4 | 10.4 | 5.4 | 95.4 | 3,380 (throughput) | -| 5 | 1024/512 | 93.7 | 121.0 | 32.6 | 45.1 | 16.8 | 29.5 | 2,297 (throughput) | -| 6 | 2048/1024 | 119.8 | 121.3 | 10.7 | 10.7 | 11.0 | 92.9 | 1,390 (throughput) | - - -**NVIDIA L40S key observations:** - -- **TTFT scales linearly** with prompt length: 37ms at 128 tokens → 120ms at 2048 tokens -- **ITL is rock-solid at ~10.3ms** for most configurations, giving ~97 tok/s decode rate per GPU -- **Throughput mode (max concurrency)** achieves 3,380–6,317 aggregate output tok/s across 2 GPUs -- **Realistic constant-rate performance (8–16 concurrent):** ~1,200–1,750 output tok/s aggregate - -#### Inferentia2 (BF16) — 1x inf2.24xlarge Standalone - -**Model:** `Qwen/Qwen3-Coder-30B-A3B-Instruct` (BF16, TP=8, 8 NeuronCores, max-model-len=8192) -**Path:** GuideLLM pod → HTTP → KServe InferenceService → vLLM-Neuron pod - - -| Test | Prompt/Out | TTFT (ms) Mdn | TTFT (ms) p95 | ITL (ms) Mdn | ITL (ms) p95 | Req Latency (s) Mdn | Sync tok/s (out) | Peak Throughput tok/s (out) | -| ---- | ---------- | ----------------------------- | ------------- | ------------ | ------------ | ------------------- | ---------------- | --------------------------- | -| 1 | 128/128 | 417 | 692 | 55.5 | 55.6 | 7.5 | 17.3 | 137 (throughput) | -| 2 | 128/256 | 416 | 419 | 55.8 | 55.8 | 14.6 | 17.6 | 137 (throughput) | -| 3 | 256/256 | 433 | 437 | 56.0 | 56.1 | 14.7 | 17.5 | 137 (throughput) | -| 4 | 512/512 | 480 | 484 | 56.9 | 56.9 | 29.5 | 17.4 | 137 (throughput) | -| 5 | 1024/512 | 586 | 590 | 59.8 | 59.9 | 31.2 | 16.6 | 127 (throughput) | -| 6 | 2048/1024 | *N/A — exceeds max_model_len* | — | — | — | — | — | — | - - -**Inferentia2 key observations:** - -- **TTFT ranges from 417ms to 586ms** — 5–10x slower than NVIDIA L40S due to BF16 precision and Neuron prefill overhead -- **ITL is stable at ~56-60ms** (~17 tok/s decode rate), compared to ~10.3ms on L40S -- **Peak throughput is ~137 output tok/s** at max concurrency, vs 3,380+ on 2x L40S -- **max_model_len was 4,096 at time of this test** (since increased to 16,384); Test 6 could not run with the old limit -- **Constant-rate (8–14 concurrent):** ~80–124 output tok/s - -#### Heterogeneous llm-d Routing — Native Same-Namespace (April 8, 2026) (128/128) - -**Configuration:** llm-d InferencePool with 3 endpoints in `llm-d-multi-gpu`: 2x NVIDIA L40S pods (FP8, HTTPS) + 1x Inferentia2 pod (BF16, HTTPS, native TLS) -**Architecture:** All pods in same namespace with matching labels. EPP sees real vLLM metrics from all 3 endpoints. Same `--served-model-name` across all backends. - -**Routing verification (10 concurrent requests):** - -- Inferentia pod: 4 requests served (`request_success_total = 4`) -- NVIDIA pods: 6 requests served (split across 2 pods) -- All responses return model name `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` -- EPP discovery via InferencePool label matching — all pods visible in same namespace - -**Architecture (verified):** - -``` -Client → llm-d EPP → ┬─ NVIDIA Pod 1 (g6e, FP8, HTTPS:8000) ← llm-d-multi-gpu - ├─ NVIDIA Pod 2 (g6e, FP8, HTTPS:8000) ← llm-d-multi-gpu - └─ Inferentia Pod (inf2, BF16, HTTPS:8000) ← llm-d-multi-gpu -``` - -**Requirements for native heterogeneous llm-d routing:** - -- All pods must be in the same namespace (InferencePool selector is namespace-scoped) -- All pods must serve HTTPS on the same port (8000) using consistent TLS certs -- All pods must carry matching InferencePool selector labels -- All pods must use the same `--served-model-name` -- Inferentia pods need separate EBS PVC for NEFF cache + model weights (≥200Gi) -- Inferentia pods need `nodeSelector` + `schedulerName: neuron-scheduler` + `securityContext` capabilities - -#### Updated Performance Comparison Summary - - -| Metric | 2x L40S (NVIDIA FP8) | 1x inf2.24xlarge (Inferentia BF16) | Hetero (2 NVIDIA + 1 Inf) | -| --------------------------------- | -------------------- | ---------------------------------- | ------------------------- | -| **TTFT 512/512 (ms)** | 62 | 480 | 64 | -| **ITL (ms)** | 10.4 | 56.9 | 10.4 | -| **Single-user decode tok/s** | 96 | 17 | 96 | -| **Peak throughput tok/s** | 3,380 | 137 | 3,392 | -| **8-concurrent tok/s** | 1,196 | ~80 | 1,069 | -| **Instance cost/hr** | $2.24 (2x) | $6.49 | $8.73 (combined) | -| **Cost per 1M output tok (peak)** | $0.18 | $13.16 | $0.71 | -| **max_model_len** | 32,768 | 8,192 | Mixed | - - -#### Heterogeneous llm-d Routing — GuideLLM Sweep (April 8, 2026) - -**Configuration:** `llm-d-multi-gpu` namespace, 3 endpoints: 2x NVIDIA L40S (FP8, HTTPS) + 1x Inferentia2 (BF16, HTTPS, native TLS). All pods in same namespace with matching InferencePool labels. -**GuideLLM version:** 0.5.4, synthetic data: 128 prompt tokens, 128 output tokens. - - -| # | Strategy | Rate (req/s) | Requests | Errors | Output tok/s | TTFT (ms) | ITL (ms) | Latency (s) | Concurrency | -| --- | ----------- | ------------ | -------- | ------ | ------------ | --------- | -------- | ----------- | ----------- | -| 1 | synchronous | - | 3 | 0 | 5.1 | 17,374 | 95.8 | 29.5 | 1.0 | -| 2 | throughput | - | 1,565 | 0 | 3,342 | 1,940 | 86.2 | 12.9 | 336.3 | -| 3 | constant | 3.3 | 166 | 0 | 354 | 114 | 55.8 | 7.2 | 19.9 | -| 4 | constant | 6.5 | 316 | 0 | 675 | 138 | 69.7 | 9.0 | 47.2 | -| 5 | constant | 9.8 | 497 | 0 | 1,062 | 316 | 73.3 | 9.6 | 79.7 | -| 6 | constant | 13.0 | 725 | 0 | 1,547 | 544 | 34.3 | 4.9 | 59.2 | -| 7 | constant | 16.3 | 894 | 0 | 1,908 | 708 | 36.1 | 5.3 | 78.8 | -| 8 | constant | 19.6 | 1,049 | 0 | 2,239 | 851 | 39.4 | 5.9 | 102.3 | -| 9 | constant | 22.8 | 1,192 | 0 | 2,545 | 997 | 42.2 | 6.4 | 126.2 | -| 10 | constant | 26.1 | 1,374 | 0 | 2,934 | 747 | 45.7 | 6.6 | 149.9 | - - -**Total requests across all strategies:** 7,781 — **0 errors** - -**Traffic distribution (from pod metrics):** - -- Inferentia pod: 1,125 requests served (~14.5%) -- NVIDIA pods: 6,656 requests served (~85.5%, split across 2 pods) - -**Key observations (native same-namespace approach):** - -1. **Zero errors across all 10 strategies** (7,781 total requests) — confirms production-grade reliability -2. **Peak throughput: 3,342 output tok/s** — comparable to the NVIDIA-only baseline (3,380), confirming that Inferentia adds capacity without degrading NVIDIA pod throughput -3. **The synchronous TTFT is high (17.4s)** because the EPP routed to the Inferentia pod, which has slower time-to-first-token due to BF16 precision and Neuron prefill overhead -4. **At moderate concurrency (3.3-13 req/s), ITL ranges 34-74ms** — the EPP correctly balances load, with the Inferentia pod absorbing overflow traffic -5. **EPP scoring works correctly** — real vLLM metrics from all pods enable KV-cache and queue-depth-aware routing decisions -6. **Direct pod-to-pod communication** via the InferencePool service — no intermediary overhead - -> **Conclusion:** The same-namespace native TLS deployment is the **only recommended method** for heterogeneous GPU model deployments with llm-d. The EPP's namespace-scoped discovery and metrics-based scoring require direct access to all backend vLLM pods. The Inferentia pod serves as effective overflow capacity (~14.5% of traffic), while NVIDIA pods handle the bulk of requests with their superior throughput. - -#### Heterogeneous llm-d Routing — Equal Split GuideLLM Sweep (April 8, 2026) - -**Configuration:** `llm-d-multi-gpu` namespace, 2 endpoints: 1x NVIDIA L40S (FP8, HTTPS) + 1x Inferentia2 (BF16, HTTPS, max-model-len=8192). All pods in same namespace with matching InferencePool labels. -**GuideLLM version:** 0.5.4, synthetic data: 128 prompt tokens, 128 output tokens. - - -| # | Strategy | Rate (req/s) | Requests | Errors | Output tok/s | TTFT (ms) | ITL (ms) | Latency (s) | Concurrency | -| --- | ----------- | ------------ | -------- | ------ | ------------ | --------- | -------- | ----------- | ----------- | -| 1 | synchronous | - | 8 | 0 | 17 | 504 | 55.5 | 7.6 | 1.0 | -| 2 | throughput | - | 1,421 | 0 | 3,052 | 1,688 | 75.0 | 11.2 | 265.8 | -| 3 | constant | 2.8 | 171 | 0 | 367 | 179 | 35.1 | 4.6 | 13.2 | -| 4 | constant | 5.1 | 308 | 0 | 659 | 2,439 | 38.7 | 7.4 | 37.8 | -| 5 | constant | 7.5 | 449 | 0 | 959 | 2,198 | 38.7 | 7.1 | 53.2 | -| 6 | constant | 9.9 | 595 | 0 | 1,277 | 1,699 | 42.4 | 7.1 | 70.3 | -| 7 | constant | 12.0 | 721 | 0 | 1,540 | 1,693 | 49.8 | 8.0 | 96.4 | -| 8 | constant | 13.5 | 809 | 0 | 1,729 | 1,589 | 58.0 | 9.0 | 120.7 | -| 9 | constant | 14.4 | 864 | 0 | 1,848 | 1,619 | 64.9 | 9.9 | 142.0 | -| 10 | constant | 15.4 | 926 | 0 | 1,980 | 1,565 | 75.9 | 11.2 | 173.0 | - - -**Total requests across all strategies:** 6,272 — **0 errors** - -**Traffic distribution (from pod metrics):** - -- NVIDIA pod: 5,711 requests served (~91%) -- Inferentia pod: 566 requests served (~9%) - -**Key observations (1:1 node ratio, equal split configuration):** - -1. **Zero errors across all 10 strategies** — consistent with the previous 2:1 (NVIDIA:Inferentia) test -2. **Peak throughput: 3,052 output tok/s** — lower than the 2-NVIDIA baseline (3,342) because only 1 NVIDIA pod is available. The single NVIDIA L40S is doing ~91% of the work. -3. **Synchronous TTFT: 504ms** — dominated by the Inferentia pod's prefill latency when the EPP routes there -4. **EPP heavily favors NVIDIA** despite 1:1 node ratio — the queue-depth and latency-based scoring correctly routes ~91% of traffic to the faster pod -5. **Inferentia serves as overflow capacity** absorbing ~9% of requests. This is less than the 14.5% in the 2:1 test because with only 1 NVIDIA pod, it saturates faster and the Inferentia pod's slower response time still limits its share. -6. **At moderate concurrency (2.8-9.9 req/s), ITL is 35-42ms** — acceptable for code assistant use cases - -> **Conclusion:** Even with a 1:1 NVIDIA:Inferentia ratio, the EPP correctly prioritizes the faster NVIDIA pod. Inferentia provides meaningful overflow capacity, absorbing 566 requests that would otherwise have queued behind the NVIDIA pod. The total throughput (3,052 tok/s) exceeds what a single NVIDIA pod alone would deliver (~1,690 tok/s), confirming the value of heterogeneous capacity. - -#### Inferentia2 Standalone — GuideLLM Sweep via llm-d (April 8, 2026) - -**Configuration:** `llm-d-multi-gpu` namespace, 1 endpoint: 1x Inferentia2 (BF16, HTTPS, max-model-len=8192). NVIDIA machine pool scaled to 0. All requests routed to the single Inferentia pod via llm-d EPP. -**GuideLLM version:** 0.5.4, synthetic data: 128 prompt tokens, 128 output tokens. - - -| # | Strategy | Rate (req/s) | Requests | Errors | Output tok/s | TTFT (ms) | ITL (ms) | Latency (s) | Concurrency | -| --- | ----------- | ------------ | -------- | ------ | ------------ | --------- | -------- | ----------- | ----------- | -| 1 | synchronous | - | 9 | 0 | 17.2 | 418 | 55.5 | 7.5 | 1.0 | -| 2 | throughput | - | 64 | 1 | 151.0 | 24,028 | 80.1 | 34.2 | 36.5 | -| 3 | constant | 0.2 | 14 | 0 | 30.1 | 452 | 58.9 | 7.9 | 1.8 | -| 4 | constant | 0.3 | 19 | 0 | 42.3 | 453 | 65.5 | 8.8 | 2.8 | -| 5 | constant | 0.4 | 25 | 0 | 54.7 | 456 | 68.8 | 9.2 | 3.8 | -| 6 | constant | 0.5 | 31 | 0 | 67.1 | 453 | 72.0 | 9.6 | 5.0 | -| 7 | constant | 0.6 | 36 | 0 | 78.3 | 448 | 78.5 | 10.4 | 6.2 | -| 8 | constant | 0.7 | 41 | 0 | 89.8 | 445 | 83.6 | 11.1 | 7.6 | -| 9 | constant | 0.8 | 47 | 0 | 100.1 | 455 | 91.6 | 12.1 | 9.5 | -| 10 | constant | 0.9 | 52 | 0 | 110.1 | 453 | 98.5 | 13.0 | 11.2 | - - -**Total requests:** 338 — **1 error** (timeout at max throughput) - -**Key observations (Inferentia-only via llm-d):** - -1. **Single-user (synchronous): 17.2 output tok/s**, 418ms TTFT, 55.5ms ITL — consistent with previous standalone Inferentia measurements -2. **Peak throughput: 151 output tok/s** at max concurrency (36.5 concurrent requests) — but with 24s TTFT and 1 timeout error -3. **Sustainable throughput: ~110 tok/s** at 0.9 req/s with 13s latency and 0 errors -4. **Sweet spot: 0.4-0.6 req/s** — ITL stays under 73ms and latency under 10s, suitable for interactive coding -5. **TTFT remains stable at ~450ms** across constant-rate strategies, confirming the Neuron prefill cost is constant regardless of queue depth -6. **Compared to NVIDIA L40S**: Inferentia delivers ~6.5% of NVIDIA's peak throughput (151 vs 3,052 tok/s) and ~5.4x higher ITL (55ms vs 10ms), confirming its role as overflow capacity rather than primary fleet - -> **Conclusion:** A single Inferentia2 `inf2.24xlarge` running Qwen3-Coder-30B (BF16, TP=8) provides a serviceable code assistant experience for 1-3 concurrent users at interactive latency. Beyond that, requests queue rapidly. For primary fleet use, NVIDIA L40S is 20x more cost-efficient per output token. Inferentia's value is as supplemental capacity behind llm-d when NVIDIA pods are saturated. - -#### Inferentia Context Window — Maximum Supported Length - -**Maximum `max_model_len` for Qwen3-Coder-30B on `inf2.24xlarge`: 8,192 tokens** (verified April 8, 2026). - -The context length on Inferentia2 is constrained by a combination of HBM capacity and Neuron NEFF activation buffers. Static KV cache math (48 layers, 4 KV heads, 128 head_dim, BF16 → ~96 KiB/token) suggests ~40k tokens would fit in 128 GB HBM. However, the Neuron compiler's NEFF artifacts include intermediate activation buffers and weight sharding overhead that are **not captured by the KV math alone**. The practical limit is **jointly set by three configuration knobs** that must be aligned: - - -| Knob | Deployed Value | Effect | -| ------------------------------------------------------- | ----------------------- | ----------------------------------------------------------------- | -| `--max-model-len` (vLLM) | **8,192** | Hard cap on accepted sequence length | -| `seq_len` in `--additional-config` | **8,192** | Neuron compiler graph dimension — determines NEFF artifact shapes | -| `context_encoding_buckets` / `token_generation_buckets` | Max bucket `[16, 8192]` | Largest compiled prefill/decode shape | - - -All three must increase together. Changing any one without the others causes either a runtime error or silent truncation. - -**Validated context lengths:** - - -| Instance | TP | NeuronCores | HBM | `max_model_len` | Status | -| --------------- | --- | ----------- | ------ | --------------- | ----------------------------------------------------------------- | -| `inf2.24xlarge` | 8 | 8 of 12 | 128 GB | **8,192** | **Deployed** (April 8, 2026) | -| `inf2.24xlarge` | 8 | 8 of 12 | 128 GB | 16,384 | **FAILED** — `Allocation Failure` during NEFF load (HBM exceeded) | -| `inf2.48xlarge` | 16 | 16 of 24 | 256 GB | **16,384** | Validated in prior testing | - - -**Why 16,384 fails on `inf2.24xlarge`:** Tested on April 8, 2026. NEFF compilation succeeded, but loading the compiled model onto NeuronCores failed with `RuntimeError: Could not load the model status=4 message=Allocation Failure`. The 16,384 bucket NEFFs with batch=16 require more HBM than available on 8 cores (128 GB). The static KV math underestimates total HBM because NEFF artifacts include activation buffers and weight sharding overhead. For 16k+ context, use `inf2.48xlarge` with TP=16 (256 GB HBM). - -**Why TP=12 doesn't work:** `hidden_size=2048` is not evenly divisible by 12 (2048/12 = 170.67). Valid TP values for this model: 1, 2, 4, 8, 16. - -**Current Inferentia deployment configuration:** - -```bash ---max-model-len 8192 \ ---num-gpu-blocks-override 32 \ ---additional-config '{"tp_degree":8,"moe_tp_degree":1,"moe_ep_degree":8,"batch_size":16,"seq_len":8192,"context_encoding_buckets":[[16,128],[16,512],[16,1024],[16,2048],[16,4096],[16,8192]],"token_generation_buckets":[[16,128],[16,512],[16,1024],[16,2048],[16,4096],[16,8192]],"on_device_generation":{"do_sample":true},"enable_fused_rmsnorm_quantization":true,"enable_fused_experts_quantization":true,"enable_nki_quantized_kernels":true}' -``` - -This requires a **full NEFF recompilation** (30-45 min for new bucket sizes, cached on PVC for subsequent restarts). Ensure ≥200Gi memory limit during compilation. - ---- - -### Phase 4: Trainium trn1.32xlarge with Neuron DRA Driver - -**Objective:** Deploy Qwen3-Coder-30B on AWS Trainium (`trn1.32xlarge`) using the **Neuron DRA (Dynamic Resource Allocation) driver** — the Kubernetes-native device allocation method that replaces the legacy device-plugin + neuron-scheduler stack. Validate llm-d heterogeneous routing between NVIDIA L40S (FP8) and Trainium (BF16) backends. - -**Date:** April 9, 2026 - -**Configuration:** - - -| Property | Value | -| ----------------- | ----------------------------------------------------------- | -| Instance | `trn1.32xlarge` ($21.50/hr) | -| AZ | `us-east-2c` (new private subnet `10.0.12.0/24` created) | -| NeuronCores | 16 of 32 (via DRA `ResourceClaimTemplate`, `ExactCount=16`) | -| HBM | 512 GB total (256 GB used by 16 cores) | -| Model | `Qwen/Qwen3-Coder-30B-A3B-Instruct` (BF16, 57 GB) | -| TP | 16 (tensor-parallel across 16 NeuronCores) | -| Max context | 16,384 tokens | -| Batch size | 16 | -| vLLM | 0.13.0 (Neuron build, SDK 2.28.0) | -| Device allocation | **Neuron DRA Driver v1.0.0** (Helm chart v1.5.0) | -| Neuron Operator | v1.1.2 | -| Taint | `aws.amazon.com/neuroncore=present:NoSchedule` | -| MoE routing | `moe_ep_degree=16`, `moe_tp_degree=1` | - - -**Key findings:** - -1. **DRA driver works on Trainium** — the Neuron DRA driver (`public.ecr.aws/neuron/neuron-dra-driver:1.0.0`) successfully detected all 16 NeuronDevices on `trn1.32xlarge`, advertised them to the Kubernetes API, and allocated 16 NeuronCores to the model pod via `ResourceClaimTemplate`. This is the first validated DRA deployment in this project. -2. **Subnet creation required** — `trn1.32xlarge` is only available in `us-east-2c`, but the ROSA cluster's subnets were in `us-east-2a` and `us-east-2b`. A new private subnet (`10.0.12.0/24`) was created in the existing VPC and associated with the private route table. -3. **KMM SELinux fix** — the Neuron kernel module (`neuron.ko`) failed to load via `insmod` due to SELinux context (`tmp_t`). The fix was to copy the `.ko` file with `modules_object_t` SELinux context before loading. This is a known issue on RHCOS with SELinux Enforcing. -4. **DRA DeviceClass** — the Helm chart creates a `DeviceClass neuron.aws.com` that the `ResourceClaimTemplate` references. Pods specify `resourceClaims` in their spec instead of the legacy `resources.limits["aws.amazon.com/neuroncore"]` approach. -5. **16k context window** — the larger HBM (256 GB for 16 cores vs 128 GB for 8 cores on inf2.24xlarge) allows a 16k context window compared to the 4k limit on Inferentia2. -6. **llm-d routing confirmed** — the EPP discovered the Trainium pod via label matching and started scraping its `/metrics` endpoint. Both NVIDIA and Trainium backends are routable through the same `InferencePool`. -7. **NEFF compilation** — first-time compilation took ~25 minutes on trn1.32xlarge (14 buckets: 7 context_encoding + 7 token_generation). Compiled artifacts are cached on EBS PVC for subsequent restarts. - -#### GuideLLM Sweep Results — Trainium trn1.32xlarge (April 9, 2026) - -**Test setup:** GuideLLM v0.5.4, `prompt_tokens=128, output_tokens=128`, sweep profile (10 strategies), max 120s/100 requests per strategy. Only the Trainium backend active (NVIDIA scaled to 0). - -**Request Latency:** - - -| Strategy | Request Latency (Mdn) | TTFT Mdn (ms) | TTFT p95 (ms) | ITL Mdn (ms) | TPOT Mdn (ms) | TPOT p95 (ms) | -| --------------- | --------------------- | ------------- | ------------- | ------------ | ------------- | ------------- | -| Synchronous | 6.8s | 1697.1 | 2406.6 | 40.5 | 53.4 | 59.1 | -| Throughput | 32.2s | 26991.0 | 57360.2 | 133.7 | 251.4 | 502.4 | -| Constant (low) | 8.6s | 1728.5 | 1744.5 | 53.8 | 66.9 | 67.0 | -| Constant (mid) | 11.9s | 1721.7 | 1736.8 | 80.4 | 93.2 | 93.4 | -| Constant (high) | 32.2s | 1738.6 | 1772.2 | 240.1 | 251.8 | 252.1 | - - -**Server Throughput:** - - -| Strategy | Req/s (Mdn) | Concurrency (Mdn) | Input Tok/s (Mdn) | Output Tok/s (Mean) | Total Tok/s (Mean) | -| --------------- | ----------- | ----------------- | ----------------- | ------------------- | ------------------ | -| Synchronous | 0.1 | 1 | 19.9 | 19.0 | 39.2 | -| Throughput | 0.1 | 84 | 80.6 | 65.4 | 273.9 | -| Constant (low) | 0.2 | 2 | 25.3 | 23.8 | 50.0 | -| Constant (mid) | 0.3 | 4 | 43.5 | 39.9 | 84.2 | -| Constant (high) | 0.5 | 16 | 67.6 | 56.3 | 124.9 | - - -**Key observations:** - -- **TTFT**: ~1.7s at low concurrency, consistent across load levels (MoE compilation creates fixed overhead) -- **ITL**: 40.5ms synchronous → 240ms at high concurrency (graceful degradation) -- **Peak throughput**: ~69 input tok/s, ~65 output tok/s at max concurrency -- **16 NeuronCores (TP=16)** enables concurrent request handling that scales to 16 simultaneous requests before saturation - -**Trainium DRA deployment YAML (key sections):** - -```yaml -spec: - template: - spec: - resourceClaims: - - name: neurons - resourceClaimTemplateName: trainium-16-cores - containers: - - name: kserve-container - resources: - claims: - - name: neurons - requests: - cpu: "16" - memory: 200Gi -``` - -```yaml -apiVersion: resource.k8s.io/v1 -kind: ResourceClaimTemplate -metadata: - name: trainium-16-cores -spec: - spec: - devices: - requests: - - name: neurons - exactly: - deviceClassName: neuron.aws.com - allocationMode: ExactCount - count: 16 - selectors: - - cel: - expression: "device.attributes['neuron.aws.com'].instanceType == 'trn1.32xlarge'" -``` - -#### Phase 4b: 2-Replica Scaling Investigation (April 9, 2026) - -**Objective:** Determine whether two replicas of Qwen3-Coder-30B (TP=16 each) can run simultaneously on a single `trn1.32xlarge`. - -**Capacity analysis:** - - -| Resource | Total on trn1.32xlarge | Per Replica (TP=16) | 2 Replicas | Fits? | -| ------------------- | ----------------------------- | -------------------------- | ---------- | ------------------- | -| NeuronDevices (DRA) | 16 | 8 (= 16 NeuronCores) | 16 | Yes (resource-wise) | -| CPU | 128 vCPU (127.5 allocatable) | 16 request | ~32.3 | Yes | -| Memory | 512 GiB (494 GiB allocatable) | 200 GiB request | ~407 GiB | Yes | -| HBM | 512 GB (16 devices × 32 GB) | 256 GB (8 devices × 32 GB) | 512 GB | Yes | - - -**What was attempted:** - -1. Created a new `ResourceClaimTemplate` (`trainium-8-devices`) requesting 8 DRA devices per pod -2. Deployed with `replicas: 2` — DRA correctly allocated 8 NeuronDevices to each pod with no overlap -3. Both pods scheduled on the trn1 node, sharing the RWO PVC (allowed since same node) - -**Result: NOT FEASIBLE with TP=16** - -Two critical failures blocked 2-replica operation: - -1. **NEFF execution failure (`Failed to execute model ... with status 1`)** — The compiled NEFF artifacts assume a specific NeuronLink interconnect topology. With TP=16 across 8 NeuronDevices (16 NeuronCores), the NeuronLink collective communication patterns (all-reduce, all-gather) differ from the 16-device topology. NEFFs cached from the single-replica 16-device deployment cannot execute on an 8-device partition, and NEFFs compiled for one 8-device group may not match another 8-device group's physical layout. -2. **Disk pressure from simultaneous NEFF compilation** — Both pods compiling NEFFs simultaneously wrote large intermediate HLO/NEFF artifacts to the node's ephemeral storage (`/tmp/nxd_model/`), triggering `DiskPressure: True` and kubelet eviction (exit code 137). The Neuron compiler's temporary file usage (30-50 GB per compilation) exceeds safe thresholds when two compilations run in parallel on the same node. - -**Recommendation for multi-replica Trainium:** - -- **TP=8 per replica** (4 NeuronDevices × 2 cores = 8 NeuronCores) could theoretically enable 2 replicas, but requires model recompilation and may not provide enough HBM for larger context windows. This is a future experiment. -- **Horizontal scaling across nodes** (2 × `trn1.32xlarge` with 1 replica each) is the reliable approach for scaling Trainium inference throughput. -- **Staggered startup** — if 2 replicas are attempted, deploy sequentially (start replica 1, wait for NEFF cache to populate, then start replica 2) to avoid disk pressure. - ---- - -## Planned Test Phases - -### Phase 5: Dual Qwen3-Coder-30B on inf2.48xlarge (planned) - -**Objective:** Validate running two independent Qwen3-Coder-30B BF16 instances on a single `inf2.48xlarge` node with core partitioning, benchmark aggregate throughput with GuideLLM, and compare cost-efficiency against 2x `inf2.24xlarge`. - -**Instance specs:** - - -| Attribute | inf2.48xlarge | -| ----------------- | ----------------------- | -| Inferentia2 chips | 12 | -| NeuronCores | 24 | -| HBM total | 384 GB (16 GB per core) | -| System memory | 768 GB | -| vCPUs | 192 | -| NeuronLink | v2 (192 GB/s per chip) | -| On-demand cost | $12.98/hr | - - -**Instance-level core partitioning:** - - -| Instance | Tensor Parallelism | `NEURON_RT_VISIBLE_CORES` | HBM allocated | Model weights (BF16) | KV cache headroom | -| -------- | ------------------ | ------------------------- | ------------- | -------------------- | ----------------- | -| **A** | 8 | `0-7` | 128 GB | ~60 GB | ~68 GB | -| **B** | 8 | `8-15` | 128 GB | ~60 GB | ~68 GB | -| *unused* | — | cores 16-23 | 128 GB | — | — | - - -Total: 16 of 24 cores used (8 wasted, 33% — same TP divisibility constraint as inf2.24xlarge). - -**Known risk:** In earlier testing, a second vLLM instance on separate cores (Llama-3.1-8B on cores 8-11 alongside Qwen on cores 0-7 on inf2.24xlarge) failed with `RuntimeError: The PyTorch Neuron Runtime could not be initialized`. This test will determine whether the issue was specific to the inf2.24xlarge or a general multi-instance Neuron runtime limitation. - -**Pre-requisites:** - -1. NEFF compilation cache PVC with warm cache from Phase 1/3 testing (avoids 15-20 min compilation per instance) -2. MutatingAdmissionWebhook for OVN annotation preservation deployed -3. `neuron-scheduler` extension running and healthy - -**Step 4a. Create inf2.48xlarge machine pool** - -```bash -rosa create machinepool \ - --cluster=rosa-private-ai \ - --name=inf2-48xl \ - --instance-type=inf2.48xlarge \ - --replicas=1 \ - --labels=node-role.kubernetes.io/inferentia=,accelerator=neuron \ - --taints="aws.amazon.com/neuroncore=present:NoSchedule" - -# Wait for node to be Ready -oc get nodes -l node.kubernetes.io/instance-type=inf2.48xlarge -w -``` - -**Step 4b. Create PVC and download model** - -Use the same PVC layout as Step 12 in the deployment guide but on the inf2.48xlarge node: - -```bash -# PVC for model weights + NEFF cache (shared between both instances) -oc apply -f - <<'EOF' -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: qwen3-coder-inf2-48xl-storage - namespace: llm-serving -spec: - accessModes: [ReadWriteOnce] - storageClassName: gp3-csi - resources: - requests: - storage: 200Gi -EOF - -# Download model (if not reusing existing PVC) -oc run model-downloader --rm -it --restart=Never \ - --image=python:3.11-slim \ - --overrides='{"spec":{"nodeSelector":{"node.kubernetes.io/instance-type":"inf2.48xlarge"},"tolerations":[{"key":"aws.amazon.com/neuron","operator":"Exists"}],"containers":[{"name":"model-downloader","image":"python:3.11-slim","command":["bash","-c","pip install huggingface_hub && python -c \"from huggingface_hub import snapshot_download; snapshot_download(repo_id='"'"'Qwen/Qwen3-Coder-30B-A3B-Instruct'"'"', local_dir='"'"'/models/Qwen3-Coder-30B-A3B-Instruct'"'"')\""],"env":[{"name":"HF_TOKEN","valueFrom":{"secretKeyRef":{"name":"hf-token","key":"token"}}}],"volumeMounts":[{"name":"storage","mountPath":"/models"}],"resources":{"requests":{"memory":"8Gi"},"limits":{"memory":"16Gi"}}}],"volumes":[{"name":"storage","persistentVolumeClaim":{"claimName":"qwen3-coder-inf2-48xl-storage"}}]}}' \ - -- bash -``` - -**Step 4c. Deploy two InferenceService instances** - -Both instances share the same PVC (using `seLinuxOptions.level: "s0:c0,c0"`) but use different NeuronCore ranges: - -```yaml -# Instance A: cores 0-7 -apiVersion: serving.kserve.io/v1beta1 -kind: InferenceService -metadata: - name: qwen3-coder-neuron-a - namespace: llm-serving - annotations: - serving.kserve.io/deploymentMode: RawDeployment - storage.kserve.io/readonly: "false" -spec: - predictor: - model: - runtime: qwen3-coder-neuron-runtime - storageUri: pvc://qwen3-coder-inf2-48xl-storage/Qwen3-Coder-30B-A3B-Instruct - resources: - requests: - cpu: "16" - memory: 128Gi - aws.amazon.com/neuroncore: "8" - limits: - cpu: "32" - # Neuron compilation peaks at ~120 GB RSS. 200Gi tested minimum. - memory: 200Gi - aws.amazon.com/neuroncore: "8" - nodeSelector: - node.kubernetes.io/instance-type: inf2.48xlarge - schedulerName: neuron-scheduler - securityContext: - seLinuxOptions: - level: "s0:c0,c0" - containers: - - name: kserve-container - env: - - name: NEURON_RT_VISIBLE_CORES - value: "0-7" - - name: NEURON_COMPILE_CACHE_URL - value: "/models/neuron-cache" - - name: VLLM_NEURON_FRAMEWORK - value: "neuronx-distributed-inference" - - name: HF_HOME - value: "/models/hf-cache" - - name: VLLM_ENGINE_ITERATION_TIMEOUT_S - value: "600" - - name: VLLM_RPC_TIMEOUT - value: "600000" - command: ["bash", "-c"] - args: - - | - while [ ! -d "/models/Qwen3-Coder-30B-A3B-Instruct" ]; do sleep 5; done - python -m vllm.entrypoints.openai.api_server \ - --model /models/Qwen3-Coder-30B-A3B-Instruct \ - --served-model-name qwen3-coder \ - --tensor-parallel-size 8 \ - --max-model-len 8192 \ - --block-size 8 \ - --max-num-seqs 16 \ - --no-enable-prefix-caching \ - --port 8000 ---- -# Instance B: cores 8-15 -apiVersion: serving.kserve.io/v1beta1 -kind: InferenceService -metadata: - name: qwen3-coder-neuron-b - namespace: llm-serving - annotations: - serving.kserve.io/deploymentMode: RawDeployment - storage.kserve.io/readonly: "false" -spec: - predictor: - model: - runtime: qwen3-coder-neuron-runtime - storageUri: pvc://qwen3-coder-inf2-48xl-storage/Qwen3-Coder-30B-A3B-Instruct - resources: - requests: - cpu: "16" - memory: 128Gi - aws.amazon.com/neuroncore: "8" - limits: - cpu: "32" - # Neuron compilation peaks at ~120 GB RSS. 200Gi tested minimum. - memory: 200Gi - aws.amazon.com/neuroncore: "8" - nodeSelector: - node.kubernetes.io/instance-type: inf2.48xlarge - schedulerName: neuron-scheduler - securityContext: - seLinuxOptions: - level: "s0:c0,c0" - containers: - - name: kserve-container - env: - - name: NEURON_RT_VISIBLE_CORES - value: "8-15" - - name: NEURON_COMPILE_CACHE_URL - value: "/models/neuron-cache" - - name: VLLM_NEURON_FRAMEWORK - value: "neuronx-distributed-inference" - - name: HF_HOME - value: "/models/hf-cache" - - name: VLLM_ENGINE_ITERATION_TIMEOUT_S - value: "600" - - name: VLLM_RPC_TIMEOUT - value: "600000" - command: ["bash", "-c"] - args: - - | - while [ ! -d "/models/Qwen3-Coder-30B-A3B-Instruct" ]; do sleep 5; done - python -m vllm.entrypoints.openai.api_server \ - --model /models/Qwen3-Coder-30B-A3B-Instruct \ - --served-model-name qwen3-coder \ - --tensor-parallel-size 8 \ - --max-model-len 8192 \ - --block-size 8 \ - --max-num-seqs 16 \ - --no-enable-prefix-caching \ - --port 8000 -``` - -**Step 4d. Verify both instances are serving** - -```bash -# Check both pods are Running -oc get pods -n llm-serving -l serving.kserve.io/inferenceservice -o wide - -# Verify core allocation (each pod should show different NEURON_ALLOCATED cores) -oc logs -n llm-serving deploy/qwen3-coder-neuron-a-predictor | grep -i "neuron\|core\|loaded" -oc logs -n llm-serving deploy/qwen3-coder-neuron-b-predictor | grep -i "neuron\|core\|loaded" - -# Quick smoke test on each instance -INST_A_SVC=$(oc get svc -n llm-serving -l serving.kserve.io/inferenceservice=qwen3-coder-neuron-a -o jsonpath='{.items[0].metadata.name}') -INST_B_SVC=$(oc get svc -n llm-serving -l serving.kserve.io/inferenceservice=qwen3-coder-neuron-b -o jsonpath='{.items[0].metadata.name}') - -oc run smoke-test --rm -it --restart=Never --image=curlimages/curl -- \ - curl -s http://$INST_A_SVC.llm-serving.svc:8000/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{"model":"qwen3-coder","messages":[{"role":"user","content":"Write a hello world in Python"}],"max_tokens":50}' - -oc run smoke-test-b --rm -it --restart=Never --image=curlimages/curl -- \ - curl -s http://$INST_B_SVC.llm-serving.svc:8000/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{"model":"qwen3-coder","messages":[{"role":"user","content":"Write a hello world in Rust"}],"max_tokens":50}' -``` - -**Step 4e. GuideLLM benchmark — per-instance and combined** - -Run the same 6-test GuideLLM sweep used for inf2.24xlarge (see `inferentia_vllm_test_results.md` Run 1) against each instance individually, then both concurrently: - -```bash -# Install GuideLLM -pip install guidellm==0.5.4 - -# Per-instance benchmarks (run sequentially) -for INST in a b; do - SVC=$(oc get svc -n llm-serving -l serving.kserve.io/inferenceservice=qwen3-coder-neuron-$INST -o jsonpath='{.items[0].metadata.name}') - for PROMPT OUT in "128 128" "128 256" "256 256" "512 512" "1024 512" "2048 1024"; do - P=$(echo $PROMPT | cut -d' ' -f1); O=$(echo $PROMPT | cut -d' ' -f2) - guidellm \ - --target "http://$SVC.llm-serving.svc:8000/v1" \ - --model qwen3-coder \ - --rate-type sweep \ - --data "prompt_tokens=$P,output_tokens=$O" \ - --max-seconds 120 \ - --backend-args '{"timeout": 180.0}' \ - --output-path "/results/inf2-48xl-inst-${INST}-${P}in-${O}out.json" - done -done - -# Combined load test: both instances behind a single K8s Service -# Create a unified headless Service spanning both InferenceServices -oc apply -f - <<'EOF' -apiVersion: v1 -kind: Service -metadata: - name: qwen3-coder-combined - namespace: llm-serving -spec: - clusterIP: None - ports: - - port: 8000 - targetPort: 8000 - selector: - serving.kserve.io/inferenceservice-group: qwen3-coder-neuron -EOF - -# Label both deployments for the combined service -oc label pods -n llm-serving -l serving.kserve.io/inferenceservice=qwen3-coder-neuron-a \ - serving.kserve.io/inferenceservice-group=qwen3-coder-neuron -oc label pods -n llm-serving -l serving.kserve.io/inferenceservice=qwen3-coder-neuron-b \ - serving.kserve.io/inferenceservice-group=qwen3-coder-neuron - -# Run combined sweep (512/512 reference config) -guidellm \ - --target "http://qwen3-coder-combined.llm-serving.svc:8000/v1" \ - --model qwen3-coder \ - --rate-type sweep \ - --data "prompt_tokens=512,output_tokens=512" \ - --max-seconds 120 \ - --backend-args '{"timeout": 180.0}' \ - --output-path "/results/inf2-48xl-combined-512in-512out.json" -``` - -**Expected results to capture:** - - -| Metric | Instance A (cores 0-7) | Instance B (cores 8-15) | Combined (A + B) | Comparison: 1x inf2.24xlarge | -| --------------------- | ---------------------- | ----------------------- | ---------------- | ---------------------------- | -| Single-user TTFT (ms) | | | N/A | 495-523 | -| Single-user ITL (ms) | | | N/A | 40.9-41.0 | -| Single-user tok/s | | | N/A | 22.7-24.2 | -| Peak aggregate tok/s | | | | 154.2 | -| Neuron runtime init | ✅ / ❌ | ✅ / ❌ | — | ✅ | - - -**Success criteria:** - -- Both instances start successfully (Neuron runtime initializes on separate core ranges) -- Per-instance throughput matches inf2.24xlarge baseline (~24 tok/s single-user, ~154 tok/s peak) -- Combined throughput is approximately 2× single-instance (~308 tok/s peak) -- No Neuron runtime conflicts or OOM during concurrent operation - -**Step 4f. Scale down after testing** - -```bash -rosa edit machinepool --cluster=rosa-private-ai --name=inf2-48xl --replicas=0 -oc get nodes -l node.kubernetes.io/instance-type=inf2.48xlarge # should return empty -``` - -**Do not delete the PVC** — NEFF cache artifacts persist for future testing. - -### Phase 6: NVIDIA g6e.2xlarge (L40S) Full GuideLLM Benchmark - -**Objective:** Run the complete GuideLLM sweep benchmark on 2x `g6e.2xlarge` (L40S, FP8) with llm-d EPP routing to produce the definitive performance baseline for Qwen3-Coder-30B on NVIDIA, matching the same test matrix used for Inferentia in `inferentia_vllm_test_results.md`. - -#### Node Sizing Analysis - -**Qwen3-Coder-30B-A3B memory requirements on L40S:** - - -| Component | FP8 (8-bit) | -| -------------------------- | ----------- | -| Model weights | ~30 GB | -| vLLM runtime overhead | ~2 GB | -| **Available for KV cache** | **~16 GB** | - - -**KV cache sizing** (4 KV heads × 64 head_dim × 48 layers × 2 bytes FP16 × 2 [K+V] = **~48 KB per token**): - - -| Metric | Value | -| ---------------------------------- | ------ | -| KV headroom on L40S (48 GB, FP8) | ~16 GB | -| Max tokens in KV cache | ~333K | -| Concurrent sequences @ 32K context | ~10 | -| Concurrent sequences @ 8K context | ~40 | - - -> **g6e.2xlarge is the right node size** for this model at FP8. The `.xlarge` variant (4 vCPUs, 32 GB RAM) is marginal for vLLM serving overhead. The `.2xlarge` (8 vCPUs, 64 GB RAM, 1x L40S 48 GB VRAM, $2.24/hr) is the smallest g6e size with adequate CPU/memory headroom alongside the GPU. Multi-GPU nodes (`g6e.12xlarge`, 4x L40S, $10.49/hr) cost 2.3× more than 2x `g6e.2xlarge` for the same GPU count and only make sense for TP>1 workloads. - -**Step 5a. Create g6e machine pool (if not already present from Phase 2)** - -```bash -rosa create machinepool \ - --cluster=rosa-private-ai \ - --name=gpu-l40s \ - --instance-type=g6e.2xlarge \ - --replicas=2 \ - --labels=node-role.kubernetes.io/gpu=,accelerator=nvidia \ - --taints=nvidia.com/gpu:NoSchedule -``` - -**Step 5b. Deploy LLMInferenceService (reuse Phase 2 deployment)** - -Use the same `LLMInferenceService` from Step 8 in the deployment guide with 2 replicas, `max-model-len=32768`, prefix caching enabled. Each replica node needs its own PVC (Step 7) or use a `ReadWriteMany` volume. - -**Step 5c. GuideLLM benchmark — full sweep** - -Run the same 6-test matrix used for Inferentia (see `inferentia_vllm_test_results.md` Run 1): - -```bash -pip install guidellm==0.5.4 - -GATEWAY_URL="https://$(oc get route -n llm-d-gpu -o jsonpath='{.items[0].spec.host}')" - -for PROMPT_OUT in "128:128" "128:256" "256:256" "512:512" "1024:512" "2048:1024"; do - P=$(echo $PROMPT_OUT | cut -d: -f1) - O=$(echo $PROMPT_OUT | cut -d: -f2) - TIMEOUT=120 - [ "$O" = "1024" ] && TIMEOUT=180 - guidellm \ - --target "${GATEWAY_URL}/v1" \ - --model qwen3-coder \ - --rate-type sweep \ - --data "prompt_tokens=$P,output_tokens=$O" \ - --max-seconds $TIMEOUT \ - --backend-args "{\"timeout\": $(( TIMEOUT + 60 )).0}" \ - --output-path "/results/g6e-l40s-2x-${P}in-${O}out.json" -done -``` - -**Expected results template (per test, 2x L40S via llm-d):** - - -| Test | Prompt/Output | TTFT (sync, ms) | ITL (sync, ms) | Single-user tok/s | Peak agg tok/s | Peak concurrency | -| ---- | ------------- | --------------- | -------------- | ----------------- | -------------- | ---------------- | -| 1 | 128 / 128 | | | | | | -| 2 | 128 / 256 | | | | | | -| 3 | 256 / 256 | | | | | | -| 4 | 512 / 512 | | | | | | -| 5 | 1024 / 512 | | | | | | -| 6 | 2048 / 1024 | | | | | | - - -**Step 5d. Scale down after testing** - -```bash -rosa edit machinepool --cluster=rosa-private-ai --name=gpu-l40s --replicas=0 -``` - -#### GuideLLM Sweep Results — Qwen3-Coder-30B-A3B-FP8 on NVIDIA L40S (April 20, 2026) - -**Test setup:** - - -| Parameter | Value | -| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| **Model** | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` (MoE: 30B total, 3B active) | -| **GPU** | NVIDIA L40S (48 GB VRAM) on `g6e.2xlarge` | -| **Quantization** | FP8 (W8A8) | -| **vLLM version** | 0.13.0+rhai11 (RHOAI) | -| **vLLM args** | `--max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --kv-cache-dtype fp8` | -| **Tensor Parallelism** | TP=1 (single GPU) | -| **Benchmark tool** | GuideLLM v0.6, `--scenario chat --profile sweep`, 10 strategies (1 synchronous, 1 throughput, 8 constant-rate), max 120s / 100 requests per strategy | -| **Data** | Synthetic chat: prompt ~512 tokens (stdev 128), output ~256 tokens (stdev 64) | -| **Replicas** | 1 | - - -**Server Throughput (all requests):** - - -| Strategy | Concurrency (Mdn) | Requests/sec | Input tok/s | Output tok/s | Total tok/s | -| ---------------- | ----------------- | ------------ | ----------- | ------------ | ----------- | -| synchronous | 1 | 0.4 | 199 | 93 | 288 | -| throughput (max) | 93 | 5.4 | 20,421 | 1,357 | 4,140 | -| constant-1 | 6 | 1.0 | 516 | 255 | 749 | -| constant-2 | 11 | 1.5 | 877 | 376 | 1,160 | -| constant-3 | 16 | 2.0 | 1,183 | 492 | 1,516 | -| constant-4 | 23 | 2.3 | 1,478 | 597 | 1,790 | -| constant-5 | 29 | 2.7 | 1,824 | 692 | 2,115 | -| constant-6 | 32 | 3.0 | 2,127 | 762 | 2,313 | -| constant-7 | 38 | 3.2 | 2,439 | 837 | 2,482 | -| constant-8 | 39 | 3.4 | 2,852 | 859 | 2,636 | - - -**Latency (completed requests):** - - -| Strategy | Request Latency (Mdn, s) | TTFT (Mdn, ms) | ITL (Mdn, ms) | TPOT (Mdn, ms) | -| ----------- | ------------------------ | -------------- | ------------- | -------------- | -| synchronous | 2.6 | 73.6 | 10.4 | 10.6 | -| throughput | 14.4 | 1,156.1 | 51.7 | 55.9 | -| constant-1 | 6.0 | 80.7 | 22.3 | 22.5 | -| constant-4 | 8.4 | 88.9 | 33.5 | 33.7 | -| constant-8 | 10.6 | 99.9 | 42.0 | 42.2 | - - -**Key performance metrics (single L40S):** - - -| Metric | Value | -| --------------------------------------------- | ------------------------------------------- | -| **Single-user output tok/s** | ~93 (synchronous, Mdn) | -| **Single-user TTFT** | 73.6 ms | -| **Single-user ITL** | 10.4 ms | -| **Peak aggregate output tok/s** | ~1,357 (throughput strategy, 93 concurrent) | -| **Peak aggregate total tok/s** | ~4,140 (throughput strategy) | -| **Sustainable output tok/s @ ~30 concurrent** | ~692-762 | -| **Latency at ~30 concurrent** | TTFT ~94ms, ITL ~37ms | - - -> **Analysis:** The L40S delivers excellent single-user latency (73ms TTFT, 10ms ITL) for interactive coding. Peak throughput reaches ~1,357 output tok/s at 93 concurrent requests, but at the cost of 1.2s TTFT and 52ms ITL. The sweet spot for a multi-developer code assistant is ~20-30 concurrent requests, yielding ~600-760 output tok/s with sub-100ms TTFT and sub-40ms ITL — comfortably supporting 10-15 active developers. - -#### GuideLLM Sweep Results — Qwen3-8B-FP8 on NVIDIA L40S (April 21, 2026) - -**Test setup:** - - -| Parameter | Value | -| ---------------------- | ----------------------------------------------------------------------------------------------------- | -| **Model** | `Qwen/Qwen3-8B-FP8` (dense, 8.2B params) | -| **GPU** | NVIDIA L40S (48 GB VRAM) on `g6e.2xlarge` | -| **Quantization** | FP8 (W8A8) | -| **vLLM version** | 0.13.0+rhai11 (RHOAI) | -| **vLLM args** | `--max-model-len 8192 --gpu-memory-utilization 0.90 --enable-prefix-caching` | -| **Tensor Parallelism** | TP=1 (single GPU) | -| **Data Parallelism** | DP=1 (DP=2 failed on vLLM 0.13 — engine core init crash; requires vLLM 0.15+) | -| **Benchmark tool** | GuideLLM v0.6, `--scenario chat --profile sweep`, 10 strategies, max 120s / 100 requests per strategy | -| **Data** | Synthetic chat: prompt ~512 tokens (stdev 128), output ~256 tokens (stdev 64) | -| **Deployment** | LLMInferenceService via KServe, behind llm-d EPP | - - -**Server Throughput (all requests):** - - -| Strategy | Concurrency (Mdn) | Requests/sec | Input tok/s | Output tok/s | Total tok/s | -| ---------------- | ----------------- | ------------ | ----------- | ------------ | ----------- | -| synchronous | 1 | 0.2 | 111 | 50 | 158 | -| throughput (max) | 98 | 6.2 | 13,490 | 1,571 | 4,793 | -| constant-1 | 4 | 0.9 | 492 | 241 | 709 | -| constant-2 | 8 | 1.5 | 922 | 394 | 1,213 | -| constant-3 | 12 | 2.2 | 1,295 | 553 | 1,704 | -| constant-4 | 17 | 2.6 | 1,653 | 675 | 2,024 | -| constant-5 | 22 | 3.3 | 2,072 | 841 | 2,570 | -| constant-6 | 27 | 3.7 | 2,436 | 938 | 2,850 | -| constant-7 | 35 | 4.0 | 2,815 | 1,039 | 3,083 | -| constant-8 | 37 | 4.2 | 3,310 | 1,085 | 3,330 | - - -**Latency (completed requests):** - - -| Strategy | Request Latency (Mdn, s) | TTFT (Mdn, ms) | ITL (Mdn, ms) | TPOT (Mdn, ms) | -| ----------- | ------------------------ | -------------- | ------------- | -------------- | -| synchronous | 5.0 | 60.9 | 19.8 | 20.0 | -| throughput | 12.4 | 1,798.8 | 40.6 | 48.0 | -| constant-1 | 4.9 | 64.9 | 18.2 | 18.4 | -| constant-4 | 5.7 | 73.1 | 22.0 | 22.2 | -| constant-8 | 7.2 | 80.1 | 28.8 | 29.0 | - - -**Key performance metrics (single L40S):** - - -| Metric | Value | -| --------------------------------------------- | ------------------------------------------- | -| **Single-user output tok/s** | ~50 (synchronous, Mdn) | -| **Single-user TTFT** | 60.9 ms | -| **Single-user ITL** | 19.8 ms | -| **Peak aggregate output tok/s** | ~1,571 (throughput strategy, 98 concurrent) | -| **Peak aggregate total tok/s** | ~4,793 (throughput strategy) | -| **Sustainable output tok/s @ ~22 concurrent** | ~841 | -| **Latency at ~22 concurrent** | TTFT ~76ms, ITL ~24ms | - - -#### Comparison: Qwen3-Coder-30B (MoE) vs Qwen3-8B (Dense) on Single L40S - - -| Metric | Qwen3-Coder-30B-A3B FP8 (MoE) | Qwen3-8B FP8 (Dense) | Winner | -| ------------------------------- | ----------------------------- | ---------------------- | --------------------- | -| **Model size** | 30B total, 3B active | 8.2B (all active) | — | -| **Weight memory (FP8)** | ~30 GB | ~8 GB | 8B (smaller) | -| **Context window** | 32,768 | 8,192 | 30B (4x longer) | -| **Single-user output tok/s** | 93 | 50 | **30B (1.9x faster)** | -| **Single-user TTFT** | 73.6 ms | 60.9 ms | **8B (17% faster)** | -| **Single-user ITL** | 10.4 ms | 19.8 ms | **30B (1.9x faster)** | -| **Peak output tok/s** | 1,357 | 1,571 | **8B (16% higher)** | -| **Peak total tok/s** | 4,140 | 4,793 | **8B (16% higher)** | -| **Output tok/s @ ~20-22 conc.** | ~692 | ~841 | **8B (22% higher)** | -| **TTFT @ ~20-22 conc.** | ~94 ms | ~76 ms | **8B (19% lower)** | -| **ITL @ ~20-22 conc.** | ~37 ms | ~24 ms | **8B (35% lower)** | -| **KV cache headroom** | ~16 GB (10 seqs @ 32K) | ~38 GB (475 seqs @ 8K) | 8B (massive headroom) | -| **Code quality** | MoE-specialized for coding | General-purpose | **30B (code-tuned)** | -| **Tool calling** | `qwen3_coder` parser | Not supported | **30B (native)** | - - -> **Analysis:** The 8B dense model achieves **16% higher peak throughput** and **35% lower ITL at moderate concurrency** than the 30B MoE model, thanks to its smaller weight footprint leaving more VRAM for KV cache. However, the 30B MoE model has **1.9x higher single-user generation speed** (93 vs 50 tok/s) because MoE activates only 3B params per token vs 8.2B for the dense model. The 30B model also supports **32K context** (vs 8K), native **tool calling** (`qwen3_coder` parser), and is **purpose-built for code** — making it the clear choice for an interactive coding assistant despite the lower aggregate throughput. -> -> **Multi-instance DP=2 was not feasible** on vLLM 0.13.0+rhai11 (engine core init crash). This would likely double the 8B throughput on vLLM 0.15+ but would not change the single-user latency advantage of the 30B MoE model. -> -> **Recommendation:** Use **Qwen3-Coder-30B MoE** for code assistant workloads (better per-user experience, tool calling, 32K context). Consider the **8B dense** model for high-throughput batch workloads where per-request latency is less important than aggregate tokens/sec. - -### Cross-Phase Cost Comparison (Qwen3-Coder-30B, 2 Instances) - - -| Configuration | Instances | Cost/hr | Est. peak tok/s | Cost per 1M tok | Dev capacity | -| --------------------------------- | --------- | ------- | ---------------------------------------- | ---------------------------------- | ------------ | -| 1x g6e.2xlarge (L40S, FP8) | 1 | $2.24 | 1,357 (tested, peak) / 692 (@ 30 conc.) | ~$0.46 (peak) / ~$0.90 (sustained) | 10-15 | -| 2x g6e.2xlarge (L40S, FP8) | 2 | $4.48 | ~2,714 (est. peak) / ~1,384 (@ 30 conc.) | ~$0.46 (peak) | 20-30 | -| 1x inf2.48xlarge (BF16, TP=8 × 2) | 2 | $12.98 | ~308 (est.) | ~$11.69 | 8-10 | -| 2x inf2.24xlarge (BF16, TP=8 × 1) | 2 | $12.98 | ~308 (est.) | ~$11.69 | 8-10 | - - -> **Why Inferentia costs more for this model:** Inferentia2 only supports BF16 (see "Why BF16 Only on Inferentia2" below), doubling the weight memory footprint vs FP8 and requiring TP=8 (8 NeuronCores per instance) instead of TP=1. This means each instance consumes an entire `inf2.24xlarge` worth of cores, while a single L40S GPU handles the same model at TP=1 with FP8 quantization. - -### Why BF16 Only on Inferentia2 - -AWS Inferentia2 chips use **NeuronCore-v2** processing elements with a hardware datapath designed around **BF16 and FP32** arithmetic. The chip does **not** include FP8 tensor cores or FP8 matrix multiply units. - -**Hardware-level constraints:** - -1. **No FP8 ALU**: The NeuronCore-v2 matrix multiply engine operates on BF16 operands natively. Unlike NVIDIA's Ada Lovelace (L40S) and Hopper/Blackwell (H100/H200/RTX PRO 6000) architectures, which added dedicated FP8 Tensor Core instructions (the `mma.fp8` warp-level primitives), the Inferentia2 silicon has no equivalent FP8 execution unit. There is no microcode path to perform 8-bit floating-point dot products. -2. **Neuron Compiler limitation**: The AWS Neuron Compiler (NeuronX CC) translates model graphs into NeuronCore-v2 instructions. It supports BF16, FP32, and limited INT8 (for specific quantized operations), but has **no FP8 quantization or dequantization kernels**. Even if you load FP8-quantized weights, the compiler cannot emit instructions to perform FP8 matrix multiplications — it would need to upcast everything to BF16 first, negating the memory savings. -3. **vLLM Neuron backend**: The vLLM Neuron integration (`neuronx-distributed-inference` framework) explicitly supports BF16 weight loading only. The `--quantization fp8` flag is not available in the Neuron code path. Model weights must be in BF16 (or FP32, which gets downcast to BF16 on-device). -4. **Memory impact**: BF16 uses 2 bytes per parameter vs FP8's 1 byte. For Qwen3-Coder-30B (30B total params), this means ~60 GB BF16 vs ~30 GB FP8. This forces TP=8 on Inferentia (spreading weights across 8 cores × 16 GB each = 128 GB) vs TP=1 on L40S (30 GB FP8 fits in 48 GB VRAM). -5. **Trn2 (next generation)**: AWS Trainium2 chips include hardware support for FP8 matrix operations. When Inferentia3 (or equivalent inference-focused Trainium2 derivatives) become available, FP8 inference on AWS custom silicon should be possible. As of March 2026, only Trainium2 (`trn2`) instances are available with FP8, and these are positioned for training workloads. - -**Practical consequence for this project:** The same Qwen3-Coder-30B model requires **2.9x more expensive** infrastructure on Inferentia2 ($6.49/hr for 1 instance) compared to L40S ($2.24/hr) primarily because BF16 doubles the memory footprint, forcing high tensor parallelism and consuming more NeuronCores per instance. - ---- - -## ARO Deployment: Qwen3.6-35B-A3B-FP8 on NVIDIA A100 - -In addition to the ROSA (AWS) deployment above, the Private Code Assistant has been deployed and benchmarked on **Azure Red Hat OpenShift (ARO)** with a different model and GPU configuration. - -| Component | Detail | -|-----------|--------| -| Platform | Azure Red Hat OpenShift (ARO) 4.19, Central US | -| GPU | NVIDIA A100 80 GB (`Standard_NC24ads_A100_v4`) | -| Model | `Qwen/Qwen3.6-35B-A3B-FP8` (MoE, ~3B active params, FP8 quantized) | -| Serving | vLLM 0.17.1 (upstream), KServe + llm-d AI Gateway | -| Context window | 65,536 tokens | -| RHOAI | 3.3.2 (stable-3.x) | - -### Key Performance (Single A100 80 GB) - -| Metric | Single User | Peak Throughput | -|--------|-------------|-----------------| -| Output tokens/s | 138 | 2,781 (short) / 2,008 (medium) | -| TTFT | 57–117 ms | — | -| ITL | 6.8 ms | — | -| Concurrent developers (30% active) | — | 3–4 per GPU (recommended SLO) | - -### Capacity Planning (30% Concurrency, Medium Prompt 512/256) - -| Developers | Concurrent Requests | Output tok/s | Tok/s Per User | ITL (ms) | GPUs Needed | Experience | -|:----------:|:-------------------:|:------------:|:--------------:|:--------:|:-----------:|:----------:| -| 10 | 3 | ~270 | ~90 | ~10 | 1 | Excellent | -| 20 | 6 | ~420 | ~70 | ~13 | 1 | Excellent | -| 50 | 15 | ~700 | ~47 | ~21 | 1 | Good | -| 100 | 30 | ~1,000 | ~33 | ~30 | 2 | Acceptable | - -Full benchmark results (GuideLLM v0.6.0 sweep across short/medium/long prompts) and detailed capacity planning tables are available in: - -- **[PCA_Deployment_ARO/testresults.md](PCA_Deployment_ARO/testresults.md)** — Complete performance test results and scaling recommendations -- **[PCA_Deployment_ARO/README.md](PCA_Deployment_ARO/README.md)** — ARO deployment instructions including the vLLM 0.17.1 upgrade workaround -- **[PCA_ARO.md](PCA_ARO.md)** — Architecture overview for the ARO-based deployment - ---- - -## Step-by-Step Deployment Guide - -### Prerequisites - - -| Requirement | Details | -| ---------------------------- | ------------------------------------------------------------ | -| **Cluster** | ROSA HCP 4.19+ (tested on 4.21.6) | -| **Red Hat OpenShift AI** | 3.3+ (provides `LLMInferenceService` CRD, KServe, llm-d EPP) | -| **Service Mesh 3** | 3.2.0+ (provides Gateway API with ext_proc support) | -| **cert-manager Operator** | 1.18+ (dependency of LeaderWorkerSet Operator) | -| **LeaderWorkerSet Operator** | 1.0.0 (required CRD for LLMInferenceService) | -| **Node Feature Discovery** | Deployed by default on ROSA HCP | -| **NVIDIA GPU Operator** | 26.3.0 (for GPU nodes) | -| **AWS Neuron Operator** | 1.1.1 (for Inferentia nodes) | -| **HuggingFace access** | Token with access to `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` | - - -### Step 1: Install Prerequisite Operators - -```bash -# cert-manager (required before LeaderWorkerSet) -oc apply -k "https://github.com/rhoai-rhtap/llm-d-playbook/main/kustomize/cert-manager/operator/overlays/stable" -oc apply -k "https://github.com/rhoai-rhtap/llm-d-playbook/main/kustomize/cert-manager/instance/overlays/default" - -# LeaderWorkerSet Operator -oc apply -k "https://github.com/rhoai-rhtap/llm-d-playbook/main/kustomize/lws/operator/overlays/stable" -``` - -After installing the LeaderWorkerSet Operator Subscription, create the LeaderWorkerSetOperator instance to trigger CRD creation: - -```bash -oc apply -f - < \ - --name=gpu-g6e-2xl-a \ - --instance-type=g6e.2xlarge \ - --replicas=1 \ - --labels="nvidia.com/gpu.present=true" \ - --taints="nvidia.com/gpu=present:NoSchedule" \ - --subnet= -``` - -The `NoSchedule` taint prevents non-GPU workloads from consuming CPU/memory on the GPU node. Only pods with a matching toleration (e.g., the model deployment and NVIDIA GPU Operator DaemonSets) will schedule here. - -Wait for the node to become `Ready`: - -```bash -oc get nodes -l nvidia.com/gpu.present=true -w -``` - -### Step 3: Install NVIDIA GPU Operator - -Install from OperatorHub (certified, `nvidia-gpu-operator` namespace). Then create the `ClusterPolicy`: - -```bash -oc apply -f - < feature.node.kubernetes.io/pci-10de.present=true -``` - -Verify GPU is allocatable: - -```bash -oc get node -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' -# Expected: 1 -``` - -### Step 4: Configure DataScienceCluster - -The `rawDeploymentServiceConfig` **must** be `Headed` (not the default `Headless`): - -```bash -oc patch datasciencecluster default-dsc --type='merge' \ - -p '{"spec":{"components":{"kserve":{"rawDeploymentServiceConfig":"Headed"}}}}' -``` - -### Step 5: Ensure Gateway TLS Secret - -The `maas-default-gateway` requires a TLS secret. If `data-science-gatewayconfig-tls` is missing in `openshift-ingress`, the gateway will have no HTTPS listener and all inference traffic will silently fail. - -```bash -# Check if the secret exists -oc get secret data-science-gatewayconfig-tls -n openshift-ingress - -# If missing, copy from the existing service cert -oc get secret data-science-gateway-service-tls -n openshift-ingress -o json | \ - jq '.metadata.name = "data-science-gatewayconfig-tls" | del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp)' | \ - oc apply -f - -``` - -### Step 6: Create Namespace and HuggingFace Secret - -```bash -oc new-project llm-d-gpu - -oc create secret generic hf-token -n llm-d-gpu \ - --from-literal=HF_TOKEN= -``` - -### Step 7: Create Model Cache PVC - -By default, KServe's `storage-initializer` init container downloads model weights into an `emptyDir` volume. This means every pod restart (rollout, node drain, OOM kill) triggers a full re-download of the model from HuggingFace — adding 4-15 minutes of cold-start time depending on model size and network conditions. - -To eliminate this, create a PersistentVolumeClaim backed by EBS gp3 storage and mount it as the `kserve-provision-location` volume in the LLMInferenceService template. The storage-initializer detects existing files on the PVC and skips the download entirely (0.2s warm start vs 4-8 min cold). - -```bash -oc apply -f - <<'EOF' -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: nvidia-model-cache - namespace: llm-d-gpu - labels: - app.kubernetes.io/part-of: llminferenceservice - purpose: model-weight-cache -spec: - accessModes: - - ReadWriteOnce - storageClassName: gp3-csi - resources: - requests: - storage: 100Gi -EOF -``` - -**Sizing guidance:** - - -| Model | Weights (FP8) | Weights (BF16) | Recommended PVC Size | -| ------------------------------------ | ------------- | -------------- | -------------------- | -| Qwen3-Coder-30B-A3B (MoE) | ~30 GB | ~57 GB | 100Gi | -| Gemma 4 26B-A4B (MoE) | ~14 GB | ~52 GB | 50Gi | -| Nemotron 3 Nano 30B-A3B (Hybrid MoE) | ~16 GB | ~60 GB | 50Gi | -| Qwen3-Coder-8B-A3B (MoE) | ~8 GB | ~16 GB | 50Gi | -| Dense 7B model | ~7 GB | ~14 GB | 50Gi | - - -> **Important:** Size the PVC at **2-3x the model weight size** to accommodate download cache files (`.cache/huggingface/download/` temporary files during the HuggingFace transfer). The PVC uses `ReadWriteOnce` — if running multiple replicas on different nodes, each node needs its own PVC or use `ReadWriteMany` (EFS). For single-replica or same-node multi-replica deployments, a single RWO PVC is sufficient. - -> **Do not delete PVCs when scaling down GPU nodes.** The PVC retains cached weights across node scale-down/scale-up cycles. When the node returns and the pod is rescheduled, the storage-initializer finds the cached model and starts in under 1 second. - -### Step 8: Deploy LLMInferenceService - -The `spec.template.volumes` section overrides the default `emptyDir` for the `kserve-provision-location` volume with the PVC created in Step 7. This is the volume that KServe's storage-initializer downloads model weights into and that vLLM loads from at `/mnt/models`. - -```bash -oc apply -f - <<'EOF' -apiVersion: serving.kserve.io/v1alpha1 -kind: LLMInferenceService -metadata: - name: qwen3-coder-fp8 - namespace: llm-d-gpu - annotations: - opendatahub.io/model-type: generative - openshift.io/display-name: qwen3-coder-fp8 - security.opendatahub.io/enable-auth: "false" -spec: - model: - name: Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 - uri: hf://Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 - replicas: 1 - router: - gateway: - refs: - - name: maas-default-gateway - namespace: openshift-ingress - route: {} - scheduler: - template: - containers: - - name: main - args: - - --pool-group - - inference.networking.x-k8s.io - - --pool-name - - "{{ ChildName .ObjectMeta.Name `-inference-pool` }}" - - --pool-namespace - - "{{ .ObjectMeta.Namespace }}" - - --zap-encoder - - json - - --grpc-port - - "9002" - - --grpc-health-port - - "9003" - - --secure-serving - - --model-server-metrics-scheme - - https - - --config-text - - | - apiVersion: inference.networking.x-k8s.io/v1alpha1 - kind: EndpointPickerConfig - plugins: - - type: single-profile-handler - - type: queue-scorer - - type: kv-cache-utilization-scorer - - type: prefix-cache-scorer - schedulingProfiles: - - name: default - plugins: - - pluginRef: queue-scorer - weight: 2 - - pluginRef: kv-cache-utilization-scorer - weight: 2 - - pluginRef: prefix-cache-scorer - weight: 3 - env: - - name: TOKENIZER_CACHE_DIR - value: /tmp/tokenizer-cache - - name: HF_HOME - value: /tmp/tokenizer-cache - - name: TRANSFORMERS_CACHE - value: /tmp/tokenizer-cache - - name: XDG_CACHE_HOME - value: /tmp - volumeMounts: - - mountPath: /tmp/tokenizer-cache - name: tokenizer-cache - - mountPath: /cachi2 - name: cachi2-cache - volumes: - - emptyDir: {} - name: tokenizer-cache - - emptyDir: {} - name: cachi2-cache - template: - # PVC volume: overrides the default emptyDir for kserve-provision-location. - # The storage-initializer init container downloads to /mnt/models (this volume). - # On subsequent restarts, it detects existing files and completes in <1 second. - volumes: - - name: kserve-provision-location - persistentVolumeClaim: - claimName: nvidia-model-cache - containers: - - name: main - env: - - name: VLLM_ADDITIONAL_ARGS - value: "--disable-uvicorn-access-log --max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --kv-cache-dtype fp8" - volumeMounts: - - name: kserve-provision-location - mountPath: /mnt/models - resources: - requests: - cpu: "4" - memory: "48Gi" - nvidia.com/gpu: "1" - limits: - cpu: "4" - memory: "48Gi" - nvidia.com/gpu: "1" - livenessProbe: - httpGet: - path: /health - port: 8000 - scheme: HTTPS - initialDelaySeconds: 300 - periodSeconds: 30 - timeoutSeconds: 30 - failureThreshold: 10 - readinessProbe: - httpGet: - path: /health - port: 8000 - scheme: HTTPS - initialDelaySeconds: 120 - periodSeconds: 15 - timeoutSeconds: 10 - failureThreshold: 5 - nodeSelector: - node.kubernetes.io/instance-type: g6e.2xlarge - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule -EOF -``` - -**How the PVC caching works:** - -``` -First deployment (cold start): - storage-initializer → downloads model from HuggingFace to PVC (/mnt/models) → 4-8 min - vLLM → loads weights from PVC into GPU VRAM → 3-5 min - Total: ~7-13 min - -Subsequent restarts (warm start): - storage-initializer → detects files on PVC, skips download → 0.2 sec - vLLM → loads weights from PVC into GPU VRAM → 3-5 min - Total: ~3-5 min (download eliminated) -``` - -> **Switching models:** If you change `spec.model.uri` to a different model, the storage-initializer will detect a mismatch and download the new model. Clear old files from the PVC first to avoid running out of space: -> -> ```bash -> oc exec -n llm-d-gpu deploy/-kserve -c storage-initializer -- \ -> find /mnt/models -maxdepth 1 -name "*.safetensors" -delete -> ``` -> -> Alternatively, create a separate PVC per model to enable fast switching between models without re-downloading. - -### Step 9: Verify Deployment - -```bash -# Wait for pod to be Ready -oc get pods -n llm-d-gpu -w - -# Get the gateway URL -GW_URL=$(oc get gateway -n openshift-ingress maas-default-gateway \ - -o jsonpath='{.status.addresses[0].value}') - -# Test model endpoint -curl -sk "https://${GW_URL}/llm-d-gpu/qwen3-coder-fp8/v1/models" | python3 -m json.tool - -# Test chat completions -curl -sk "https://${GW_URL}/llm-d-gpu/qwen3-coder-fp8/v1/chat/completions" \ - -H "Content-Type: application/json" \ - -d '{"model":"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8","messages":[{"role":"user","content":"Write a Python hello world"}],"max_tokens":100}' -``` - -### Step 9b: Alternative Model Deployments (Gemma 4, Nemotron 3 Nano) - -Deploy additional models alongside Qwen3-Coder so they appear in the RHOAI dashboard as deployable models. Each model has its own PVC with pre-downloaded weights and an `LLMInferenceService` at `replicas: 0`. Only **one model should be active at a time** on a single L40S GPU -- scale down the current model before scaling up another. - -> **vLLM version requirement:** Both Gemma 4 and Nemotron 3 Nano require vLLM 0.14.1+ (RHOAI 3.4 GA). These deployments can be created now but will fail to start on vLLM 0.13.x. Create the PVCs and download weights now so models are ready when RHOAI 3.4 GA ships. - -#### 9b.1: Create PVCs for each model - -```bash -# Gemma 4 26B-A4B: ~14 GB FP8 weights, 50Gi PVC gives 2-3x headroom for HF cache -oc apply -f - <<'EOF' -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: gemma4-26b-model-cache - namespace: llm-d-multi-gpu - labels: - app.kubernetes.io/part-of: llminferenceservice - purpose: model-weight-cache -spec: - accessModes: - - ReadWriteOnce - storageClassName: gp3-csi - resources: - requests: - storage: 50Gi -EOF - -# Nemotron 3 Nano 30B-A3B-FP8: ~16 GB FP8 weights, 50Gi PVC -oc apply -f - <<'EOF' -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: nemotron3-nano-model-cache - namespace: llm-d-multi-gpu - labels: - app.kubernetes.io/part-of: llminferenceservice - purpose: model-weight-cache -spec: - accessModes: - - ReadWriteOnce - storageClassName: gp3-csi - resources: - requests: - storage: 50Gi -EOF -``` - -#### 9b.2: Pre-download model weights to PVCs - -Run one-shot Jobs on the GPU node to download weights. The Jobs use `nodeSelector` to ensure the PVC binds to the same AZ as the GPU node. - -```bash -# Download Gemma 4 26B-A4B-it (BF16 base; FP8 quantized at runtime by vLLM) -oc apply -f - <<'EOF' -apiVersion: batch/v1 -kind: Job -metadata: - name: download-gemma4-26b - namespace: llm-d-multi-gpu -spec: - backoffLimit: 2 - template: - spec: - nodeSelector: - node.kubernetes.io/instance-type: g6e.2xlarge - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - restartPolicy: Never - containers: - - name: downloader - image: python:3.11-slim - command: ["/bin/bash", "-c"] - args: - - | - pip install -q huggingface_hub && \ - huggingface-cli download google/gemma-4-26B-A4B-it \ - --local-dir /mnt/models \ - --local-dir-use-symlinks False - volumeMounts: - - name: model-storage - mountPath: /mnt/models - resources: - requests: - cpu: "2" - memory: "8Gi" - limits: - cpu: "4" - memory: "16Gi" - volumes: - - name: model-storage - persistentVolumeClaim: - claimName: gemma4-26b-model-cache -EOF - -# Download Nemotron 3 Nano 30B-A3B-FP8 -oc apply -f - <<'EOF' -apiVersion: batch/v1 -kind: Job -metadata: - name: download-nemotron3-nano - namespace: llm-d-multi-gpu -spec: - backoffLimit: 2 - template: - spec: - nodeSelector: - node.kubernetes.io/instance-type: g6e.2xlarge - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - restartPolicy: Never - containers: - - name: downloader - image: python:3.11-slim - command: ["/bin/bash", "-c"] - args: - - | - pip install -q huggingface_hub && \ - huggingface-cli download nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ - --local-dir /mnt/models \ - --local-dir-use-symlinks False - volumeMounts: - - name: model-storage - mountPath: /mnt/models - resources: - requests: - cpu: "2" - memory: "8Gi" - limits: - cpu: "4" - memory: "16Gi" - volumes: - - name: model-storage - persistentVolumeClaim: - claimName: nemotron3-nano-model-cache -EOF -``` - -Monitor downloads: - -```bash -oc logs -n llm-d-multi-gpu job/download-gemma4-26b -f -oc logs -n llm-d-multi-gpu job/download-nemotron3-nano -f -``` - -Clean up Jobs after completion: - -```bash -oc delete job download-gemma4-26b download-nemotron3-nano -n llm-d-multi-gpu -``` - -#### 9b.3: Deploy Gemma 4 26B-A4B LLMInferenceService (replicas: 0) - -```bash -oc apply -f - <<'EOF' -apiVersion: serving.kserve.io/v1alpha1 -kind: LLMInferenceService -metadata: - name: gemma4-26b-a4b - namespace: llm-d-multi-gpu - annotations: - opendatahub.io/model-type: generative - openshift.io/display-name: "Gemma 4 26B-A4B (MoE)" - security.opendatahub.io/enable-auth: "false" -spec: - model: - name: google/gemma-4-26B-A4B-it - uri: hf://google/gemma-4-26B-A4B-it - replicas: 0 - router: - gateway: - refs: - - name: maas-default-gateway - namespace: openshift-ingress - route: {} - scheduler: - template: - containers: - - name: main - args: - - --pool-group - - inference.networking.x-k8s.io - - --pool-name - - "{{ ChildName .ObjectMeta.Name `-inference-pool` }}" - - --pool-namespace - - "{{ .ObjectMeta.Namespace }}" - - --zap-encoder - - json - - --grpc-port - - "9002" - - --grpc-health-port - - "9003" - - --secure-serving - - --model-server-metrics-scheme - - https - - --config-text - - | - apiVersion: inference.networking.x-k8s.io/v1alpha1 - kind: EndpointPickerConfig - plugins: - - type: single-profile-handler - - type: queue-scorer - - type: kv-cache-utilization-scorer - - type: prefix-cache-scorer - schedulingProfiles: - - name: default - plugins: - - pluginRef: queue-scorer - weight: 2 - - pluginRef: kv-cache-utilization-scorer - weight: 2 - - pluginRef: prefix-cache-scorer - weight: 3 - env: - - name: TOKENIZER_CACHE_DIR - value: /tmp/tokenizer-cache - - name: HF_HOME - value: /tmp/tokenizer-cache - - name: TRANSFORMERS_CACHE - value: /tmp/tokenizer-cache - - name: XDG_CACHE_HOME - value: /tmp - volumeMounts: - - mountPath: /tmp/tokenizer-cache - name: tokenizer-cache - - mountPath: /cachi2 - name: cachi2-cache - volumes: - - emptyDir: {} - name: tokenizer-cache - - emptyDir: {} - name: cachi2-cache - template: - volumes: - - name: kserve-provision-location - persistentVolumeClaim: - claimName: gemma4-26b-model-cache - containers: - - name: main - env: - - name: VLLM_ADDITIONAL_ARGS - value: "--disable-uvicorn-access-log --max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 --kv-cache-dtype fp8" - volumeMounts: - - name: kserve-provision-location - mountPath: /mnt/models - resources: - requests: - cpu: "4" - memory: "48Gi" - nvidia.com/gpu: "1" - limits: - cpu: "4" - memory: "48Gi" - nvidia.com/gpu: "1" - livenessProbe: - httpGet: - path: /health - port: 8000 - scheme: HTTPS - initialDelaySeconds: 300 - periodSeconds: 30 - timeoutSeconds: 30 - failureThreshold: 10 - readinessProbe: - httpGet: - path: /health - port: 8000 - scheme: HTTPS - initialDelaySeconds: 120 - periodSeconds: 15 - timeoutSeconds: 10 - failureThreshold: 5 - nodeSelector: - node.kubernetes.io/instance-type: g6e.2xlarge - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule -EOF -``` - -#### 9b.4: Deploy Nemotron 3 Nano 30B-A3B LLMInferenceService (replicas: 0) - -```bash -oc apply -f - <<'EOF' -apiVersion: serving.kserve.io/v1alpha1 -kind: LLMInferenceService -metadata: - name: nemotron3-nano-fp8 - namespace: llm-d-multi-gpu - annotations: - opendatahub.io/model-type: generative - openshift.io/display-name: "Nemotron 3 Nano 30B-A3B FP8" - security.opendatahub.io/enable-auth: "false" -spec: - model: - name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 - uri: hf://nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 - replicas: 0 - router: - gateway: - refs: - - name: maas-default-gateway - namespace: openshift-ingress - route: {} - scheduler: - template: - containers: - - name: main - args: - - --pool-group - - inference.networking.x-k8s.io - - --pool-name - - "{{ ChildName .ObjectMeta.Name `-inference-pool` }}" - - --pool-namespace - - "{{ .ObjectMeta.Namespace }}" - - --zap-encoder - - json - - --grpc-port - - "9002" - - --grpc-health-port - - "9003" - - --secure-serving - - --model-server-metrics-scheme - - https - - --config-text - - | - apiVersion: inference.networking.x-k8s.io/v1alpha1 - kind: EndpointPickerConfig - plugins: - - type: single-profile-handler - - type: queue-scorer - - type: kv-cache-utilization-scorer - - type: prefix-cache-scorer - schedulingProfiles: - - name: default - plugins: - - pluginRef: queue-scorer - weight: 2 - - pluginRef: kv-cache-utilization-scorer - weight: 2 - - pluginRef: prefix-cache-scorer - weight: 3 - env: - - name: TOKENIZER_CACHE_DIR - value: /tmp/tokenizer-cache - - name: HF_HOME - value: /tmp/tokenizer-cache - - name: TRANSFORMERS_CACHE - value: /tmp/tokenizer-cache - - name: XDG_CACHE_HOME - value: /tmp - volumeMounts: - - mountPath: /tmp/tokenizer-cache - name: tokenizer-cache - - mountPath: /cachi2 - name: cachi2-cache - volumes: - - emptyDir: {} - name: tokenizer-cache - - emptyDir: {} - name: cachi2-cache - template: - volumes: - - name: kserve-provision-location - persistentVolumeClaim: - claimName: nemotron3-nano-model-cache - containers: - - name: main - env: - - name: VLLM_ADDITIONAL_ARGS - value: "--disable-uvicorn-access-log --max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser nemotron_nano --kv-cache-dtype fp8" - volumeMounts: - - name: kserve-provision-location - mountPath: /mnt/models - resources: - requests: - cpu: "4" - memory: "48Gi" - nvidia.com/gpu: "1" - limits: - cpu: "4" - memory: "48Gi" - nvidia.com/gpu: "1" - livenessProbe: - httpGet: - path: /health - port: 8000 - scheme: HTTPS - initialDelaySeconds: 300 - periodSeconds: 30 - timeoutSeconds: 30 - failureThreshold: 10 - readinessProbe: - httpGet: - path: /health - port: 8000 - scheme: HTTPS - initialDelaySeconds: 120 - periodSeconds: 15 - timeoutSeconds: 10 - failureThreshold: 5 - nodeSelector: - node.kubernetes.io/instance-type: g6e.2xlarge - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule -EOF -``` - -#### 9b.5: Model Switching Procedure - -All three models appear in the RHOAI dashboard under **Deployed Models**. Only one should have `replicas: 1` at a time (single L40S GPU constraint). - -**Switch from Qwen3-Coder to Gemma 4:** - -```bash -# 1. Scale down current model -oc patch llminferenceservice qwen3-coder-fp8 -n llm-d-multi-gpu \ - --type=merge -p '{"spec":{"replicas":0}}' - -# 2. Wait for pod termination -oc wait --for=delete pod -l app.kubernetes.io/name=qwen3-coder-fp8 \ - -n llm-d-multi-gpu --timeout=120s - -# 3. Scale up new model -oc patch llminferenceservice gemma4-26b-a4b -n llm-d-multi-gpu \ - --type=merge -p '{"spec":{"replicas":1}}' - -# 4. Wait for new model pod to be Ready (~3-5 min with PVC cache) -oc get pods -n llm-d-multi-gpu -l app.kubernetes.io/name=gemma4-26b-a4b -w -``` - -**Update extension model name** (required -- each model has its own InferenceModel route): - - -| Model | `model` value for extensions | -| --------------- | ------------------------------------------- | -| Qwen3-Coder | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` | -| Gemma 4 26B-A4B | `google/gemma-4-26B-A4B-it` | -| Nemotron 3 Nano | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8` | - - -Update the model name in the Continue `config.yaml`, Roo Code ConfigMap, and Cline settings. The gateway URL (`https://llm-d-gateway-data-science-gateway-class.llm-d-multi-gpu.svc.cluster.local/v1`) stays the same -- only the `model` field changes. - -> **Why the model name must change:** Each `LLMInferenceService` creates its own `InferenceModel` CRD with its HuggingFace model ID. The llm-d EPP routes requests by matching the `model` field in the API request to the `InferenceModel`. Using the wrong model name results in a 404 from the gateway. The gateway URL itself is shared across all models in the namespace. - -#### Alternative Model Summary - - -| Model | Total / Active | FP8 Weights | PVC | Tool Parser | Reasoning Parser | vLLM Required | -| ----------------------------- | -------------- | ----------- | ----------------------------------- | --------------- | ---------------- | ------------- | -| Qwen3-Coder-30B-A3B (current) | 30B / 3B | ~16 GB | `nvidia-model-cache` (100Gi) | `qwen3_coder` | `qwen3` | 0.13+ | -| Gemma 4 26B-A4B | 26B / 4B | ~14 GB | `gemma4-26b-model-cache` (50Gi) | `gemma4` | `gemma4` | **0.14.1+** | -| Nemotron 3 Nano 30B-A3B | 30B / 3.2B | ~16 GB | `nemotron3-nano-model-cache` (50Gi) | `nemotron_nano` | *(none)* | **0.14+** | - - -### Step 10: Scale to Multiple Replicas (Phase 2) - -```bash -# Scale machine pool to 2 nodes -rosa edit machinepool --cluster= gpu-g6e-2xl-a --replicas=2 - -# Wait for second node to be Ready -oc get nodes -l nvidia.com/gpu.present=true -w - -# Scale the LLMInferenceService -oc patch llminferenceservice -n llm-d-gpu qwen3-coder-fp8 --type='merge' \ - -p '{"spec":{"replicas":2}}' -``` - -### Step 11: Verify Load Balancing - -```bash -# Check both pods are running -oc get pods -n llm-d-gpu -o wide - -# Verify prefix-cache-aware routing by sending repeated prompts -# and checking per-pod metrics: -for POD in $(oc get pods -n llm-d-gpu -l app=qwen3-coder-fp8-kserve -o name); do - echo "--- $POD ---" - oc exec -n llm-d-gpu $POD -- curl -sk https://localhost:8000/metrics 2>/dev/null | \ - grep -E "prefix_cache_(hits|queries)_total|num_requests" -done -``` - -### Step 12: Deploy Inferentia Model via KServe InferenceService - -For Inferentia2 nodes, deploy using a standard `InferenceService` with a custom `ServingRuntime` (the `LLMInferenceService` CRD does not support Neuron accelerators). - -**11a. Create namespace and PVC:** - -```bash -oc new-project llm-serving - -# EBS PVC for model weights and Neuron compilation cache -oc apply -f - <<'EOF' -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: model-storage - namespace: llm-serving -spec: - accessModes: [ReadWriteOnce] - resources: - requests: - storage: 200Gi - storageClassName: gp3-csi -EOF -``` - -**11b. Download the model to PVC:** - -```bash -oc apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: model-downloader - namespace: llm-serving -spec: - nodeSelector: - node.kubernetes.io/instance-type: inf2.24xlarge - restartPolicy: Never - containers: - - name: downloader - image: python:3.11-slim - command: ["/bin/bash", "-c"] - args: - - | - pip install huggingface_hub && \ - huggingface-cli download Qwen/Qwen3-Coder-30B-A3B-Instruct \ - --local-dir /models/qwen3-coder-30b \ - --token $HF_TOKEN - env: - - name: HF_TOKEN - valueFrom: - secretKeyRef: - name: hf-token - key: token - volumeMounts: - - name: models - mountPath: /models - volumes: - - name: models - persistentVolumeClaim: - claimName: model-storage -EOF -``` - -**11c. Create ServingRuntime:** - -```bash -oc apply -f - <<'EOF' -apiVersion: serving.kserve.io/v1alpha1 -kind: ServingRuntime -metadata: - name: vllm-neuron-runtime - namespace: llm-serving - annotations: - openshift.io/display-name: "vLLM Neuron Runtime" -spec: - supportedModelFormats: - - name: vllm-neuron - autoSelect: true - multiModel: false - containers: - - name: kserve-container - image: public.ecr.aws/neuron/pytorch-inference-vllm-neuronx:0.13.0-neuronx-py312-sdk2.28.0-ubuntu24.04 - command: ["python3", "-m", "vllm.entrypoints.openai.api_server"] - ports: - - containerPort: 8000 - name: http - protocol: TCP -EOF -``` - -**11d. Deploy InferenceService with PVC NEFF cache and right-sized memory:** - -> **CRITICAL REQUIREMENTS — Inferentia Model Deployments:** -> -> 1. **EBS-backed PVC for ALL caches (MANDATORY):** Both `NEURON_COMPILE_CACHE_URL` and `HF_HOME` **must** point to paths on an EBS-backed PVC (`gp3-csi`), never to ephemeral container storage (`/tmp`, `emptyDir`). NEFF compilation artifacts are 20-40 GB and take 30-45 minutes to regenerate. Losing them on pod restart wastes compute hours and delays recovery. -> 2. **Memory limits for Neuron compilation (MANDATORY):** Qwen3-Coder-30B MoE (BF16) requires **≥200Gi memory limit** during first-time Neuron compilation. The compilation phase peaks at ~120 GB RSS (57 GB model weights + 60+ GB compiler workspace). After compilation, steady-state serving uses only ~15-20 GB. A 64Gi limit causes OOMKill during weight loading; a 128Gi limit causes OOMKill during bucket compilation. Set **requests: 128Gi, limits: 200Gi** minimum. -> 3. **PVC sizing:** Allocate ≥200Gi for the EBS PVC: ~57 GB for HuggingFace model cache + 20-40 GB for NEFF artifacts + headroom. -> 4. **Do NOT delete the PVC** when scaling down Inferentia nodes. The EBS volume persists independently. - -```bash -oc apply -f - <<'EOF' -apiVersion: serving.kserve.io/v1beta1 -kind: InferenceService -metadata: - name: qwen3-coder-neuron - namespace: llm-serving - annotations: - serving.kserve.io/deploymentMode: RawDeployment - opendatahub.io/model-type: generative - openshift.io/display-name: "Qwen3-Coder-30B (Inferentia2, TP=8)" - security.opendatahub.io/enable-auth: "false" -spec: - predictor: - nodeSelector: - node.kubernetes.io/instance-type: inf2.24xlarge - schedulerName: neuron-scheduler - tolerations: - - key: aws.amazon.com/neuroncore - operator: Exists - effect: NoSchedule - securityContext: - fsGroup: 0 - runAsUser: 0 - seLinuxOptions: - # Required when multiple pods share the same ReadWriteOnce PVC -- - # forces identical MCS labels so both pods can read/write the volume - level: "s0:c0,c0" - terminationGracePeriodSeconds: 60 - volumes: - - name: models - persistentVolumeClaim: - claimName: model-storage - - name: dshm - emptyDir: - medium: Memory - sizeLimit: 16Gi - initContainers: - - name: fix-perms - image: registry.access.redhat.com/ubi9/ubi-minimal:latest - command: ["sh", "-c", "chmod -R 777 /models/ && echo perms fixed"] - securityContext: - runAsUser: 0 - volumeMounts: - - name: models - mountPath: /models - model: - modelFormat: - name: vllm-neuron - runtime: vllm-neuron-runtime - command: - - /bin/bash - - -c - - | - echo "Waiting for model files to be accessible..." - for i in $(seq 1 60); do - if python3 -c "import os; os.stat('/models/qwen3-coder-30b/config.json')" 2>/dev/null; then - echo "Model directory accessible after ${i}s" - break - fi - sleep 1 - done - exec python3 -m vllm.entrypoints.openai.api_server \ - --model /models/qwen3-coder-30b \ - --served-model-name qwen3-coder-neuron \ - --tensor-parallel-size 8 \ - --max-model-len 8192 \ - --max-num-seqs 16 \ - --block-size 32 \ - --no-enable-prefix-caching \ - --num-gpu-blocks-override 32 \ - --additional-config '{"tp_degree":8,"moe_tp_degree":1,"moe_ep_degree":8,"batch_size":16,"seq_len":8192,"context_encoding_buckets":[[16,128],[16,512],[16,1024],[16,2048],[16,4096],[16,8192]],"token_generation_buckets":[[16,128],[16,512],[16,1024],[16,2048],[16,4096],[16,8192]],"on_device_generation":{"do_sample":true},"enable_fused_rmsnorm_quantization":true,"enable_fused_experts_quantization":true,"enable_nki_quantized_kernels":true}' \ - --port 8000 --host 0.0.0.0 - env: - - name: NEURON_RT_VISIBLE_CORES - value: "0-7" - - name: NEURON_COMPILE_CACHE_URL - # CRITICAL: Must point to EBS PVC path, NOT ephemeral storage. - # First compilation takes 30-45 min; warm-cache restarts take ~10s. - value: "/models/neuron-cache" - - name: VLLM_ENGINE_ITERATION_TIMEOUT_S - value: "7200" - - name: VLLM_RPC_TIMEOUT - value: "7200" - - name: VLLM_NEURON_FRAMEWORK - value: "neuronx-distributed-inference" - - name: HF_HOME - # CRITICAL: Must point to EBS PVC path for persistent HF model cache. - # Model download is ~57 GB; caching avoids re-download on pod restart. - value: "/models/hf-cache" - ports: - - containerPort: 8000 - name: http - protocol: TCP - securityContext: - capabilities: - add: [IPC_LOCK, SYS_ADMIN] - resources: - requests: - cpu: "8" - memory: 128Gi - aws.amazon.com/neuroncore: "8" - limits: - cpu: "16" - # RIGHT-SIZING: Neuron compilation of Qwen3-Coder-30B MoE peaks - # at ~120 GB RSS (57 GB weights + 60+ GB compiler workspace). - # 64Gi → OOMKill during weight loading. - # 128Gi → OOMKill during bucket compilation. - # 200Gi is the tested minimum for cold-cache compilation. - memory: 200Gi - aws.amazon.com/neuroncore: "8" - volumeMounts: - - name: models - mountPath: /models - - name: dshm - mountPath: /dev/shm - readinessProbe: - httpGet: - path: /health - port: 8000 - initialDelaySeconds: 10 - periodSeconds: 30 - timeoutSeconds: 5 - failureThreshold: 60 - livenessProbe: - httpGet: - path: /health - port: 8000 - initialDelaySeconds: 3600 - periodSeconds: 60 - timeoutSeconds: 10 - failureThreshold: 10 -EOF -``` - -**Key deployment notes:** - -- **Memory right-sizing (CRITICAL)**: Neuron compilation of Qwen3-Coder-30B MoE peaks at ~120 GB RSS (57 GB BF16 weights + 60+ GB compiler workspace for bucket compilation). A 64Gi limit causes OOMKill during weight loading; a 128Gi limit causes OOMKill during bucket compilation. **Set memory limits to 200Gi minimum** for first-time (cold-cache) compilation. After compilation completes and NEFF artifacts are cached on the PVC, steady-state serving uses only ~15-20 GB. Always check `oc describe pod` for OOMKill exit codes (137) and adjust limits upward if compilation fails. -- **EBS PVC for all caches (CRITICAL)**: Both `NEURON_COMPILE_CACHE_URL` and `HF_HOME` **must** point to paths on the EBS-backed PVC, not to ephemeral storage (`/tmp`, `emptyDir`). NEFF artifacts are 20-40 GB and take 30-45 minutes to regenerate; the HuggingFace model cache is ~57 GB. Using ephemeral storage means every pod restart triggers a full re-download and re-compilation cycle (30-45+ minutes). KServe 0.14+ (RHOAI 3.3) supports read/write PVC access in `RawDeployment` mode. -- **SELinux for shared PVC**: When multiple pods share a `ReadWriteOnce` PVC (e.g., model downloader + inference server), set `seLinuxOptions.level: "s0:c0,c0"` on all pods to ensure matching MCS labels. Without this, SELinux denies cross-pod file access even with `privileged` SCC. -- **MutatingAdmissionWebhook**: Deploy the OVN annotation preserver webhook (Appendix B) **before** creating this InferenceService to prevent the neuron-scheduler / OVN annotation race. - -### Step 13: Verify Inferentia Deployment - -```bash -# Wait for the InferenceService to be Ready -oc get inferenceservice qwen3-coder-neuron -n llm-serving -w - -# Test the model endpoint -INF_URL=$(oc get inferenceservice qwen3-coder-neuron -n llm-serving \ - -o jsonpath='{.status.url}') -curl -s "${INF_URL}/v1/chat/completions" \ - -H "Content-Type: application/json" \ - -d '{"model":"qwen3-coder-neuron","messages":[{"role":"user","content":"Write a Python hello world"}],"max_tokens":100}' | python3 -m json.tool -``` - -### Step 14: Deploy OpenShift Dev Spaces - -Install the **Red Hat OpenShift Dev Spaces** operator (v3.27) from OperatorHub. Create the `CheCluster`: - -```bash -oc apply -f - < **Why the symlink approach (not a copy):** -> Mounting the ConfigMap directly **at** `CONTINUE_GLOBAL_DIR` (`/home/user/.continue-data`) makes the entire directory read-only (owned by root, mode `drwxr-xr-x`). Continue cannot write session files → activation fails with "error activating continue extension". A script-based copy into a pre-created writable directory works but is racy: Continue initialises before the postStart script finishes and writes a blank `models: []` default, overwriting the copy. A symlink is created once and is never stale. +Deploy a private, self-hosted AI coding assistant on OpenShift so developers get AI-powered IDEs while inference stays on-cluster — no source code needs to leave your environment. -> `**systemMessage`:** Added to the chat model in `config.yaml` to prevent Qwen3-Coder from spontaneously generating XML tool-call syntax (``) in Continue responses. Without it, vLLM's `--enable-auto-tool-choice` causes the model's tool-call XML to be parsed into `tool_calls` even for plain Continue requests, producing "Invalid tool name" errors. - -Apply the source ConfigMap to `openshift-devspaces` — it syncs to user namespaces automatically: - -```bash -cat << 'EOF' | oc apply -f - -apiVersion: v1 -kind: ConfigMap -metadata: - name: continue-config - namespace: openshift-devspaces - labels: - app.kubernetes.io/component: workspaces-config - app.kubernetes.io/part-of: che.eclipse.org - controller.devfile.io/mount-to-devworkspace: "true" - controller.devfile.io/watch-configmap: "true" - annotations: - controller.devfile.io/mount-as: subpath - controller.devfile.io/mount-path: /home/user/.continue -data: - config.yaml: | - name: Private Code Assistant - version: 1.0.0 - schema: v1 - models: - - name: Qwen3-Coder (llm-d) - provider: openai - model: Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 - apiBase: https://llm-d-gateway-data-science-gateway-class.llm-d-multi-gpu.svc.cluster.local/v1 - apiKey: EMPTY - systemMessage: "You are a helpful coding assistant. Respond using natural language and code blocks only. Never use XML tool-call syntax such as , , or in your responses." - requestOptions: - verifySsl: false - tabAutocompleteModel: - name: Qwen3-Coder Autocomplete (llm-d) - provider: openai - model: Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 - apiBase: https://llm-d-gateway-data-science-gateway-class.llm-d-multi-gpu.svc.cluster.local/v1 - apiKey: EMPTY - requestOptions: - verifySsl: false -EOF -``` - -The DevWorkspace postStart script sets up the symlink on every pod start: - -```bash -# Create writable CONTINUE_GLOBAL_DIR -mkdir -p /home/user/.continue-data -# Symlink config.yaml from read-only ConfigMap mount (resolves at read-time, never stale) -rm -f /home/user/.continue-data/config.yaml -ln -sf /home/user/.continue/config.yaml /home/user/.continue-data/config.yaml -# Suppress Continue ENOENT warning about missing ~/Desktop -mkdir -p /home/user/Desktop -``` - -**Verifying inside a running pod:** - -```bash -WS_POD=$(oc get pods -n -devspaces -o name | grep workspace | head -1 | cut -d/ -f2) -oc exec $WS_POD -n -devspaces -c dev-tools -- sh -c ' - echo "=== Symlink ===" - ls -la /home/user/.continue-data/config.yaml - echo "=== Config (first 4 lines) ===" - head -4 /home/user/.continue-data/config.yaml - echo "=== Write test ===" - touch /home/user/.continue-data/test && echo WRITABLE && rm /home/user/.continue-data/test -' -# Expected: -# lrwxrwxrwx ... config.yaml -> /home/user/.continue/config.yaml -# name: Private Code Assistant -# WRITABLE -``` - -**Required container env vars** (set in DevWorkspace `spec.template.components[].container.env`): - - -| Variable | Value | Purpose | -| --------------------- | --------------------------- | ---------------------------------------------------------------------------------------- | -| `CONTINUE_GLOBAL_DIR` | `/home/user/.continue-data` | Points Continue to the writable directory containing the config symlink | -| `KUBEDOCK_ENABLED` | `true` | Enables docker CLI inside the workspace via Kubernetes API (required for workspace logs) | - - -**Known cosmetic warning:** Continue logs `Error loading colocated rule files: ENOENT: no such file or directory, scandir '/home/user/Desktop'`. Harmless — suppressed by `mkdir -p /home/user/Desktop` in the postStart script. - -**14c. Cline extension configuration (manual UI setup):** - -Cline v3.79+ stores API provider settings in VS Code's internal globalState (not in files). It cannot be fully pre-configured via ConfigMaps. The extension is installed by the `postStart` script; the user must configure the API provider once through the Cline UI: - -1. Open the Cline sidebar (click the Cline icon or `Ctrl+Shift+P` → "Cline: Focus on View") -2. Click the **Settings** gear icon -3. Under **API Provider**, select **"OpenAI Compatible"** -4. Fill in: - - **Base URL:** `https://llm-d-gateway-data-science-gateway-class.llm-d-multi-gpu.svc.cluster.local/v1` - - **Model ID:** `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` - - **API Key:** `EMPTY` - -These settings persist across workspace restarts (stored in the workspace's persistent volume). - -**14d. Roo Code extension configuration (ConfigMap-based, fully automated):** - -Roo Code is the only extension that can be **fully pre-configured via ConfigMaps** for zero-touch setup. Two ConfigMaps work together: - -1. `**roo-code-provider-config`** — mounted at the Roo Code globalStorage path, sets the active API provider: - -```bash -oc apply -f - <<'EOF' -apiVersion: v1 -kind: ConfigMap -metadata: - name: roo-code-provider-config - namespace: -devspaces - labels: - controller.devfile.io/mount-to-devworkspace: "true" - controller.devfile.io/watch-configmap: "true" - annotations: - controller.devfile.io/mount-as: subpath - # Roo Code 3.x reads from /checode/remote/data/ (NOT /checode/checode-linux-libc/) - controller.devfile.io/mount-path: /checode/remote/data/User/globalStorage/rooveterinaryinc.roo-cline/settings -data: - cline_custom_modes.json: '[]' - provider_profiles.json: | - { - "currentApiConfigName": "llm-d-qwen3-coder", - "apiConfigs": { - "llm-d-qwen3-coder": { - "id": "llm-d-qwen3-coder", - "apiProvider": "openai", - "openAiBaseUrl": "https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1", - "openAiApiKey": "EMPTY", - "openAiModelId": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8", - "openAiStreamingEnabled": false - } - }, - "modeApiConfigs": { - "code": "llm-d-qwen3-coder", - "architect": "llm-d-qwen3-coder", - "ask": "llm-d-qwen3-coder", - "debug": "llm-d-qwen3-coder", - "orchestrator": "llm-d-qwen3-coder" - } - } -EOF -``` - -1. `**roo-code-config**` — mounted as VS Code `settings.json`, sets provider profiles as a backup: - -```bash -oc apply -f - <<'EOF' -apiVersion: v1 -kind: ConfigMap -metadata: - name: roo-code-config - namespace: -devspaces - labels: - controller.devfile.io/mount-to-devworkspace: "true" - controller.devfile.io/watch-configmap: "true" - annotations: - controller.devfile.io/mount-as: subpath - controller.devfile.io/mount-path: /checode/checode-linux-libc/User -data: - settings.json: | - { - "roo-cline.enableRooCodeFeatures": true, - "roo-cline.providerProfiles": { - "currentApiConfigName": "llm-d-qwen3-coder", - "apiConfigs": { - "llm-d-qwen3-coder": { - "id": "llm-d-qwen3-coder", - "apiProvider": "openai", - "openAiBaseUrl": "https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1", - "openAiApiKey": "EMPTY", - "openAiModelId": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8", - "openAiStreamingEnabled": false - } - }, - "modeApiConfigs": { - "code": "llm-d-qwen3-coder", - "architect": "llm-d-qwen3-coder", - "ask": "llm-d-qwen3-coder", - "debug": "llm-d-qwen3-coder", - "orchestrator": "llm-d-qwen3-coder" - } - } - } -EOF -``` - -**Critical:** Roo Code **must** use the `openai` provider with `openAiStreamingEnabled: false` (not `openai-native`). See [Configuring vLLM for Roo Code Tool Calling](#configuring-vllm-for-roo-code-tool-calling-qwen3-coder) for the full explanation. - -**14e. Endpoint URL pattern:** - -``` -# Internal (cluster-routed via llm-d Gateway + EPP -- used by all extensions inside Dev Spaces): -https://llm-d-gateway-data-science-gateway-class.llm-d-multi-gpu.svc.cluster.local/v1 - -# External (via llm-d LoadBalancer NLB -- for clients outside the cluster, e.g. local Cline/Continue): -https://a48dea195a3a84e51a93abfdb57f3e1f-837289238.us-east-2.elb.amazonaws.com/v1 -# Paths: /v1/models /v1/chat/completions /v1/completions - -# Direct vLLM (bypasses EPP routing, fallback only): -https://qwen3-coder-fp8-kserve-workload-svc..svc.cluster.local:8000/v1 -``` - -**Important:** All extensions should use the **llm-d Gateway** endpoint. This routes through the EPP for intelligent load balancing (KV-cache aware, prefix-cache affinity, queue-depth balancing) and is the Red Hat recommended architecture for scalable multi-model deployments. The direct workload service endpoint bypasses EPP routing and should only be used as a fallback. - -When the model-serving namespace changes (e.g., from `llm-d-gpu` to `llm-d-multi-gpu`), update the endpoint in the Continue `config.yaml`, the Roo Code ConfigMaps, and the Cline UI settings. - ---- - -## Component Versions - - -| Component | Current Version | Latest Available | Upgrade Action | -| ---------------------------- | ----------------------------------- | ---------------------------- | ------------------------- | -| **Platform** | | | | -| ROSA HCP | 4.21.7 | 4.21.7 | Current | -| OpenShift / Kubernetes | 4.21.7 / v1.34.4 | 4.21.7 / v1.34.4 | Current | -| RHCOS | 9.6 (Plow) | 9.6 | Current | -| CRI-O | 1.34.6-2 | 1.34.6-2 | Current | -| **AI / ML Platform** | | | | -| Red Hat OpenShift AI (RHOAI) | **3.4.0-ea.1** ✓ | 3.4 GA (pending) | **Upgraded** Apr 21, 2026 | -| KServe | 0.15 (RHOAI 3.4 EA1) | 0.15 | Current | -| llm-d EPP | 0.4 (RHOAI 3.4 EA1) | 0.4 | Current | -| Service Mesh 3 | 3.2.0 | **3.3.0** | **Upgrade** | -| **Model Serving** | | | | -| vLLM (CUDA) | **0.14.1+rhai4** ✓ | 0.14.1+rhai4 | **Upgraded** Apr 21, 2026 | -| vLLM (Neuron) | 0.13.0 | 0.13.0 | Current (Neuron image) | -| Model (NVIDIA) | Qwen3-Coder-30B-A3B-Instruct-FP8 | **Qwen3.5-35B-A3B-FP8** | **Upgrade** (see notes) | -| Model (Inferentia) | Qwen3-Coder-30B-A3B-Instruct | Qwen3-Coder-30B-A3B-Instruct | Current | -| Model (Trainium) | Qwen3-Coder-30B-A3B-Instruct | Qwen3-Coder-30B-A3B-Instruct | Current | -| AWS Neuron SDK | 2.28.0 | 2.28.0 | Current | -| **Operators** | | | | -| NVIDIA GPU Operator | 26.3.0 | 26.3.0 | Current | -| AWS Neuron Operator | 1.1.2 | 1.1.2 | Current | -| Neuron DRA Driver | 1.0.0 | 1.0.0 | Current | -| Neuron Helm Chart | 1.5.0 | 1.5.0 | Current | -| cert-manager | 1.18.1 | **1.20.2** | **Upgrade** | -| LeaderWorkerSet Operator | 1.0.0 | **0.8.0** (upstream) | See notes | -| Node Feature Discovery | 4.20.0-202603051149 | 4.20.0 | Current | -| Kernel Module Management | 2.6.0 | **2.4** (OCP 4.21 bundled) | See notes | -| **Developer Tools** | | | | -| OpenShift Dev Spaces | 3.27.0 | 3.27.0 | Current | -| DevWorkspace Operator | 0.40.0 | 0.40.0 | Current | -| Continue Extension | v1.1.80 | v1.1.80 | Current | -| Cline Extension | saoudrizwan.claude-dev | saoudrizwan.claude-dev | Current | -| Roo Code Extension | v3.51+ (RooVeterinaryInc.roo-cline) | v3.51+ | Current | -| **Benchmarking** | | | | -| GuideLLM | 0.5.4 | **0.6.0** | **Upgrade** | -| **AWS Infrastructure** | | | | -| NVIDIA Instance | g6e.2xlarge | g6e.2xlarge | Current | -| Inferentia Instance | inf2.24xlarge | inf2.24xlarge | Current | -| Trainium Instance | trn1.32xlarge | trn1.32xlarge | Current | - - -### Upgrade Notes (April 21, 2026) - -#### 1. RHOAI 3.3 → 3.4.0-ea.1 (vLLM 0.13.0 → 0.14.1) — **COMPLETED April 21, 2026** - -**Status: DONE.** Upgraded via `beta` OLM channel. RHOAI 3.4.0-ea.1 is running, vLLM CUDA image is `registry.redhat.io/rhaii-early-access/vllm-cuda-rhel9` at vLLM **0.14.1+rhai4**. - -**Upgrade path taken:** - -```bash -# Switch subscription to beta channel (carries 3.4.0-ea.1) -oc patch subscription rhods-operator -n openshift-operators \ - --type=merge -p '{"spec":{"channel":"beta"}}' - -# OLM upgrade graph was disconnected (3.3.2 → 3.4.0-ea.1 has no replaces chain) -# Required re-seating the subscription with startingCSV: -oc delete subscription rhods-operator -n openshift-operators -oc delete csv rhods-operator.3.3.2 -n openshift-operators -oc apply -f - <<'EOF' -apiVersion: operators.coreos.com/v1alpha1 -kind: Subscription -metadata: - name: rhods-operator - namespace: openshift-operators -spec: - channel: beta - installPlanApproval: Automatic - name: rhods-operator - source: redhat-operators - sourceNamespace: openshift-marketplace - startingCSV: rhods-operator.3.4.0-ea.1 -EOF - -# Approve the install plan -oc patch installplan install-nvsfz -n openshift-operators --type=merge -p '{"spec":{"approved":true}}' -``` - -**Qwen model restart (required to pick up new vLLM image):** - -```bash -# Scale to 0, then back to 1 (avoids single-GPU RollingUpdate deadlock) -oc patch llminferenceservice qwen3-coder-fp8 -n llm-d-multi-gpu \ - --type=merge -p '{"spec":{"replicas":0}}' -# Wait for pod termination, then: -oc patch llminferenceservice qwen3-coder-fp8 -n llm-d-multi-gpu \ - --type=merge -p '{"spec":{"replicas":1}}' - -# KServe assigned a cached old image to the new pod. Force the new RS with explicit image: -NEW_IMAGE="registry.redhat.io/rhaii-early-access/vllm-cuda-rhel9@sha256:0fdc98bf2fc0f1a552a028d34ef0c0fa9b98e0792772effe5b3410b31486267a" -oc patch llminferenceservice qwen3-coder-fp8 -n llm-d-multi-gpu --type=json -p "[ - {\"op\":\"add\",\"path\":\"/spec/template/containers/0/image\",\"value\":\"$NEW_IMAGE\"} -]" - -# Resolve single-GPU rollout deadlock: -oc scale rs qwen3-coder-fp8-kserve-5cb84c6c75 --replicas=0 -n llm-d-multi-gpu -oc delete pod qwen3-coder-fp8-kserve-5cb84c6c75-qv4rs -n llm-d-multi-gpu --grace-period=0 --force -``` - -**What changed with vLLM 0.14.1+rhai4:** - -- Async scheduling enabled by default (overlaps engine core scheduling with GPU execution) -- `--max-model-len auto` to automatically fit context length to available GPU memory -- gRPC server entrypoint as alternative to REST API -- Qwen3.5 MoE architecture support (`Qwen3_5ForCausalLM`) -- enables **Qwen3.5-35B-A3B-FP8** on L40S -- Streaming tool-call parser fixes (resolves the vLLM 0.13.x `qwen3_coder` streaming bug) -- Gemma 4 and Nemotron 3 Nano are now deployable - -**What vLLM 0.14.1 does NOT support:** - -- Qwen3.6 DeltaNet architecture (requires vLLM 0.19+) -- TurboQuant KV cache (requires vLLM 0.16+) - -**Known Issue: Gateway namespace allowance (RHOAI 3.4 EA1 regression)** - -After the upgrade, the RHOAI 3.4 operator reconciles the `data-science-gateway` in `openshift-ingress` and restricts `allowedRoutes` to only `openshift-ingress` and `redhat-ods-applications`. This causes `HTTPRoutesReady: False` on `LLMInferenceService` objects in `llm-d-multi-gpu`. - -**Impact:** Only affects external ROSA NLB access to models. The internal ClusterIP URL used by Roo Code, Cline, and Continue (`https://llm-d-gateway-data-science-gateway-class.llm-d-multi-gpu.svc.cluster.local/v1`) is unaffected -- all DevSpaces-based AI assistant traffic continues to work normally. - -**Workaround (operator reconciles it back immediately, not persistent):** - -```bash -oc patch gateway data-science-gateway -n openshift-ingress --type=json -p '[ - {"op":"replace","path":"/spec/listeners/0/allowedRoutes/namespaces/selector/matchExpressions/0/values", - "value":["openshift-ingress","redhat-ods-applications","llm-d-multi-gpu"]} -]' -``` - -Permanent fix requires RHOAI 3.4 GA which is expected to address this EA1 regression. Raise with Red Hat support if needed. - -#### 2. Service Mesh 3.2.0 → 3.3.0 - -**Safe to upgrade now.** Service Mesh 3.3 adds: - -- Post-quantum cryptography (PQC) with hybrid X25519MLKEM768 key exchange -- Istio 1.28 with Gateway API v1.4 and BackendTLSPolicy v1 -- Kiali 2.22 -- Ambient mode FIPS support - -No breaking changes for the current llm-d + ext_proc gateway configuration. Upgrade via: - -```bash -# Update the ServiceMeshControlPlane channel in OperatorHub, then: -oc get smcp -A # verify the control plane reconciles to 3.3 -``` - -#### 3. cert-manager 1.18.1 → 1.20.2 - -**Safe to upgrade.** cert-manager 1.20.x adds ListenerSet alpha support and Azure Private DNS support. No breaking changes. Upgrade via OperatorHub subscription channel update. - -#### 4. GuideLLM 0.5.4 → 0.6.0 - -**Upgrade when benchmarking.** GuideLLM 0.6.0 adds multi-turn support, Responses API support, and vLLM Python backend. Install with: - -```bash -pip install guidellm[recommended]==0.6.0 -``` - -#### 5. Model Upgrade Path: Qwen3.5-35B-A3B-FP8 (requires RHOAI 3.4 GA) - -Once RHOAI ships vLLM 0.14.1 at GA, **Qwen3.5-35B-A3B-FP8** becomes deployable on L40S: - -- 36B total / 3B active params (MoE), same active count as current model -- FP8 weights ~37.5 GB (fits in 48 GB L40S with ~8 GB for KV cache) -- Multimodal (image + text) -- can process code screenshots and diagrams -- Same `--tool-call-parser qwen3_coder --reasoning-parser qwen3` flags -- Apache 2.0 license - -**Future (vLLM 0.19+):** Qwen3.6-35B-A3B (73.4% SWE-bench Verified) uses DeltaNet hybrid architecture requiring vLLM 0.19. RHOAI will need to ship a 0.19+ build before this model can be deployed. - -#### 6. LeaderWorkerSet and KMM Version Notes - -The "LeaderWorkerSet Operator 1.0.0" in this table refers to the RHOAI-managed CRD version, not upstream LWS (which is at v0.8.0). KMM 2.6.0 is the version installed via the Neuron Operator Helm chart; the OCP 4.21 bundled KMM is 2.4. Both are compatible; the Neuron Operator manages its own KMM lifecycle. - ---- - -## Model Caching and Startup Optimization Analysis - -Cold-start times are dominated by three phases: **model weight download** (~~10-15 min for 57 GB from HuggingFace), **Neuron NEFF compilation** (30-60 min for new bucket configurations), and **vLLM torch.compile** (~~2-5 min on NVIDIA). This section evaluates strategies to eliminate or reduce these delays across clusters and test cycles. - -### Current Cold-Start Time Budget - - -| Phase | NVIDIA (FP8, 16 GB) | Inferentia (BF16, 57 GB) | -| ----------------------------------------- | --------------------------------------------------------- | --------------------------------------- | -| Model download (HuggingFace → pod) | ~4-8 min (via KServe storage-initializer, emptyDir) | ~8 min (via `HF_HOME` on PVC, cached) | -| Model download (PVC cache hit) | **~0.2 sec** (storage-initializer detects existing files) | **~0.2 sec** (HF cache hit) | -| NEFF compilation | N/A | **30-60 min** (new buckets, cold cache) | -| torch.compile / model load | ~3-5 min | N/A (Neuron runtime) | -| **Total cold start (no PVC)** | **~7-13 min** | **38-68 min** | -| **Total cold start (first PVC download)** | **~7-13 min** | **38-68 min** | -| **Warm start (PVC cache hit)** | **~3-5 min** (GPU load only) | **~10 sec** (NEFF cache hit) | - - -### Strategy Comparison - - -| Approach | Cold Start | Warm Start | NEFF Support | Portability | Cost | Complexity | -| ------------------------------------------- | ----------------------------------------------- | ----------------------------------- | ----------------------------------------- | -------------------------------- | ---------------------------------- | ------------------------------ | -| **EBS PVC (current)** | Full download + compile | ~10s (Inferentia), ~3 min (NVIDIA) | Excellent -- cached on disk | Same AZ only | EBS gp3 ($0.08/GB/mo) | Low | -| **EBS Snapshots** | PVC create from snapshot: instant API, lazy I/O | Same as PVC | Excellent -- entire tree snapshotted | Same region, cross-AZ with FSR | Snapshot: $0.05/GB/mo + FSR fee | Medium | -| **OCI Model Images (`oci://`)** | Node pull: ~5-10 min (cached layers skip) | Near-instant (image cached on node) | Possible -- bake NEFF into image | Any cluster with registry access | Registry storage + egress | Medium-High | -| **S3 Model Mirror** | Download from S3: ~5 min (same region) | Same as first pull (no node cache) | Yes -- `NEURON_COMPILE_CACHE_URL=s3://` | Cross-region with replication | S3: $0.023/GB/mo | Medium | -| **RHOAI Model Registry** | Same as underlying storage (S3/OCI) | Same | Indirect -- registry stores metadata only | High (metadata portable) | Registry: minimal; storage: varies | Medium | -| **Pre-built Image (model + NEFF baked in)** | Image pull only: ~10-15 min | Near-instant (image cached) | Excellent -- but fragile on SDK mismatch | Any cluster with registry access | Largest images (~80+ GB) | High (rebuild per SDK version) | - - -### Recommended Strategy by Use Case - -**1. For repeated testing on the same cluster (current scenario):** - -**EBS PVC persistence (implemented for both NVIDIA and Inferentia)** is optimal. On NVIDIA, the `kserve-provision-location` volume in the LLMInferenceService template is overridden with a PVC, so the KServe storage-initializer downloads weights once and skips on subsequent restarts (0.2s vs 4-8 min). On Inferentia, NEFF cache and HF model cache on EBS gp3 PVCs mean warm restarts take ~10 seconds. NVIDIA warm starts take ~3-5 minutes (GPU weight loading only). **Do not delete PVCs when scaling down nodes.** - -**2. For provisioning new clusters or cross-cluster portability:** - -**EBS Snapshots + S3 mirror** (two-tier approach): - -```bash -# Tier 1: Snapshot the warm PVC (captures weights + NEFF artifacts) -oc apply -f - <<'EOF' -apiVersion: snapshot.storage.k8s.io/v1 -kind: VolumeSnapshot -metadata: - name: inferentia-cache-snapshot-16k - namespace: llm-d-multi-gpu -spec: - volumeSnapshotClassName: ebs-csi-snapclass - source: - persistentVolumeClaimName: inferentia-model-cache -EOF - -# Tier 2: Create PVC from snapshot on new cluster (same region) -oc apply -f - <<'EOF' -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: inferentia-model-cache - namespace: llm-d-multi-gpu -spec: - accessModes: [ReadWriteOnce] - storageClassName: gp3-csi - resources: - requests: - storage: 200Gi - dataSource: - name: inferentia-cache-snapshot-16k - kind: VolumeSnapshot - apiGroup: snapshot.storage.k8s.io -EOF -``` - -Enable **Fast Snapshot Restore (FSR)** for the AZ to avoid lazy-load latency on first read: - -```bash -aws ec2 enable-fast-snapshot-restores \ - --availability-zones us-east-2a \ - --source-snapshot-ids -``` - -**3. For cross-region / multi-cluster enterprise deployment:** - -**OCI model images + S3 NEFF cache** (two-part approach): - -- **Model weights:** Package into an OCI image, push to ECR/Quay. Use `storageUri: oci://` in KServe (Modelcar sidecar, no download step). ECR max layer size is ~~52 GB, so split FP8 model (~~16 GB) into 1 layer or BF16 (~57 GB) into 2 layers. -- **NEFF artifacts:** Store in S3 with `NEURON_COMPILE_CACHE_URL=s3:///neuron-cache`. Neuron SDK natively supports S3 read/write for cache. First pod in a new cluster still compiles (populating S3), but all subsequent pods in any cluster sharing that bucket skip compilation entirely. - -### OpenShift AI Model Registry Role - -The RHOAI Model Registry (Model Catalog in 3.3) is a **governance and metadata system**, not a weight storage backend. It stores: - -- Model name, version, description, owner -- Artifact URI pointers (to S3, OCI, or other storage) -- Deployment history and lineage - -It does **not** store model weights or NEFF artifacts directly. Use it to: - -- Standardize the canonical URI for `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` (pointing to your S3 mirror or OCI image) -- Track which version is deployed to which cluster -- Enforce approval workflows before production deployment - -The registry connects to KServe via `storageUri` resolution -- register the model with an S3 or OCI URI, and KServe's storage initializer handles the actual download. +## Table of Contents -### Neuron NEFF Cache Specifics +1. [Detailed description](#detailed-description) + - [The challenge](#the-challenge) + - [Our solution](#our-solution) + - [Features](#features) + - [Solution stack](#solution-stack) + - [Architecture](#architecture) +2. [Requirements](#requirements) +3. [Deploy](#deploy) + - [Option 1: ROSA (AWS)](#option-1-rosa-aws) + - [Option 2: ARO (Azure)](#option-2-aro-azure) + - [Option 3: Existing OpenShift](#option-3-existing-openshift) + - [Verify](#verify) +4. [Documentation](#documentation) +5. [References](#references) +6. [Tags](#tags) +## Detailed description -| Property | Value | -| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------ | -| Cache location env var | `NEURON_COMPILE_CACHE_URL` | -| Supported backends | Local filesystem, `s3://` URI | -| Cache key components | Model architecture + TP degree + `seq_len` + bucket shapes + Neuron SDK version + compiler flags | -| Invalidation triggers | Any change to the above (new SDK version, different `max_model_len`, different buckets) | -| Estimated NEFF size (Qwen3-Coder-30B, 7 buckets, TP=8) | ~25-40 GB (measure with `du -sh /cache/neuron-cache/` after compilation) | +### The challenge +Teams want IDE-integrated AI coding help without sending proprietary source to public model APIs. They also need governance: who can call the model, which model is served, and where data resides. -**Packaging NEFF into a container image** is technically possible but **fragile**: NEFF artifacts are tightly coupled to the Neuron SDK version. If the vLLM-Neuron base image is updated (e.g., from SDK 2.28 to 2.29), all pre-built NEFFs become invalid and must be recompiled. The S3 cache approach is more maintainable because the cache self-invalidates on version mismatch and re-populates automatically. +### Our solution -### Action Items +This quickstart deploys: -1. **Immediate:** After current compilation completes, snapshot the Inferentia PVC (`inferentia-model-cache`) to preserve the 16,384 NEFF artifacts -2. **Short-term:** Mirror model weights to an S3 bucket in us-east-2; update `HF_HOME` or KServe `storageUri` to point there instead of HuggingFace -3. **Medium-term:** Evaluate OCI model images for NVIDIA (smaller FP8 model fits in 1 ECR layer); keep Inferentia on PVC + S3 NEFF cache -4. **Long-term:** Use RHOAI Model Registry to catalog approved model versions with standardized URIs +- **OpenShift Dev Spaces** workspaces with Continue, Cline, Roo Code (and optionally OpenCode) +- **On-cluster inference** with vLLM behind **llm-d** intelligent routing +- An optional **RHCL AI Gateway** front door with per-developer API keys +- GitOps Helm charts (ROSA/ARO) or Helm-only install on an existing cluster ---- +### Features -## KV Cache Optimization: FP8 Quantization +- Source and inference stay inside your OpenShift network boundary when configured that way +- OpenAI-compatible `/v1` API for IDE extensions +- Prefix-cache–aware routing via llm-d Endpoint Picker (EPP) +- Multi-developer namespaces with ConfigMap-prewired extensions +- Optional observability (Grafana; Langfuse/OTel), guardrails, and MCP -### Background +### Solution stack -The KV (Key-Value) cache stores attention state for all processed tokens and is the primary consumer of GPU memory beyond model weights. For Qwen3-Coder-30B-A3B (48 layers, 4 KV heads via GQA, 128 head_dim), the per-token KV cache footprint is: +| Layer | Technology | +|-------|------------| +| IDE | OpenShift Dev Spaces | +| Auth front door | RHCL AI Gateway (`pca-ai-gateway`) | +| Routing | llm-d + Gateway API + EPP | +| Serving | vLLM / KServe `LLMInferenceService` | +| Platform | Red Hat OpenShift AI, NVIDIA GPU Operator, NFD | +| Delivery | Unified Helm charts in `charts/` (ArgoCD or `make`) | -``` -2 (K+V) × 48 layers × 4 KV heads × 128 head_dim × bytes_per_element -``` +### Architecture +Developers → Dev Spaces → RHCL AI Gateway → llm-d → EPP → vLLM on NVIDIA GPU. -| KV Cache Dtype | Bytes/Element | Per-Token Size | Compression vs BF16 | -| -------------- | ------------- | ----------------- | ------------------- | -| BF16 (default) | 2 | 98,304 B (~96 KB) | 1x baseline | -| **FP8 (E4M3)** | 1 | 49,152 B (~48 KB) | **2x** | +![Inference traffic flow](docs/images/architecture-traffic-flow.svg) +Full diagrams, chart waves, and request-path detail: **[docs/architecture.md](docs/architecture.md)**. -On the L40S (48 GB VRAM) with ~16 GB used by FP8 model weights and ~2-4 GB for CUDA overhead, approximately 28-30 GB is available for KV cache. FP8 KV cache doubles the token budget from ~300K tokens to ~600K tokens, enabling either 2x more concurrent requests at the same context length or the same concurrency at 2x longer context. +## Requirements -### Implementation (Deployed) +| Area | Summary | +|------|---------| +| Software | OpenShift 4.18+, RHOAI 3.x, Dev Spaces, GPU Operator + NFD, `oc` / `helm` (plus Terraform + cloud CLIs for ROSA/ARO) | +| Hardware | At least one NVIDIA GPU sized for your model (see [docs/benchmarks.md](docs/benchmarks.md)) | +| Permissions | `cluster-admin` (or equivalent) for operators / GitOps; cloud admin for from-scratch ROSA/ARO | +| Secrets | Hugging Face token for model download | -Added `--kv-cache-dtype fp8` to `VLLM_ADDITIONAL_ARGS` in the `LLMInferenceService` spec. This quantizes the KV cache from BF16 to FP8 E4M3 format at store time with negligible quality impact. +Details: **[docs/requirements.md](docs/requirements.md)**. -**Observed result:** `GPU KV cache size: 205,888 tokens` with 9.42 GiB available KV cache memory at `gpu-memory-utilization=0.90`. +## Deploy -**Key configuration decisions:** +Choose one path. Do not mix ArgoCD GitOps (ROSA/ARO) with the existing-OpenShift Helm flow on the same cluster unless you know how ownership overlaps. -- **No `--calculate-kv-scales`** -- Default scales (1.0) are used intentionally. There is a [known vLLM bug](https://github.com/vllm-project/vllm/issues/37554) where `--calculate-kv-scales` causes catastrophic output corruption on hybrid GDN+Attention architectures in the Qwen3 family. Default scales work correctly because BF16 KV activations are well within FP8 E4M3's representable range (±448). -- `**gpu-memory-utilization` reduced to 0.90** -- FP8 KV cache doubles the number of allocated cache blocks. At 0.95, the vLLM sampler warmup OOMs during `topk_topp` sorting (needs ~446 MB for 256 dummy requests, only 291 MB free). 0.90 provides sufficient headroom. -- **Compatible with prefix caching** -- FP8 KV cache works with `--enable-prefix-caching` (already enabled). -- **No model re-download or recalibration** -- This is a runtime-only change; the model checkpoint is unchanged. +### Option 1: ROSA (AWS) -**Rollout procedure:** +Terraform provisions ROSA HCP and a GPU pool; ArgoCD syncs `charts/` with `values-rosa.yaml`. -After patching the `LLMInferenceService`, a `RollingUpdate` creates a new pod that hangs in `Pending` because the old pod still holds the GPU. Resolve the deadlock by scaling down the old ReplicaSet and force-deleting the old pod: +→ **[PCA_Deployment_ROSA/README.md](PCA_Deployment_ROSA/README.md)** -```bash -# 1. Apply the config change -oc patch llminferenceservice qwen3-coder-fp8 -n ai-serving --type=json -p '[ - {"op":"replace","path":"/spec/template/containers/0/env/0/value", - "value":"--disable-uvicorn-access-log --max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --kv-cache-dtype fp8"} -]' +### Option 2: ARO (Azure) -# 2. Identify old and new ReplicaSets -oc get rs -n ai-serving -l app.kubernetes.io/name=qwen3-coder-fp8 +Terraform provisions ARO and a GPU MachineSet; ArgoCD syncs `charts/` with `values-aro.yaml`. -# 3. Scale down the OLD ReplicaSet and delete its pod to free the GPU -oc scale rs --replicas=0 -n ai-serving -oc delete pod -n ai-serving --grace-period=0 --force +→ **[PCA_Deployment_ARO/README.md](PCA_Deployment_ARO/README.md)** -# 4. The new pod will schedule onto the freed GPU and start loading the model -# Total startup time: ~5-6 min (model load + CUDA graph capture + warmup) -``` +### Option 3: Existing OpenShift -**Verification:** +Cluster already has RHOAI, GPU Operator, and Dev Spaces. Install RHCL first, then: ```bash -# Confirm FP8 KV cache in vLLM startup logs -oc logs -n ai-serving -l app.kubernetes.io/name=qwen3-coder-fp8 | grep -i "kv.cache\|cache_dtype" - -# Check GPU blocks allocated (should approximately double vs BF16 baseline) -curl -sk https://qwen3-coder-fp8-kserve-workload-svc.ai-serving.svc.cluster.local:8000/metrics \ - | grep gpu_cache_usage -``` - -### Future: TurboQuant (3-5x Compression) - -[TurboQuant](https://turbo-quant.com/en) (Google Research, ICLR 2026) achieves 2.6-4.9x KV cache compression using PolarQuant (Walsh-Hadamard rotation + Lloyd-Max scalar quantization). It was [merged into upstream vLLM](https://github.com/vllm-project/vllm/pull/38479) on April 15, 2026 and will be available in vLLM 0.16+. - - -| TurboQuant Preset | Compression | Quality (GSM8K) | NIAH | Usage | -| -------------------- | ----------- | --------------- | ---- | ------------------------------------- | -| `turboquant_k8v4` | 2.6x | 96% of baseline | 100% | `--kv-cache-dtype turboquant_k8v4` | -| `turboquant_4bit_nc` | 3.8x | 93% of baseline | 100% | `--kv-cache-dtype turboquant_4bit_nc` | -| `turboquant_3bit_nc` | 4.9x | 80% of baseline | 100% | `--kv-cache-dtype turboquant_3bit_nc` | - - -TurboQuant is not available in vLLM 0.13.0+rhai11 (current RHOAI build). When RHOAI ships a vLLM 0.16+ build, switching from `fp8` to `turboquant_k8v4` is a single-flag change for an additional 30% KV cache compression beyond FP8, with no offline calibration or model changes required. - ---- - -## Project Considerations - -- **Version currency** -- Prefer latest **Red Hat-supported** stable releases for all components. **llm-d-playbook** examples may lag the cluster; always verify **CRD schemas** and **image tags** against the live cluster. -- **vLLM image** -- Deployments use the **Red Hat AI** build (`0.13.0+rhai11`), not the community `vllm/vllm-openai` image. The `**LLMInferenceService`** selects the appropriate image when used as designed. -- **Security** -- Model traffic uses **TLS** (pod-to-pod via KServe-related certs, external paths via **Service Mesh** gateway and **AWS ELB** as configured). -- **Service Mesh version** -- Cluster runs Service Mesh **3.2.0**, but **3.3.1** is available on the `stable` channel and **3.2.3** is the latest patch for the `stable-3.2` channel. Upgrading is recommended for improved Gateway API Inference Extension support and ext_proc stability. -- **DSC Configuration** -- The `rawDeploymentServiceConfig` in the `DataScienceCluster` **must be set to `Headed`** (not the default `Headless`) for llm-d load balancing to function. The `Headless` mode is only intended for WatsonX integrations. -- **TLS Secret for Gateway** -- The `maas-default-gateway` requires a TLS secret (`data-science-gatewayconfig-tls`) in `openshift-ingress`. If this secret is missing, the gateway envoy will have no HTTPS listener and all inference traffic will fail silently. -- **Neuron-scheduler / OVN incompatibility** -- On ROSA HCP, the AWS neuron-scheduler's annotation updates overwrite OVN network annotations, preventing multi-core Inferentia pods from starting. **Verified fix**: MutatingAdmissionWebhook preserves OVN annotations (see Appendix B). **Long-term**: Neuron DRA driver eliminates the neuron-scheduler entirely (see Appendix A). -- **NFD vendor labels** -- The NVIDIA GPU Operator may not detect GPU nodes if only device-specific PCI labels are present. Manually add `feature.node.kubernetes.io/pci-10de.present=true` if needed. -- **MoE prefix caching on Neuron** -- The NxD MoE engine does not support prefix caching. Use dense models on Inferentia when prefix-cache-aware routing is required. -- **Neuron NEFF cache on EBS PVC (MANDATORY)** -- Set `NEURON_COMPILE_CACHE_URL` **and** `HF_HOME` to paths on an EBS-backed PVC (e.g., `/models/neuron-cache` and `/models/hf-cache`), **never** to ephemeral storage (`/tmp`, `emptyDir`). First-time Neuron compilation for Qwen3-Coder-30B takes **30-45 minutes**; warm-cache restarts take ~10 seconds. Losing NEFF artifacts on pod restart wastes 30-45 minutes of compilation time per pod. KServe 0.14+ (RHOAI 3.3) supports read/write PVC access in `RawDeployment` mode. -- **Container memory right-sizing for Neuron compilation (MANDATORY)** -- Neuron compilation of Qwen3-Coder-30B MoE peaks at ~120 GB RSS (57 GB BF16 model weights + 60+ GB compiler workspace during bucket compilation). A 64Gi limit causes OOMKill during weight loading; a 128Gi limit causes OOMKill during bucket compilation. **Set limits to 200Gi minimum** for cold-cache compilation. Steady-state serving uses only ~15-20 GB. Always patch the `InferenceService` (not the Deployment), as KServe reconciles the Deployment from the InferenceService spec. -- **SELinux and shared PVCs** -- When multiple pods share a `ReadWriteOnce` PVC on OpenShift, set `seLinuxOptions.level` to the same value (e.g., `s0:c0,c0`) on all pods. Without matching MCS labels, SELinux denies cross-pod file access regardless of Unix permissions. -- **Heterogeneous llm-d routing** -- Unified routing across NVIDIA and Inferentia requires: (1) same namespace for all backends with `nodeSelector`-based scheduling, (2) **taints on accelerator machine pools** (`nvidia.com/gpu=present:NoSchedule` for GPU, `aws.amazon.com/neuroncore=present:NoSchedule` for Inferentia) with matching tolerations on model deployments — without taints, platform pods consume GPU node CPU and block model scheduling, (3) identical `--served-model-name`, (4) HTTPS on all endpoints via vLLM native `--ssl-certfile`/`--ssl-keyfile` flags (EPP `--model-server-metrics-scheme` is global), (5) separate PVCs per accelerator type, (6) matching `InferencePool` labels on all pods. The prefix-cache scorer naturally favors GPU replicas (which support prefix caching), while Inferentia handles overflow. - ---- - -## Enterprise Customization for Implementation Services - -This section describes the customization work that Red Hat Professional Services (or a qualified partner) performs when delivering the Private AI Code Assistant as a managed implementation. The two primary customization tracks are **team-practice alignment** and **quality-gate integration**. Both are delivered as configuration, not code changes — the core platform stack remains fully supported and upgradeable. - ---- - -### Track 1: Customizing for Customer Development Practices - -A generic code assistant gives generic output. The value of a privately hosted model increases significantly when the model and extensions are tuned to reflect the specific languages, frameworks, tooling conventions, and architectural patterns that the customer's teams actually use. The following customizations are typically delivered as part of an implementation engagement. - -#### 1.1 System Prompt and Context Engineering - -The AI extensions (Continue, Cline, Roo Code) all support custom system prompts that are injected before every conversation. These are configured via the Dev Spaces ConfigMap approach (Step 15), meaning they are centrally managed by the platform administrator and applied automatically to every developer workspace without manual action. - -**Typical system prompt customisations:** - -- **Language and framework context** — explicitly declare the primary stack: *"This is a Java 21 / Spring Boot 3.3 microservices project. Use constructor injection, not field injection. Prefer records for DTOs. Use OpenAPI 3.1 annotations on all REST controllers."* -- **Internal library preferences** — reference internal SDKs, platform libraries, or banned dependencies: *"Use the internal `com.acme.security:jwt-utils` library for token handling — never use `io.jsonwebtoken` directly."* -- **Architectural patterns** — enforce agreed patterns: *"All service classes must implement a domain interface. Repository access is only permitted from the service layer, never from controllers or event handlers."* -- **Naming conventions and style** — package structures, method naming, annotation usage, logging standards: *"Use SLF4J with `@Slf4j`. Log at DEBUG for internal state, INFO for business events, WARN for recoverable errors, ERROR only for exceptions that bubble to the boundary."* -- **Test framework standards** — preferred test libraries and patterns: *"Write unit tests with JUnit 5 + Mockito. Use AssertJ for assertions. Test class names end in `Test`. Integration tests end in `IT` and are annotated with `@SpringBootTest(webEnvironment = RANDOM_PORT)`."* - -#### 1.2 Per-Project Context via Rules Files - -All three extensions support project-scoped instructions that override or supplement the global system prompt when a developer is working inside a specific repository: - -| Extension | Rules file | Scope | -|-----------|-----------|-------| -| Continue | `.continuerules` (root of repo) | Per-repository system prompt append | -| Cline | `.clinerules` | Per-repository instruction file | -| Roo Code | `.roo/rules/`, `.roo/rules-{mode}/` | Per-mode rules (Code, Architect, Debug, etc.) | - -These files are committed to the customer's source repositories and version-controlled alongside the code. This means the AI's context evolves with the codebase — as new patterns are adopted, the rules file is updated in a PR, reviewed, and applied automatically to all developers on the next workspace restart. - -**Example `.roo/rules/` structure for a polyglot microservices org:** - -``` -.roo/ - rules/ ← applied to all modes - 000-stack.md ← Java 21, Spring Boot 3.3, PostgreSQL 16, Kafka 3.7 - 001-patterns.md ← hexagonal architecture, port/adapter naming - 002-banned.md ← forbidden libraries and patterns - rules-code/ ← applied only in Code mode - java-style.md - sql-conventions.md - rules-architect/ ← applied only in Architect mode - adr-template.md - domain-model.md -``` - -#### 1.3 DevWorkspace Templates per Technology Stack +# AI serving once +make ai-serving-deploy-existing-openshift HF_TOKEN=hf_xxx -The Dev Spaces `DevWorkspace` CR (Step 15) defines which container image, tools, runtimes, and environment variables the workspace pod runs. Customers with multiple tech stacks are given dedicated workspace templates: - -| Team / Stack | DevWorkspace image | Pre-installed tooling | -|---|---|---| -| Java / Spring | UBI 9 + JDK 21 + Maven 3.9 | `mvn`, `java`, `oc`, `git`, `stern` | -| Python / FastAPI | UBI 9 + Python 3.12 | `uv`, `ruff`, `pytest`, `oc` | -| Node.js / NestJS | UBI 9 + Node 22 | `npm`, `pnpm`, `eslint`, `jest`, `oc` | -| Go | UBI 9 + Go 1.22 | `go`, `golangci-lint`, `oc` | -| IaC / Platform Eng. | UBI 9 + toolbox | `terraform`, `ansible`, `oc`, `helm`, `kustomize` | - -The platform administrator maintains a catalog of `DevWorkspace` templates in a central ConfigMap. Developers select their workspace template from the Dev Spaces Dashboard — no local setup required. - -#### 1.4 MCP Server Integration with Internal Systems - -The Model Context Protocol (MCP) allows AI extensions to query live data sources as part of a conversation. For customers with internal knowledge systems, the following MCP server integrations are commonly delivered: - -- **Internal Confluence / Notion** — the AI can retrieve architecture decision records (ADRs), runbooks, and design documents when asked to generate code that must conform to documented patterns. -- **Internal OpenAPI catalogs** — the AI has live access to the customer's internal service catalog, enabling accurate client code generation without hallucinating endpoint paths or schemas. -- **Jira / issue tracker** — the AI can read the acceptance criteria and DoD from the linked issue, grounding its code generation in the actual requirements. -- **Internal GitLab / GitHub** — code search across the customer's repos surfaces existing implementations of patterns, reducing duplication. - -MCP server connections are configured in the `roo-code-provider-config` ConfigMap (for Roo Code) or `.continue/config.yaml` (for Continue), and are distributed to all workspace pods centrally — developers get access to internal context without any local configuration. - ---- - -### Track 2: SonarQube-Aware Code Generation - -SonarQube (or SonarCloud) is widely used to enforce code quality and security standards at the gate before code reaches test and production environments. A common friction point is that AI-generated code, while functionally correct, may carry code smells, security hotspots, or vulnerability patterns that fail the SonarQube quality gate — shifting remediation effort to the end of the development cycle when it is most expensive to fix. - -This track configures the AI assistant to **apply SonarQube rules at generation time**, so that the code produced is already aligned with the quality gate requirements before the developer commits it. - -#### 2.1 Embedding SonarQube Rule Profiles in System Prompts - -The primary mechanism is injecting the customer's active SonarQube quality profile as structured context in the system prompt. The platform administrator exports the quality profile rules from SonarQube (`Administration → Quality Profiles → Export`) and converts the relevant rules into plain-language instructions that are added to the global or per-project system prompt. - -**Example — Java security rules (derived from SonarQube Java Security Hotspot profile):** - -``` -## Code Quality Requirements (SonarQube Profile: ACME-Java-Strict) - -Security: -- Never use java.util.Random for security-sensitive operations — use java.security.SecureRandom. -- Do not log request parameters, headers, or authentication tokens at any log level (SQUID:S2068, S5145). -- All SQL queries must use parameterized statements or JPA criteria. String concatenation into SQL is a blocker violation (SQUID:S2077). -- Validate and sanitize all inputs received from HTTP requests before use (OWASP A03). -- Do not suppress FindBugs / SpotBugs annotations without a documented justification comment. - -Reliability: -- Resources opened in try blocks must be closed — use try-with-resources (SQUID:S2093). -- Do not catch generic Exception or Throwable except at application boundaries (SQUID:S2221). -- Null checks must be performed before dereferencing objects returned from external calls (SQUID:S2259). - -Maintainability: -- Methods must not exceed 30 lines. Extract helper methods for logical sub-steps. -- Cyclomatic complexity must stay below 10 per method (SQUID:S1541). -- Do not duplicate code blocks of 5 or more lines — extract to a shared utility. -- Magic numbers must be assigned to a named constant before use. -``` - -When this profile is in the system prompt, every code completion and agent task is evaluated by the model against these constraints before the output is presented to the developer. - -#### 2.2 SonarQube Findings as Live AI Context - -For codebases that already have accumulated SonarQube debt, the implementation delivers an MCP server that queries the customer's SonarQube instance in real time. When a developer asks the AI to fix a class or method, the MCP server fetches the open issues for that file from SonarQube and injects them as context: - -``` -[SonarQube context — src/main/java/com/acme/OrderService.java] -Open issues: - LINE 47 BLOCKER sql-injection S2077 String concatenation in SQL query - LINE 89 CRITICAL security-hotspot S5344 HTTP response splitting risk - LINE 112 MAJOR code-smell S1541 Method cyclomatic complexity is 14 (limit 10) - LINE 134 MINOR code-smell S1192 String literal duplicated 4 times +# Each developer workspace +make devspace-deploy-existing-openshift DEV_NAMESPACE= +# 2nd+ developers: HELM_ARGS='--set devspacesGlobalConfig.enabled=false' ``` -The model sees these issues as part of the conversation and addresses them alongside the developer's explicit request, without requiring the developer to context-switch to the SonarQube dashboard. This substantially reduces the number of SonarQube findings that survive to the CI gate. +→ **[deploy_existing_openshift/README.md](deploy_existing_openshift/README.md)** -#### 2.3 Pre-Commit and CI Integration +### Verify -For teams using the Dev Spaces workspace as their primary development environment, the `postStart` lifecycle hook in the `DevWorkspace` CR can be extended to run a SonarQube scanner (`sonar-scanner` CLI) against changed files before a commit is made. Findings above a configured severity threshold are surfaced directly in the terminal, paired with an AI-generated suggested fix via a simple shell wrapper: +After deploy, run cluster smoke tests (not CI): ```bash -# sonar-precommit-hook.sh (injected via DevWorkspace postStart or git hook) -sonar-scanner --define sonar.inclusions="$(git diff --name-only HEAD)" \ - --define sonar.host.url="${SONAR_HOST_URL}" \ - --define sonar.token="${SONAR_TOKEN}" - -# If blockers found, automatically open Roo Code with findings as context -if [ "$SONAR_BLOCKER_COUNT" -gt 0 ]; then - echo "SonarQube found $SONAR_BLOCKER_COUNT blocker(s). Roo Code context loaded — ask 'Fix SonarQube blockers' to address them." -fi -``` - -#### 2.4 Language-Specific SonarQube Customization Matrix - -The following matrix summarises the per-language customizations typically delivered as part of a SonarQube-aligned engagement: - -| Language | Common SonarQube Rules Addressed | AI Customization Applied | -|---|---|---| -| **Java** | SQL injection (S2077), log injection (S5145), resource leaks (S2093), complexity (S1541), null dereference (S2259) | System prompt: enforce parameterized queries, try-with-resources, method length limits | -| **Python** | Hardcoded credentials (S2068), broad except clauses (S110), SQL injection (S3649), unused imports (S1128) | System prompt: enforce `secrets` module, specific exception types, SQLAlchemy ORM patterns | -| **JavaScript / TypeScript** | Prototype pollution (S6268), XSS (S5728), `eval()` use (S1523), promise rejection (S4822), `==` vs `===` (S1440) | System prompt: enforce strict equality, ban `eval`, enforce `await/catch` patterns | -| **Go** | Error return ignored (S1005), race conditions (S2696), hardcoded secrets (S2068), HTTP redirect (S5144) | System prompt: always handle error returns, use `sync` primitives, validate redirect URLs | -| **Infrastructure (Terraform / YAML)** | Overly permissive IAM (S6302), unencrypted storage (S6245), public S3 buckets (S6249), missing TLS (S4423) | System prompt: enforce least-privilege IAM, encryption-at-rest, deny public bucket ACLs | - -#### 2.5 Continuous Improvement Loop - -The SonarQube customization is treated as a living configuration: - -1. **Baseline measurement** — before deployment, the SonarQube quality gate pass rate and mean issue density per PR are recorded. -2. **Monthly review** — the platform administrator exports the top recurring issue types from SonarQube and updates the system prompt and MCP context rules to address the highest-frequency findings. -3. **Quality gate trend reporting** — the reduction in new SonarQube issues per PR over time is tracked as the primary success metric for the implementation engagement. -4. **Profile drift alerts** — when the SonarQube quality profile is updated by the security team, an automated job (delivered as an OpenShift CronJob) re-exports the profile and opens a PR against the platform's ConfigMap repository, ensuring the AI context stays in sync with the enforced rules. - ---- - -## Appendix A: Neuron-Scheduler / OVN-Kubernetes Annotation Conflict - -### Problem - -On ROSA HCP with OVN-Kubernetes CNI, pods using `schedulerName: neuron-scheduler` get stuck in `ContainerCreating` with: - -``` -FailedCreatePodSandBox: failed to get pod annotation: timed out waiting for annotations: context deadline exceeded -``` - -**Root cause:** The `neuron-scheduler` writes device-allocation annotations (`AWS_NEURON_IDS`, `NEURON_ALLOCATED`, `NEURON_ALLOC_TIME`) using a **full annotation replacement** that overwrites the `k8s.ovn.org/pod-networks` annotation set by OVN-Kubernetes. Without this networking annotation, the CNI plugin cannot assign an IP address and times out after 120 seconds. - -This affects **all pods** using `neuron-scheduler` on OVN-Kubernetes — it is not specific to llm-d, KServe, or any particular model. - - -| Factor | Detail | -| ---------------------- | ------------------------------------------------------------------------------------- | -| **Root cause** | `neuron-scheduler` uses full annotation replacement, not strategic merge patch | -| **Affected platforms** | ROSA HCP (and any OpenShift/Kubernetes with OVN-Kubernetes CNI + neuron-scheduler) | -| **Affected workloads** | Any pod requesting `aws.amazon.com/neuroncore` with `schedulerName: neuron-scheduler` | -| **Not affected** | NVIDIA GPU pods (use `default-scheduler`, no annotation conflict) | - - -### Implemented Workaround: MutatingAdmissionWebhook (Deployed) - -A custom `MutatingAdmissionWebhook` intercepts pod UPDATE operations and preserves the `k8s.ovn.org/pod-networks` annotation when the neuron-scheduler attempts to overwrite it. This makes the neuron-scheduler's full-replacement writes harmless — the OVN annotation is always restored. - -**Status:** Deployed and verified on ROSA HCP 4.21.7 (April 2, 2026). See Appendix B for live test results comparing this approach against `hostNetwork: true` and Kyverno policies. - -### Long-Term Solution: Neuron DRA Driver - -The [Neuron DRA driver](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/containers/neuron-dra.html) (released with Neuron SDK 2.28.0, Feb 2026) eliminates the neuron-scheduler entirely by using Kubernetes-native `ResourceClaim` objects for device allocation instead of pod annotations. With DRA, pods use `schedulerName: default-scheduler` and receive NeuronCore assignments through the Container Device Interface (CDI) — OVN annotations are never overwritten. - -**DRA pod spec (replaces legacy neuron-scheduler approach):** - -```yaml -spec: - containers: - - name: vllm - resources: - claims: - - name: neurons - resourceClaims: - - name: neurons - resourceClaimTemplateName: inferentia-8-cores ---- -apiVersion: resource.k8s.io/v1 -kind: ResourceClaimTemplate -metadata: - name: inferentia-8-cores -spec: - spec: - devices: - requests: - - name: neurons - exactly: - deviceClassName: neuron.aws.com - allocationMode: ExactCount - count: 8 - selectors: - - cel: - expression: "device.attributes['neuron.aws.com'].instanceType == 'inf2.24xlarge'" - constraints: - - requests: ["neurons"] - matchAttribute: "resource.aws.com/devicegroup8_id" +make smoke +make smoke AI_NAMESPACE=ai-serving DEV_NAMESPACE= ``` -**DRA driver test results (April 6, 2026):** - -Tested `neuron-dra-driver:1.0.0` via Helm chart `neuron-helm-chart:1.5.0` on ROSA HCP 4.21.7 with `inf2.24xlarge`. - - -| Step | Result | -| ------------------------------------------ | ---------------------------------------------------------------- | -| Helm install with `draDriver.enabled=true` | Succeeded | -| DaemonSet scheduling on inf2 node | **Failed** (`DESIRED=0`) — patched manually to add inf2 affinity | -| DRA driver startup after affinity patch | **Crashed**: `unsupported instance type: inf2.24xlarge` | - - -**Blockers (as of April 8, 2026):** - - -| Blocker | Status | Who must fix | -| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | -| **DRA driver binary rejects inf2 instance types** | **OPEN** — `neuron-dra-driver:1.0.0` is the only version available. The compiled binary's `mappings.go` lacks inf2 entries. | AWS Neuron team (new image build required) | -| **KServe lacks `resourceClaims` support** | **OPEN** — no upstream PR found | KServe / Red Hat RHOAI team | -| **Helm chart DaemonSet excludes inf2 in affinity** | **Likely FIXED** — chart v1.5.0 `neuronInstances` list includes inf2 types | Verify with retest | - - -**Version investigation (April 8, 2026):** Checked AWS ECR, Neuron Helm chart repo (latest: v1.5.0), Neuron SDK release notes (SDK 2.28.0/2.28.1), and Neuron DRA documentation. Only `neuron-dra-driver:1.0.0` exists. The DRA documentation explicitly describes the driver as targeting "AWS Trainium instances" and uses `Trn2.48xlarge` in all examples — no inf2 examples exist. - -> **Note:** The Neuron Operator (`awslabs/operator-for-ai-chips-on-aws`) is at v1.1.1 (released March 26, 2026) and manages the legacy device-plugin + neuron-scheduler stack. The DRA driver is distributed separately via the Neuron Helm chart (`aws-neuron/neuron-helm-charts`). A future Neuron Operator release or Neuron SDK update may bundle an updated DRA driver with Inferentia2 support. **Check for new versions of both the Neuron Operator and the Neuron Helm chart before each deployment** — the DRA driver fix for Inferentia2 will likely arrive as either a new `neuron-dra-driver` image tag or as part of a new Neuron SDK release. - -**Recommended path:** - -- **Now:** Continue using the MutatingAdmissionWebhook (deployed and verified). -- **Before each new cluster:** Check for newer Neuron Operator, Helm chart, and DRA driver versions. If a DRA driver newer than `1.0.0` is available, retest on inf2. -- **Upstream action:** File issue on `aws-neuron/neuron-helm-charts` requesting inf2 support in the DRA driver binary. -- **Migrate to DRA:** When all three blockers are resolved (driver supports inf2, KServe supports `resourceClaims`, Helm chart affinity includes inf2). - ---- - -## Appendix B: OVN Annotation Fix -- Live Test Results (April 2, 2026) - -Three candidate fixes for the neuron-scheduler / OVN-Kubernetes annotation conflict were tested on the live cluster against a `qwen3-coder-neuron` InferenceService pod requesting 8 NeuronCores on an `inf2.24xlarge` node. - -### Test 1: hostNetwork: true - -**Approach:** Set `hostNetwork: true` and `dnsPolicy: ClusterFirstWithHostNet` on the InferenceService predictor spec. The pod uses the node's network namespace, bypassing the OVN CNI entirely. - -**Result: PARTIAL SUCCESS -- CNI bypassed, but port conflict prevents container startup.** - - -| Step | Outcome | -| ---------------------- | ---------------------------------------------------------- | -| Pod scheduling | Scheduled via `neuron-scheduler` to Inferentia node | -| CNI / sandbox creation | **Bypassed** -- no `FailedCreatePodSandBox` error | -| Image pull | Completed (18.9 GB in 1m38s, cached on subsequent attempt) | -| Container start | **Failed** -- `OSError: [Errno 98] Address already in use` | -| Pod IP | Node IP (10.0.10.21) -- confirms hostNetwork active | - - -**Root cause of failure:** vLLM binds to port 8080 on the node's network namespace. Another process (likely the Neuron monitor DaemonSet or another node service) already occupies that port. With hostNetwork, there is no port isolation. - -**Additional finding:** KServe's reconciler **reverts manual patches** to the Deployment. The `hostNetwork` setting had to be applied via the **InferenceService** spec (not the Deployment) for it to persist. - -**Verdict:** Not production-viable. Bypasses the CNI issue but introduces port conflicts, breaks NetworkPolicy, and is incompatible with Service Mesh. - -### Test 2: Kyverno Mutate Policy - -**Approach:** Install Kyverno v1.17.1 on OpenShift via Helm, then create a ClusterPolicy to restore the `k8s.ovn.org/pod-networks` annotation on pod UPDATE operations. - -**Result: BLOCKED -- Kyverno admission controller fails TLS initialization on OpenShift.** - - -| Step | Outcome | -| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -| Kyverno Helm install | Background, cleanup, and reports controllers started (3/4 healthy) | -| Admission controller | **CrashLooping** -- startup probe fails with `tls: internal error` | -| TLS secret | `kyverno-svc.kyverno.svc.kyverno-tls-pair` **never created** by init container | -| Root cause | Kyverno v1.17.1's init container (`kyverno-pre`) does not generate the admission controller's TLS certificate on OpenShift/ROSA HCP | -| Policy test | **Not reachable** -- admission controller never became healthy | - - -**Additional finding:** Reinstalling with `securityContext=null` overrides did not resolve the issue. The TLS generation failure appears to be a Kyverno + OpenShift compatibility issue requiring either manual TLS provisioning or OpenShift-specific Helm values not documented in the standard install guide. - -**Verdict:** Requires additional OpenShift-specific configuration work. Feasible but adds significant operational overhead for a single policy use case. - -### Test 3: MutatingAdmissionWebhook (Custom) - -**Approach:** Deploy a lightweight Python webhook that intercepts pod UPDATE operations in the `llm-serving` namespace. When the webhook detects that `k8s.ovn.org/pod-networks` was present in the old object but missing in the new object (i.e., the neuron-scheduler stripped it), the webhook returns a JSON Patch that restores the annotation. - -**Result: SUCCESS -- Pod starts with both OVN and Neuron annotations. No OVN restart needed.** - - -| Step | Outcome | -| ---------------------- | ---------------------------------------------------------------------------------------------- | -| Webhook deployment | 1 pod, TLS via OpenShift Service CA (`service.beta.openshift.io/serving-cert-secret-name`) | -| CA injection | `service.beta.openshift.io/inject-cabundle: "true"` on MutatingWebhookConfiguration | -| Webhook scope | Only pod UPDATEs in `llm-serving` namespace (`failurePolicy: Ignore`) | -| Pod scheduling | Scheduled via `neuron-scheduler` to Inferentia node | -| Webhook interception | Webhook log: `RESTORED OVN annotation for pod qwen3-coder-neuron-predictor-... in llm-serving` | -| CNI / sandbox creation | **Succeeded** -- `k8s.ovn.org/pod-networks` preserved by webhook | -| Container start | **Succeeded** -- vLLM container started, model loading | -| Pod IP | 10.130.2.30 (proper pod CIDR IP) | -| Annotations after fix | Both `k8s.ovn.org/pod-networks` AND `AWS_NEURON_IDS` + `NEURON_ALLOCATED` present | - - -**Webhook implementation details:** - -- **Language:** Python 3.9 (UBI9 base image, already in cluster) -- **Size:** ~60 lines of Python, deployed as a ConfigMap -- **TLS:** Auto-provisioned by OpenShift Service CA operator (zero manual cert management) -- **Scope:** Namespaced to `llm-serving` via `namespaceSelector` -- **Failure mode:** `failurePolicy: Ignore` -- if webhook is down, pod updates proceed normally (annotation race may recur, but other workloads are unaffected) -- **Latency:** <5ms per intercepted UPDATE (Python HTTPServer with JSON patch) -- **Resource cost:** 50m CPU, 64Mi RAM - -**Deployed resources:** - - -| Resource | Namespace | Purpose | -| -------------------------------------------------------- | -------------- | -------------------------------------------- | -| `Namespace/neuron-webhook` | -- | Isolated namespace for webhook components | -| `ConfigMap/webhook-code` | neuron-webhook | Python webhook source code | -| `Deployment/ovn-annotation-preserver` | neuron-webhook | Webhook pod (1 replica) | -| `Service/ovn-annotation-preserver` | neuron-webhook | ClusterIP service with auto-provisioned TLS | -| `Secret/webhook-tls` | neuron-webhook | TLS cert/key (auto-generated by Service CA) | -| `MutatingWebhookConfiguration/preserve-ovn-pod-networks` | cluster-scoped | Webhook registration with injected CA bundle | - - -**Verdict:** Production-viable. This is the recommended fix until AWS resolves the neuron-scheduler annotation overwrite upstream. +→ **[tests/cluster-smoke/README.md](tests/cluster-smoke/README.md)** -### Test Summary +## Documentation +| Doc | Topic | +|-----|--------| +| [docs/architecture.md](docs/architecture.md) | Stack, traffic path, diagrams | +| [docs/requirements.md](docs/requirements.md) | Hardware, software, permissions | +| [docs/ide-and-extensions.md](docs/ide-and-extensions.md) | Dev Spaces, Continue / Cline / Roo | +| [docs/models-and-routing.md](docs/models-and-routing.md) | vLLM, llm-d, RHCL | +| [docs/benchmarks.md](docs/benchmarks.md) | Performance highlights + links | +| [docs/customization.md](docs/customization.md) | Prompts, rules, MCP, quality gates | +| [AGENTS.md](AGENTS.md) | Maintainer chart / wave map | -| Fix | CNI Bypass | Pod Starts | Port Conflict | Mesh Compatible | Effort | Verdict | -| --------------- | -------------------------- | ------------------ | ------------- | ---------------- | ----------- | -------------------- | -| **hostNetwork** | Yes | No (port conflict) | Yes | No | Low | Not viable | -| **Kyverno** | N/A (blocked) | N/A | N/A | Yes (if working) | Medium-High | Needs OpenShift work | -| **Webhook** | Yes (annotation preserved) | **Yes** | No | Yes | Medium | **Recommended** | +## References +- [Red Hat OpenShift AI](https://www.redhat.com/en/technologies/cloud-computing/openshift/openshift-ai) +- [OpenShift Dev Spaces](https://developers.redhat.com/products/openshift-dev-spaces) +- [llm-d](https://github.com/llm-d/llm-d) +- [vLLM](https://github.com/vllm-project/vllm) +- [Kuadrant / RHCL](https://docs.kuadrant.io/) ---- +## Tags -Document Version: 2.4 -Last Updated: April 2026 -Status: Active Development -- Phases 1-4 verified, Phase 6 completed. Phase 6 GuideLLM sweep on L40S completed (April 20, 2026): Qwen3-Coder-30B-A3B-FP8, single L40S, peak 1,357 output tok/s, 93 tok/s single-user, 73ms TTFT. Phase 4b 2-replica scaling investigation completed (April 9, 2026): TP=16 requires all 16 NeuronDevices for NeuronLink topology — 8-device split causes NEFF execution failures and disk pressure; horizontal scaling across nodes recommended instead. Phase 4 Trainium trn1.32xlarge with DRA driver deployed (April 9, 2026): 16 NeuronCores via ResourceClaimTemplate, TP=16, 16k context, llm-d routing verified, GuideLLM sweep confirmed. Phase 3 heterogeneous routing verified (April 8, 2026): same-namespace, native TLS, 0 errors in GuideLLM sweep. Phase 5 (inf2.48xlarge dual-instance) planned. +- **Product:** OpenShift AI +- **Use case:** AI coding assistant, private inference +- **Business challenge:** Adopt and scale AI without sending code to public APIs diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..4147088 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,81 @@ +# Architecture + +Private AI Code Assistant runs on OpenShift: developers use OpenShift Dev Spaces; inference stays on-cluster via an OpenAI-compatible gateway in front of vLLM (llm-d). + +## Request path (RHCL + llm-d) + +```mermaid +flowchart TD + IDE[DevSpaces] + RHCL[RHCL AI Gateway] + Local[llm-d] + Other[Other cluster] + Ext[External APIs] + EPP[EPP] + VLLM[vLLM] + + IDE --> RHCL + RHCL --> Local + RHCL -.-> Other + RHCL -.-> Ext + Local --> EPP --> VLLM +``` + +1. **Developer workspace** — Continue, Cline, Roo Code, or OpenCode in a Dev Spaces pod call the RHCL AI Gateway (`pca-ai-gateway`) over cluster-internal HTTPS with a per-namespace API key. +2. **RHCL AI Gateway** — AuthPolicy validates the API key; traffic is forwarded to the local llm-d gateway (or optionally other backends). +3. **llm-d Endpoint Picker (EPP)** — Scores replicas (prefix-cache affinity, KV-cache headroom, queue depth) and returns the chosen pod to Envoy. +4. **vLLM** — Serves the model on NVIDIA GPU via `LLMInferenceService` (KServe). + +![Inference traffic flow](images/architecture-traffic-flow.svg) + +## Platform stack + +| Layer | Typical components | +|-------|--------------------| +| IDE | OpenShift Dev Spaces (che-code), AI extensions / OpenCode | +| Front door | RHCL AI Gateway (`pca-ai-gateway`) + AuthPolicy | +| Routing | llm-d gateway, Gateway API HTTPRoute, EPP | +| Serving | vLLM via `LLMInferenceService` (KServe RawDeployment) | +| Platform | RHOAI, NVIDIA GPU Operator, NFD, cert-manager, LWS, Service Mesh | +| Cloud | ROSA (AWS), ARO (Azure), or an existing OpenShift cluster | + +Cloud-specific versions and GPU SKUs are documented in the deploy guides: + +- [ROSA](../PCA_Deployment_ROSA/README.md) +- [ARO](../PCA_Deployment_ARO/README.md) +- [Existing OpenShift](../deploy_existing_openshift/README.md) + +## GitOps chart waves + +ArgoCD (ROSA/ARO) syncs charts from `charts/` via `pca-app-of-apps`: + +| Chart | Wave | Role | +|-------|------|------| +| `pca-app-of-apps` | root | AppProject + child Applications | +| `pca-operators` | 1 | Operator Subscriptions (RHOAI, GPU, Dev Spaces, NFD, RHCL, …) | +| `pca-platform-config` | 2 | Namespaces, DSC/DSCI, secrets, CheCluster, optional guardrails/MCP | +| `pca-ai-serving` | 3 | PVC, HardwareProfile, LLMInferenceService, llm-d, RHCL gateway, observability | +| `pca-devspaces` | 4 | Per-developer DevWorkspaces + extension ConfigMaps | +| `pca-benchmarks` | 5 | Optional GuideLLM Job (opt-in) | + +Existing OpenShift deploys only `pca-platform-config`, `pca-ai-serving`, and `pca-devspaces` via Helm. + +## Diagrams + +> Open each file in a browser or IDE for full resolution. + +![Full-stack architecture](images/architecture-diagram-landscape.png) + +![Deployment view (layered)](images/deployment-view-layered.png) + +![AWS architectural view](images/aws-architectural-view.png) + +![Red Hat components view](images/redhat-components-view.png) + +SVG sources for the same views live alongside the PNGs in [`docs/images/`](images/). + +## Related + +- [Models and routing](models-and-routing.md) +- [IDE and extensions](ide-and-extensions.md) +- Maintainer chart map: [AGENTS.md](../AGENTS.md) diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..63a54ed --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,48 @@ +# Benchmarks + +Short highlights only. Full GuideLLM sweeps live next to the cloud deploy trees and in the GPU sizing assets. + +## ARO — Qwen3.6-35B-A3B-FP8 + +### NVIDIA A100 80 GB (Central US) + +| Metric | Single user | Peak (short / medium) | +|--------|-------------|------------------------| +| Output tok/s | ~138 | ~2,781 / ~2,008 | +| TTFT | ~57–117 ms (short/medium) | — | +| ITL | ~6.8 ms | — | + +Capacity planning (≈30% concurrent, medium 512/256): roughly **3–4 interactive developers per GPU** at a comfortable SLO; 50 developers ≈ 1 GPU “good”; 100 developers ≈ 2 GPUs “acceptable.” + +Full tables: [PCA_Deployment_ARO/testresults.md](../PCA_Deployment_ARO/testresults.md) + +### NVIDIA H100 NVL 94 GB (Australia East) + +| Workload | Prompt / output | Peak total tok/s | Sync latency | Sync TTFT | +|----------|-----------------|------------------|--------------|-----------| +| Code completion | 256 / 128 | ~4,512 | 0.70 s | ~36 ms | +| Code generation | 1,024 / 512 | ~12,790 | 2.78 s | ~83 ms | +| Code review | 4,096 / 1,024 | ~16,133 | 5.59 s | ~157 ms | +| File generation | 8,192 / 2,048 | ~13,976 | 11.15 s | ~208 ms | + +Full tables: [PCA_Deployment_ARO/testresults_h100.md](../PCA_Deployment_ARO/testresults_h100.md) + +## ROSA — Qwen3-Coder-30B (NVIDIA L40S, historical) + +Validated patterns on ROSA HCP with `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` on g6e (L40S): + +- Single-user ITL ~10 ms; TTFT often ~265–335 ms through the gateway stack +- Two-replica llm-d load balancing with prefix-cache affinity observed under repeated system prompts +- Prefer NVIDIA FP8 for MoE code models vs Inferentia BF16 for latency and $/token + +Use current chart values and [GPU sizing](../assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v3.md) for capacity planning rather than dated lab dumps. + +## Sizing and TCO + +- [GPU sizing considerations (v3)](../assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v3.md) +- Optional GuideLLM Job: `charts/pca-benchmarks` (disabled by default; opt-in via cloud values) + +## Related + +- [Models and routing](models-and-routing.md) +- [ARO deploy guide](../PCA_Deployment_ARO/README.md) diff --git a/docs/customization.md b/docs/customization.md new file mode 100644 index 0000000..8c6d919 --- /dev/null +++ b/docs/customization.md @@ -0,0 +1,34 @@ +# Customization + +Configuration-only customizations for implementation engagements. The Helm charts and operators stay the supported upgrade path; tune prompts, rules, and workspace templates rather than forking the platform. + +## System prompts and context + +Continue, Cline, and Roo Code accept custom system prompts. Deliver them through the Dev Spaces ConfigMaps owned by `pca-devspaces` so every workspace gets the same baseline (languages, internal libraries, architecture rules, logging/test standards). + +## Per-repository rules + +| Extension | Rules location | +|-----------|----------------| +| Continue | `.continuerules` | +| Cline | `.clinerules` | +| Roo Code | `.roo/rules/`, `.roo/rules-{mode}/` | + +Commit rules next to application code so AI context evolves with the repo. + +## DevWorkspace templates + +Offer stack-specific DevWorkspace images/templates (Java, Python, Node, Go, IaC) from the Dev Spaces dashboard so developers pick a ready environment instead of installing toolchains locally. See `assets/devfile.yaml` and OpenCode image assets under [`assets/`](../assets/). + +## MCP and internal systems + +Optional MCP servers (docs, OpenAPI catalogs, issue trackers, SCM search) can be wired via `pca-mcp` and IDE ConfigMaps. Enablement is documented in [deploy_existing_openshift/README.md](../deploy_existing_openshift/README.md). + +## Quality gates (e.g. SonarQube) + +Encode the customer’s quality profile as plain-language constraints in the system prompt or rules files so generation aligns with the gate before commit. Export rules from the customer’s SonarQube/SonarCloud profile; do not hard-code a sample “ACME” profile into the platform charts. + +## Related + +- [IDE and extensions](ide-and-extensions.md) +- Chart: `charts/pca-devspaces` diff --git a/docs/ide-and-extensions.md b/docs/ide-and-extensions.md new file mode 100644 index 0000000..73bfd30 --- /dev/null +++ b/docs/ide-and-extensions.md @@ -0,0 +1,56 @@ +# IDE and extensions + +Developers use **OpenShift Dev Spaces**. AI extensions call the cluster-internal RHCL AI Gateway (OpenAI-compatible `/v1`), which forwards to llm-d / vLLM. Source code and inference stay on the cluster when configured that way. + +## Access modes + +### Web IDE (default) + +Open the Dev Spaces dashboard in a browser. A workspace pod runs VS Code for the Web (che-code) plus AI tooling. No desktop install required. + +![Dev Spaces AI assistants](images/devspaces-ai-code-assistants-architecture.png) + +### Remote SSH (optional) + +Desktop IDEs can attach to the workspace over an authenticated `oc` tunnel (SSHD on port 2022 is not exposed via Route). See Dev Spaces docs for the SSH flow. + +![Remote SSH security flow](images/devspaces-remote-ssh-security-flow.png) + +## Extension comparison + +Continue, Cline, and Roo Code are Apache 2.0, speak OpenAI-compatible APIs, and are installable on Open VSX (Dev Spaces che-code). The charts pre-wire them via ConfigMaps in `pca-devspaces`. + +| Capability | Continue | Cline | Roo Code | +|------------|----------|-------|----------| +| Primary UX | Chat, edit, agent, **tab autocomplete** | Autonomous agent sidebar | Multi-mode agent (Code, Architect, Ask, Debug, …) | +| Tab autocomplete | Yes | No | No | +| File / terminal | Yes | Yes | Yes | +| MCP | Yes | Yes | Yes | +| Config delivery | `config.yaml` ConfigMap | Settings ConfigMap + one-time UI fields | `settings.json` + provider profiles ConfigMaps | +| Self-hosted vLLM | `apiBase` + model | OpenAI-compatible base URL | Needs native `tool_calls` from the model | + +**Practical split:** Continue for autocomplete; Cline/Roo for agentic workflows. Deploy all three if you want developers to choose. + +OpenCode is also supported on ARO / existing-OpenShift paths via a custom Dev Spaces image — see [deploy_existing_openshift/README.md](../deploy_existing_openshift/README.md) and [PCA_Deployment_ARO/README.md](../PCA_Deployment_ARO/README.md). + +## Gateway URL (in-cluster) + +Default IDE endpoint (RHCL front door): + +```text +https://pca-ai-gateway-data-science-gateway-class..svc.cluster.local/v1 +``` + +Each Dev Spaces namespace gets an API key Secret (`pca-ai-gw-apikey`). Break-glass (no API key): point IDEs at llm-d directly with `aiGateway.escapeHatchToLlmd=true`. + +## Tool calling (Roo Code / agents) + +Agent modes need the model to return OpenAI-style `tool_calls`. For Qwen3 / Qwen3.6 families, vLLM must set the correct `--tool-call-parser` and `--reasoning-parser` (values live in cloud `values-*.yaml` / ServingRuntime args). Wrong parsers show up as “model did not call any of the required tools.” + +See the ARO deploy guide section on tool calling for the current parser matrix. + +## Related + +- [Architecture](architecture.md) +- [Models and routing](models-and-routing.md) +- [Customization](customization.md) diff --git a/images/architecture-diagram-landscape.png b/docs/images/architecture-diagram-landscape.png similarity index 100% rename from images/architecture-diagram-landscape.png rename to docs/images/architecture-diagram-landscape.png diff --git a/images/architecture-diagram-landscape.svg b/docs/images/architecture-diagram-landscape.svg similarity index 100% rename from images/architecture-diagram-landscape.svg rename to docs/images/architecture-diagram-landscape.svg diff --git a/images/architecture-traffic-flow.svg b/docs/images/architecture-traffic-flow.svg similarity index 100% rename from images/architecture-traffic-flow.svg rename to docs/images/architecture-traffic-flow.svg diff --git a/images/aws-architectural-view.png b/docs/images/aws-architectural-view.png similarity index 100% rename from images/aws-architectural-view.png rename to docs/images/aws-architectural-view.png diff --git a/images/aws-architectural-view.svg b/docs/images/aws-architectural-view.svg similarity index 100% rename from images/aws-architectural-view.svg rename to docs/images/aws-architectural-view.svg diff --git a/images/deployment-view-layered.png b/docs/images/deployment-view-layered.png similarity index 100% rename from images/deployment-view-layered.png rename to docs/images/deployment-view-layered.png diff --git a/images/deployment-view-layered.svg b/docs/images/deployment-view-layered.svg similarity index 100% rename from images/deployment-view-layered.svg rename to docs/images/deployment-view-layered.svg diff --git a/docs/images/devspaces-ai-code-assistants-architecture.png b/docs/images/devspaces-ai-code-assistants-architecture.png new file mode 100644 index 0000000000000000000000000000000000000000..ad6ae52f175a2ee61a53da94088a60b074ae05d2 GIT binary patch literal 91024 zcmbrl1zej?&@dP(6m1(QE-ezAP=ae)JZSMkkOCnzxKp4|r9jbyAVG^e6pKI=cL@#w ziaRaEinYi8m3!sxyZ64kd1PmIW_M@z_w3H@>^y%b{(cAC)YXD&0VpT{0E&wX@OOb? z0;;ZV|Ioxp3#zaAuY^v(1zfoc0C;%%`3$A86@|G|U&z5OqE z9{z*-nwh9y;P4Av$mKubj{gaF^7j3QKlXx0*~2~HpSu2med11px37008UMf28rh2LNuo0RV6V|B?0|ed6ul z>+mndUAcH)LLdNuU&R0by%hk!I0yjH!2X4H@%mrLcISe`c_EkA#pMEU2RH-n0H6R* zfD-_Gfk6Ow0g?dOztaG9z~xJq{<$yU%Ef(^>gv@iSFX}ryGBVxM?*(POG8Ubf1Tkb z{dM4VTH2e;H-U_|n3$O8Zm_U2-(qFB#dPbRLMSd@eBryfXfuTfJ;D%%Rq|1tpM(e?!FL>;-63P-{30cHLA;3s4r2_T+m%^0sxd0mo8no zMny|Yb&c}MH2}q>i#AYR12XVXF-ktT#msB$fEi+9fykO$_`Kv(Gw}}yL?tEHlBijw zP|$+Exu}TaLV|zf_;)ET z!YfxVU8bbCc0q&!|B><^F{r7o(q6fILA;Oxc;zY&gCyevV;{;}ybu$Im#85oW>d3^ zDoK1vwcqBYq-E619i4n%4gXZf_k*%- zc5@!8EsyZ&fxsnYT0xr{}%ip6Zv>ddo1uUk-nLt z1p>Rvt0JHtjZhMj0Swy(mHR~ z>}=;b9|&{k3HYMMf<^>>-Xx2V)K#88oaQ$u=!xLRfIu~294{un-X!@cQdz`}=`>b5 zwhg>y>Jri`J7I7kr1A0bLR31xhQA2BakpXbNWDRcb6mghsT!Z|u$Uo9#`#r0ugxY) zD|}wKe`L4vdO_^xrG$^yir@1|X1=>_A8$Tup==^rk%tFM6(06g@)zVRK{#2kRrg}$ zrvr}y4&oKMs8d1o?4Ms5CsM!X1iENvaY6V#Pj3kkqT9!Gp{Y5(H@7z^AqS;na*1Q$ z)&!?0tVc#cxF^>Uh3aMdlZS1i(H2h?>xb`_wDSsl6LgIM;gX?lzDwECq^;KV)~^eDjTk8mwoGla$0;uRm_iZlgf&4f=%V zXCK5AV-K09QGqWe?z4Xzk|+}Pf9)r4zR4s%C0xl!Nh_4vT0Hp7&Nv-6m#+7rsm|dvVma1r+^UVOA@SI<$j!jjlZ&Cey&@r z8fhdB71E-scxTTBrzUog3r}f?rc>Kf&d;!?IMRWj=>Wr_Sio zyR4Cq4^x-979qUk0mr=;=M+O_AFDokcv>1JDAeDkNoE@%K<`OzIhll*?3a$P$GPSY zsc@7;D9P8x&HK8G5gVB5K%#IImU-!oFrv^?0|o?Qf6=#6yvJM2C`iARXc|t6bJYyh z?BK;+rj|2_|E2pdWz6L)TjG8D!ejQAyqkuSgPNah>!%wSfWSL+vi<#KB~zKBwZ*9b;uTU(YmWnS&IS82IKvoE2CBVqJcS>Hy1?- zghtmBBKTr4Nm|ppz+V&B2C1H4v#y-ViRRnhN2z>?WFiWgMxU5y6OzLY7p05se8gp6 z&FEQeY1k!EbY7y#QS5-Q8ETb(uB=I%YjG7TkS{pr*Al${KdG62r(Z!B@48O^(EjqU zH7&MVRU8(!cC^7tefQt#|6@pDHPXBBMgIfsS=mstE4{v;Gr?sR3e9d?l~UDE@9vK} z=aqg{saTh^aj+TZ%r;!D%#ln`1_W9(50>A4!6bH%lMflP--23#~hD}39g);zR);bkeDJ_m*T zy4IKiHf+g}YPdJM(7TC+VAfZ4!&FBNZnvh}Fh8}zSC%3P+l5yuG0(2n1V1r}Iy^-) z`lsK{kU1^z+EB+B;olM;ZEC{v!Q3+1(&)gsup^ZhE_P`~6YXm)rQlH?%>J;-XvT9Y zqTloe*-?UA7p>o+X}#)3T}>Alx~c-5j-BBXkG6jhNZ(Lj5f=axxiV(T%U^`)=@Y|) zay4sE=UU1?l85T-8aY-ij@Vg7oC%s4MSE#y`3t5t$1Idf2u4XwzWPvDB$ z>nJX9pQ?nKs%59&>AV8XGsY~i^a@HfSt;<1P1 z9$)DnE7NIo&Lpq*ZzYH3^a?ft(bHrnHdQelOfl2lF%ynOU88W#njPn!?^sV5)ofyz zz#bQTXq&dR#HE_6VZ_&id3dp85NX6FTTsw>#Gg)x?QGJCYHIDzlObgk&WI{L_j=hj zY~!jPb!6AK8hf(De*}SYw>uNPnl(bPERsnln*MBdrumPUEVB9y5U>}Z2Gh=98~(q5 z8`SA$Rn3+RPz{%!u%m4X!;yRBb`n`a^B1iMalEGXT{>2ZW?o9UI8$B31YH0GdY3!# z@lG774zs#w9+b^J_zUm@FYLS=P6(2_6H>)rE8ee|DJ3Isht_4nr}ZWfoSU-lufHsj zzxtV8P|wXxp!3DPLh6ruwOsDg($(c!r|cLQ%H5^+^jgq;MfRCxu4`5G)yl6{V&0Y4&Xw!Wg{O}b~NT;9hRMO*)H+r2wO4BTj2IfCdZ}<8_ z+m+CACzYOXSjK7U$uMfe6Jc4LEA)%>pF@2fQ~PmdFbNLd~55bZThxtiCx2FfvX6Wg~`a(QR4&RTA znmX_Boi(bcMNN`pe;cLpsx;}uc7FUW`bqpC>(iA%x;invXw4*_cyU*w?ZT!dnVv6B zs(ca<{a|8ldg1%}8M-~rYZ6_vE;2osv#xlNGSeB2H#x`P9Z(&}yOjS^=c$b`{fG6T ze8ra}bu<4Cs)}N?wI74E;H-6_h5;J$nNb|S^5jV>zHq3{;E8NHgv6nhz0mrG9}OGU zlA8D=@PfVWG&z0Z)BQoKG%Cs^TL?eB$(ee+2QGVp!0G_tSG zk873k#u{%JybIr@4h$|(^%ly_eSG`FSOI_4e)(5Vji0w+v=rHxhZn; zSft0M<}8wzH2eZ%ihtz!RLX4{i@no%G>3PZtE)HKlvB4k$=8);WXOB9Vj#*|qaUNa zSVCaqvl51R9TxU}&dm;}d#L!}@l>3Gf4Yl#Nu&Si6Fnf`m->Q|m~NR1op?;p|8z%E zTl4H%-fS2K?!7;LhZuYC$#K57txzxc&4J6$6pXxv8w7kl`r(^=kb0jyas2>=G)wbK z07h6W6Q95?^vM8kBib74wrY=-9 z909j~w7F@Q;>Mid-=13082q+asjBYf_jq;nk0KA!P4qxOPv<^oSAIXqbkuo6maJa+g7&hz#oXG(*tplr-MlbKl-T%HUp4hzi-x)J_ud zw)k5b`7PALhtKI$M5i}-nIXnc;sJgBc%0xEU7ZRk%Rl;+=BK98R~YMgqmR1sJD&NI z9I44a@#Lu9Sp0-ph~WbJ0ST?j!xJJlCQ4J*2F^J);YzQaQA&@2r$5xNeW7?R>;3k? zuh=kd(Nqj%UgM`T?A86M>KgjBS%ermjI5!RlWQ9}!K>)CZ`vNh<-{6aiZt8Qr7f|f zSGNKi6AaliV9|DG34gjY3J!&makI~ME37#(;9>I8Zn9PRA%;>=i?@OU&{SGl-sirm zq34ayh3dlBwGx-pN8g**%X_HfEh_ChQV3gEU=q}MKb?3Shm@$GYzSW;d*|q}ZXjFN z9s{cA;IDhpI)3XoLu<0x`;gzG^G|bp`0KscsUr>DZHBeUOuMyun9lL}3S=5;0^*3T z6nNr8g5&r==}&(Lu6~~On|RddvhUv=(2-}$ES}Jzjs;15D8{fb>&V67m`T%osx~l1 zgk3mSQwJq_LjuZ~ukR~jW$kwDW|s&uJ;6Y+UoE;69o0o61x-z=r5e$^*kftIZN4Hc zQvd7|LxQZJr5LQ8csA&B6=`=N(MtC0jg}9sY7l$TO19ekBvX65@-t#Zck9Vq&nb6WKZW!qADlJsvLcXh2f74P*9wijdl?mZ9Re7g=JdnoL{*_%#Gu&*>7CGS_)>d zwNLmO^6t_AmIpqSSeft_aQRFBt=ShLTrqBQ8p`MxaYv1JzmMYe)SQ$(L@V?(YkuG# zPt3k)ypf1x(aK0;;}_&xk=yCT$`tFz4O^3tYG%%JQ_p|DaoO>CQe>Nd!WX8F9Z`Wg zfZIB5_7rI$BJRY*B+}cQ4^%~^2USX}?)`e=XOD`T<9WB|XpDpE8X#;m@v2XFWJ*p% zO{?xTIy*($7-mso1Fmendhx?zaT_;wZd|@&J}Y*@_jdo~T3o4B7MDP}T$&RW)TL)o z#_ye5)aN=Gin{G+7Fuw)qAt96BADKBKGc;Sd(P3f03KH0tiXe>f0*K(-#dKqRfVTs zw)p+wC#0%z;c$gWzRg4)UFQuQxtsOrq-5svb?NdCM=L5&HNs#ri$ltyVaW5fBA4=D z0klq=upJJOy$*2^&gb2Lf!_a4c>aKuY!O8wQJq5h?Em`lNKD0Zuiw5xC~(Q$V|9 zz^TO0UjVtkC4W>e{PWle$=2WPl9V88RH}um^ljwrZ&0>C=F=b=n~m!+!lX7l?Va9F zzlJ!M@F-4d%?|09D7+D6>uXE#qXqEHfWw--PW;>bkm{j+$_%IW89t1#N4GXcyLmba zTI`JVo78Y29{#9_Wz;ND5&PNelPRkeGiqTp`GID@kYZS4b zteq!z;j8a{={cfAm523M42YZ1!ehy4HRgHj;g3;q8mVn57m^fruK$?E7xG42Mn@V3 z!BV5gKCgze7ECw5lX<~?iOG#Q%Y6NEH;i&^`iHy6{1nqCXH zb8zlycs%Q}S;@KPcAm<3JsD@Lx(h_#AieFXXA35U`)IG54-N~DgqfC%#)XlIb-e`nNBO$6QG@IP{!jYT`YlQFnm4@uew>_y7v!nLu&A5pgV|1R%mrY$_ zDVUzHyys8*$n`|sUyT&oo^}13;{ZsFxp^R<-9If%)bRZDw!=|MyV&ue7iq458|3sN z1=G@!JN===f*G9!F*3=U>}RxVg`u#%3*paRoo9ClCAx%N3i~K$>zm#R+aQ_E9PE(Q&Vp(RaX{o7h~Xt#k>y-HQR0G>6DeqqkWnV{R~)qG&}99btY~X zKc1#iJ&m(hx|mY&RuJd)KF}J!mEMulyGGyl=EK z1q18IV7c2h*QIkZ?PX%}#tQe}t)0gzqz;t7tYAHRopW77YBGX*2~HaLklaa3StbI8 z=~nRhGFq8=T3NFz=8dT~Z#tvB2_{MN476DE$5;HGpeARH$#A*FTDJ)-;!y)hp!P^-OVVzCd}EkLg@L#9SDY4F5S!fM_*_)3J}x6Fyo zI4zr`-lpH}UpF7I_WfAx;6AmgoC~GiJ)VD?2^mF=k5^&BnjRaljZ&?ypdzytBu04 zD09J6; z$5M!}Wagi+ej^u2X71I1TH9y+N_wc8pDbojW0q-mO)|(`Okm16g!Pkt|GO%hPqWC_ zcV!(tzC1ivNEXL}^K3h>Ey#!aFZQjDrr)(BDCVgf6pS_8b^tQ3II-FIXOpIAHlTZZ zx@)`Obf1$|{Sjvnef3LM!`5ffHnC1Saai*z_1CWYGGx83U0q^=X2*}Bl%3v$5zkr~ zcN=&Fwjg;{rb)B50(JwpHtmqw_wv&F8n+P+9KY5$uty~Zd;2cO#18Iz+~aD5HfCIu z72cUyr~c94PVeEw;fnT`z?S?s{%?mrX@Am`mBVN@)WKW>Bqwz{`-CrsU_ngcP@KJu zQo4#~w|B}ryn%C3O_9ARB^k<0{VZ;L`T@!_o-3+1#%aPGf!bh}bc?YWo!tl&MZ&j0 z;-yc?@YZlnfBd1QrzQ~_xO3T|_G5n%Wwm3#LCnQ|3Qg9Ud3Jbl;;{9GrCRS{`X zBW68oV4h44bYIjkJ8~!vAiQOqs^dp(*&99iYAcm)79W?Eq8>Gru;p93m5wu>b$ zDONYG)}-YQF7>U?Cb`Zkp?Id^p16M2pTzlEwl-`dPSg!#4dpg^|lR=86t#i+iXnWIk9J9~{LS4h{5r;xKE zj5Cn|9m?jl&hw(t{s93$`X^y%chxNe7dYGVZQQl1i8bMC>_FVr-k!*?PfBz*1b&FN zK|FLn5bC{HyKOr+N+5D2ya5LH^XGk_nL{35Cgpf#u)+L@%On6hwO^TnDw|d5L8C*e z^$M##l}uHN#%^YZt8Iw(m0D!q=bo<}lYy(v2#LmfCFW^w8=zI~2b2isqSAItWRR1G zXfu0xO6o%ctSa0_V8N_T_LQLW)gx^CAW*_sWNg-rX>{nJr;3Pf7d;OBLrPNrYsmfe zl@geeWz4d-cm?bM@q48ZX(7?x9c|144AF6%%>deDL?X}Y(-Nk>j}px|=+OLdjx6^} zpkvnuelVmJu28TSc66Pt1y(=Row8tuNmf-a`5+|FU7|>^adnBP{A~b*G6YTFpod%V z%ZOG1y>wH@>mxE=Dg;BAfoCIw&31^|9~xRxC@aav)YyFu{1eNL%$jyJI3KmavN37C zF4}Xil*H8f57z|H=(Xzb2kpRxXMc>sZ>*_IZh)!`atwR!eDTqgeehEYwah*=;&BMB z41W;Zd9C29^2V@-#>lg3E{VzgYsD#m#WIyptE8H}dcTa_JV4(TFPbh7e zMG2b3c@JQ*k;EnnCQ?DRrJzDYDo$EUY08x^G(}}1LGE4p_)9e<9NkgBOz|mb!!_yb zQ|yZY)@&DOmW6_Yx?^8~z?`qh?BgS-;uOImrywEG={QTk95O3z5F=G!rYmon@#IMU z2hj70Ww8@a+02cGaigD=%J}?GJ+2#-XD)U3E~AfVv(1v`m)BVrn%hU449sk`446|y zr{4T3=m>B99{PU#VUoZwpA*0Pv*W`T`}_lrOxF%x)5IlM&3;0U}Z=GZ)XK!-jiDme<70Xs+saz?tfqRSLgrJp0 zc7|GitUT2pAY9Ai%MT8q|gL z5NDfWePDsS&EwN8v_n89ro{^g*Sn|8UH)z7+*9wp>1olgKlZ6E$K{ZaH7af8BUurs z{5rmspiLV%Ge2?ihv;f;zB283U#|b%1~(LH?`3lWA0LP(mkbEjw5Om=ZrMZGSbyA* z-rhUwa)}>=ehOkLuKc>Tw3u*OWnQVgF@I;~SuY8Gk~q)sR4FPZRdNlSrt>+u`04bd z+M+^Y)n!4IPS*wJ4uW3vlY5Y+@DI%1|8-eGaXUEl#)Y!LR&jVJ2}eDFh#&c^nhZN1 zR9={pbZ7)>&n(N%oC^Nz#a=r6uQ>sCRuKE<$L@5F1{rj7M)5@B#}$V@D~WJDlV_)F zDGT0iKlDYA(i8m(Win!=J#qJHHS zB>c@;oW)5AV}(_95&SMK?FGf_l0Hy$RBY3Yv_bvuKjF5y*2~q21xmov4#zPkG%b3n zJWjlbRPbYRtYXIM_=x9rb$XwZWty`Bvyzvel}G&22T|(i)KWEO+h>6@{P*W)^u@uV z&R(-FU+!=;lr*-yT^A8ePPK2#7iFrvIQ@$W!l3z#`J)oTqB=WVC`LJr%pP~9zME}r za#ZZX$3u=}|4j|}?j?@@NtGG5Xj~rp1y)@4v%LOj`hXS(pN)mxm$GE1#UAsAh6J+j zK1aK-!uyPIA?h-0byyi&N{vw|XZk4rf{0RLWx11dQc)h10DpFsa6=yuyvipuz)G5a zG~@>o<$f+w905~oTq@RoYxoh<`DBhiWgTK-2K#|qQFG#N|HUrK6;K^r5^SaM)h!=p z0`mKfiOh6%Ggb`sTnKVQ-JW5k-v{o@jZ69Wsjbvz$fcY1waTPEsuyg~uyZ88jfZWx zO=k7Aeo`M1wW08?+1RA>RSE6jB33D}W(I8IR2wwb(NC4zc{cL8>&HHc^!W8;gowre zU<`-W6U*YKlWrCyxfd!0^UpJwJ^3L;vG%CQ<)q^+(luXDxRcjRNzl)^IJVD?cN?Os zb8)I1Cot5fHd0Y-T+#e9)E$Z33V5C@X&l8N2K8?eNV#W@`-pf}HUbWE6M~t`t#-k3(e z(;&<%-rLjn;k;t}lPa+&#*J3AI_Z-Bh^6!|PE}vapVp9RU*qH0x0xE<>vfHEj(=AE z6dTrxz>qZ?)qLhYCR4yIUH`#?pT#kRzkoY0`s$j_R#`O7Ku+Z!US^HSv49I+c&KG> zpsR%qel18Ja(`8!t?6AC@m1`mP7&)M)Txmp>0OpTVm#WBCvrUEyy_1Z38y=`DKKcI zr+l8S33>&A2zw*l#9uH_+V~LJ@fmDuh2#p5%`!_*(*_!wjuyuuLbF#idlQ-*P>uIZ zdOJh74WlEh`nsJs<##v>6y1^>=)oW(peQOG^U89#mr|`GL~s4ARgzr7$VzgON0xDJ z$(nahrEwtomUC^wnq-nz{h}iN{m}typ*Y2G=SPTMK)}%VzkpFAeLSl-vm3>5$dCy^KheDn%3Z>GYPRZZZ%@4NdO?4+InSBdmMMSL zZ6t;Ye~9ae`n6qwX*GREK4S36umT?FS{vgZg=Ju`3!Qv(mY^=|81)N_b+nVODhG9a zBK`ao+wm%?GEceINFe104gGUhg!7lX9Dgnkz0=v$Q`yh&K{ z5Y`n&4eq8+noR!m+bikxoV~$Nmcf}EOS1XmeJ*#)Wd_T?q~kHVGtXFl5BIC{{OSiP z&hi3W+Ps2D3Aj#VLbe2=S2H%BW9BlU2jSyCM~D|@8*?qV>V9j`H@jSCbbaM@td{1O z&lwjn_302Ni%kA-e?pQ^OfX?apj^Ests-63%7!RmRcYAx%mc;3mHPI4|3C|@XEeNpYU9hsyD;gzCjQeZ|JME4ZOxG4~P#g~id zn8&A1+hJ(h;TqNqZjTMse2o#Yw#g(vVLHd516d_wPd;VjPx(Kz%N)AXT^A}6|8!zD z)=IfFBqnQdL_v9L*l2UYq5BlO!?8jx{4sVnH4qE|Yp+pYr0h^39>td7i%V8v0nR9Q zx7dX~UH60!_KS3gnZvAdqdIB1Oi`_0K8hF4F0> zB}~}H9mKuI<%Ba&BWE`$q(KSk8S?u?RFn?_+i-REIT$xCQ#-7p@;ad-<6U0;=v9W4 z0&L~;Yu(b^tp+b3Zf&oh zfof!#)iw?Yle-xRD^C!b=fpFDkjT+rzsw4N+~i0@M8WbTrT7E({#oZZ(*_0irzs4^ zUz_ATv?e1{V16o1r>FNrZY0=$?JoIlW8SIXxZYc&5i~S$;bKF|JJA8zk3YagSn;}v zO&I8ovO~seXZSA$KlmIk{s)ySUu4qz`%&NZ;Or^ntS~IXLN7P)(_SpLy0miEd%R}1 zw1GhfwZ0#<=HAug_a}k)ws46>nh`8oVI5-CW2V8LF8fXa<*_8Z1QdpzI^E@d3$t#` zE6Z?J+1GqcpWK;vnsXI5m}G(pf5j`sL<}KDek*RTRZPARLb zL|F85yU0W{M9%2!9S1kn*v~*!167ya@xwpi-#I#Bos)y#D43MNf9pvWe!4p9&HmI5 ziHKzEZHi4uOXCiAI1J`4a7EC+>+bP%6RD^@TgQH0>D;ghPV29C$G2*B#`uY!21@NE zkf)V%t~981&lyWdZfm!TXTOeKq*r{#kS_}rav9k^d`@-h9Nc>Z6^_CLJQr zyuddGaVBen`4%X<{yq+Qf zgq#yKj}8C~BeD2AfUUxbiWskSj;j`JE87s-r5MA{kcqfy^#(L8G? zujyTWJrCk_;Gs9prS-Wn02K*iXk_QreAoLV3ViEB5Bv=g-H z&8H_6$lx2c<#=-a89TvCjwKbw@whD2i(Y6Wn0%YOd`>Qom>>cG{{l1NS$)_IQ-#G}3&2>(D zt#Ppuz%o}|ZQMi36i1TOFe3tJMA9}N#K__J&6JrQR9HY4nzH0P;Y7K7F8rvXds|3S zS>*H{O23N3>Qnr=CwnA?Q-%O3nTnuDd&tbV!o^;jdmM^+@Y;35}rG_sm84t_}dQ#~+R&Rn7r%kyy z|5>$4W{?`32)nYW$iN$3GL9H_EW(pC-$^NgJ#_r|D<@aE@gzxZ@o0SYO4fu z(pMYylZnLK>Tsdm30)40vjl&Xd+57a^ZegMu*gRXzeJo`ifVj{K^-rq9&MiBP3A#1 z{4i8HRcupd@5NEDU-sfvH9ZdpB% z&}dx-yX?HyhRQJuA5hV!im(z|*gzzgO750n2fE^4H(;hUbP?(m+#ALyPs9+GCGD%N zsN}m4D~ZnPvIt6z*E8cz+Ul-;$6A`ic!8fAl*O4#CYHQNf?+16)je|&+~(u)`47r$ zAYhMbIA}L;$IFqgT_LXJ6>drPi7Lc6sZ7e)eHf$LI zyaAL*-BaacI-*``d(*c&%i_}Y8mZPZ>Fky*{Nj1u=S%vH@b>D@FbG%C*i z{#p)IGl}R4yz1T$soNiRMHEk&24V~@NiUxXi)*Rb+$+?8#o9;s=XU(sfBZuU=3q*D z*SXE24~@y72^KX8i4aOa54`Cmx;qKM*_^orns0w3Ru?(DG^dJzc;1wpwYovb*Q`RG z#}JM`S9sU zBu&|sqbIra<$rN(Vgof+GSo{>3$W!*5>Q{xJ;;X0iNy}c)>5m~J8DQ}-6J;wc<4T# zL;Yj_#g@ohqFMjQZ#w@mMN*oLhAi~Sw6QyK9t>k+d-pkgU7ud2HLLwmD(_TcmQ{O< zQ(Yljl1dkvOw3i8j;9BYu~%B=q|}(iO?Fg_J|aQ`z;YlBv?V*eG9fsIL!n=ex7*gw z>MAS+$WYi76PpY=9h;aehO9j|a(b+`nCw}a%4xHR?`I_Det}ZL$osp4UD+w1h;m~{Apqdgg zf)YCN3h>n3g7BmY)zK)Y)7FW2x^3l4b*(ydt9L%>#6ye=dZT(s6b_Mt(>)5NrJ_Y$ z(gBG5s7gYeQHG!R;r@*#F%$HLF8_q-*DM}tY0#u(;sQEHk`0wjKD}lw5E^S#mg>U7 zSDFO7`5WGg@Y8ip+7pLYCQaEE$-`X@`511z{emIY-KaLRr}AyX;Kv0$1%AiDe3+LT zL-l+q=5nG*)(uRh^*=%QThWq(XK*nruXt3VmL1d=Ja9KjjbV!6!nF*O(ux+3is@D5 zgCgL8bMYIu9a_j55_O!@Q!t3>)BQC|@WpSs~+1f%vx zhsX1(yomzDCdCf-RT}ueD>1Sx{@&)Rc#9PN2J5bV5R1qb({ijcr!J|e$n?NyKj5JS zw*ftBZWllS*DKNqez(<0ymwDKbrT~wJw!}Qo>N8j5`jnYIv*xpvk`W&p2~qe5%kh% zT@yP;-X#r%7gzMfwBm8!{1!|t+cLjIfYM6TBlk;Ue16BOMwDMQ)D-tcl=jyzVJx~@ z>}M`!+p~e30=*YhtCCa2Zr2+E5=U(iF5e@$D92Vpu|GZVNbKQ_w>2RMS0=L={afbc z=v)Cs>ZB86+}t1taK+u-HII{$N79ly&0XHGfeScjfU8yMf*MW(QF~-!6j=gL^7VXL zu_z@K?4FHF-z=9Y?I2J4kx!%07*r4tNP7nmofB<5^+-c%jnKP!Ld(#OfZ0K5_!%#XQ zVA&GN^&PW!#RS^RZtZ^vtv7MXZT31=LTw*V-=&D(saJi?Gx`YoDdtJ~zU%7}P5-wy zm%^`qATLgIp5;kR*jlPTT6E717`pWqW@-S^=kKZ)!kL)Bd@qKmhSK*-0^uNSH*P6L zJZ%q zX2Q&Le*xFTC)P|$_T2NuJO8hT!vEohp9I8ex0QQ+;yzcW5cqEs%fCcL6|qs=?~!9h zu|?1p^EUUWt2`=|8F8hYn)_aaIsm-|X~wRrd_3b^cq73M8m|RJ-NN$K&esJZpr5N$At(! zjk@%r-N;%+J?u!Wk9-4a@}&`X+r*9H@R|nM-BdFd;%#=LrnLPMfTQh|PF@Q$t>J2F zTpgiZl=rv)CUc^e);wEoVW|jYlL0-^Np!c(zXGln9b7veW8!1$*(#T);wF@K-A%n| zqCca`=RC?DH9c|c<0t-hQKK~Ck{MAlueLSX71S9}4-_5+x@+bCP>FP>#5_KcDT(we zFkGEoUn*1L5SAK3^0i#2bK&4crdm9UJzNU)B^#I4+gYt`K2_YGeFiHD9toC}5MiK> z_I?Ix<0sAEPQkqI@x2%aLja1rlj>aTRaL(d-4-A1=?>n3jvu_^0gtv2Y8T@YMxJcIg;JZQ0np%xQe^eLU5?(K%zjUZm#vjqFjZ6ZmKC;!s?^ zS_YKuXWW%fd-YCY>{mISZZhlhW91HIsZTCHXdtKCYUQe{{k1^@nnqmL zerfF7&2R%5ca8$(z5?Hk2^nRrX0y27BqH0ECh_LxVxuW)!MFx`EPKNR^Bof;M=!zV zP#e$=?ZC(Jh?Q#SJrMNeokOwi0|2gbnjZybzp##(eQjTPtkXoqoby8~L4|iCK0rO5 z2}|+vTB>T3B@KBBWdp^L{i3w`6*XHM@uN4Eu zJJgB%XocR>0$i=h&F+tQ4qY=Fgl41VbSDCwcp_yU*WUtnoH+OtJpL`2tA3cIUH1a9 zYTjrUzYulO7T{bLLDtfGcI>eR$X!$O${6Z>@G(0btP(QgXV6{#M#Df$I;HguiVYRa zpRExJ+|itnc=6vS>PyiTUzj$&u&&?xxJWyZ=O)n>Qr)g-^h5ru#N+?YdlA2E_Ny|e zM|!bk4S(-E;Ywm5 zsvM4GW1hY(b<1X&t9ytf%HEN;p{0F2Z-=wznv*n+U@aS`ETpyeTX~=!aj~oQaW`&k z9UD7PT~%I++AuA*MlVKvw~p=Y?d?-0Mn^wQHrq<4yniOG;6;(j8pr>K@rfN7Zg1Fn z*?iY9@i<#aQ|043pp|JKuA$G*9 zjc2|oN`tV-#|zXdZl&NF1E6;$0+y=UKOYoFir@Q+#k;ULS_tn%Kjd{?^^olPsKgML z%|)Ph#(SP}FUriU$4jB0gz)CdQt)(#r`V|I^E%lnLSoVh-5Whc`j;KO9Niopjz!B2 zjL}b{enXqV41TY_?{4?RczktG{G7S#>A_kVlOSxejAx1}@^Vjg5pPY}Jyo(+>(bt* z>#A(baw{Z^(9&!Of^Cw7C17d&L&=B3#EK_lPgk2z zgZ9#NBScABWc*xc;T*bYe_p`_?b&~$AF@}_P&0a10qGicV{(-FLt@HCp!%YQz?Os# z3f%=km6b;N6*}3{;zlBQE!NHQP}1;C{%< zf-wrS(ED#62Wv2;dA!i9zG7$la$vGly0Y}0xj~+Ip7Bk*-lNq%L?BVEGC|kW;bF_Y zNPioX$h#7=lKzX5Ob2oWw6nT>CtQ$eqvB@E45h-Gt=mR3ODT{pX3$|Va6t(d>x&Ac?j20y{P?}1&6T@=;%P9D8#`nvWT_GnhT{3sl$WGHRuETO_3aq z&*)zIbFk78j^_NX@=6}_wDNdx`HtS}){ zCGF9({;Al7QG9AB6V!d+#hyf-M4%{KD`?Okt8%UaVB@H~e-3pFW}G*72mEAaGg!|M z1L6`=)KA3Ys2Hdcf5+0M1jHJFqsc8&`+GM_L=2e_buJ)d;j%aVor*eJk4K}h$}0V` znVzC_a-zBZ>$&!3Oa_G;@cgoZziXm;jWu95YG4*^&^Ra3dQ%e-Q`N&1II{^uilPp3AZ)C=xd8$IEV**ZnvMyX=37QIdYPP8 zuJl<_?2}?ay>tFO{mQvNtI^cdD5#$%)oD}_PmWWXCJ}@@s0ypz1Hpdwn!XLUL(I4L z*@}K&UNt$Pt`Re*%uG&vxeH+5brDwx6pIgmeFqq;o@mFh&k;x_ulDm6Ixt4m`A|8y zsedjH$Gcgi*W35018U(ClcO*sGBaqVFvdT)e9TPz;UQqO$m`fz&d--W9;7@tdeLiQ zaerBh85a*M2_2Z>nOn_7t?+gI*es~b1s_>Of!Ac_+N_FohPY>kY=H#A4o;i!GfBv* z^7s4K)qtMYKT^=i_58I_A&I#()O0`bi@mFHr*SS%8Izk$jjJ>s0SOVx4h+qfpYnwu&@m5xFr{d%x)N{K*_y8cE#*Emc6Jf5@FmLDBR-v*=}=o*LxUIgm4A&xGZf74WnKSC?vVz z2WDl}G<3_l^QG6av)h}5#YPCEZIN%UmSAB_(ipX2Ik!}z_x8j-AvvaM{AcssCRY1H zX_JW8z36V#A$rcVpwoDAa$L)$2^8bGBzwZ3|pE+NHeK$GmatU|K^xehO?s2Uw?$kC3@c|3u zM8Z}B=XO23MgkTUB_%cm67A%jV2{Ll>@AUxf8;LeKQoo?eLehw6}V!*tY|PB-xIGe z3Qa6p25u$%F6@X$OL#EZz(c}t zCw#AW>azBfZCzb%A}TYvHrhM4oZ+nO`TA)Nx{dkqcjNs9XY~3kTvE5m+PTSK0M`Ks zI5HEmN8Rk1f3D0;OiuD1YhrU*_7W8t(`TouE?1hQp5+pi4;#$P#6)afP)+G-B=EJy zJ(AgB_=b4~qa5>pD5c4q`|HDiVw0o2k(2dICB+n8&fdBp{ z$3u$T@E`@{xb#Su*o&@bFGsK5|``^)giEmowLNxsoyIOhw?@SHnT{ zqHjj~i}=7PFRVM>18y8fS07}2hDNzOl0NFp`-C0B(X=h@EaiN1{MNjX=jeGq|Mq{J zTBaTdnnqztxNZ9Cn?nd#!-?k}h$81;?z)th%6>rmxse~VbVphN2f)=e!fv^?@O;AP zWM2S)g7*FvWLbfJ6uL~rLG2AwD}#QQyJ5nM&#u|zA`P;8%A&Mk&icyuO-G!T@mMr) zaxj$v1Uxxnyg#p-XA0a|-e+yc3Vm2vXwrV*{mCy|DqD-1dZj1hj#M^4fd;ZOR}0xo z^dbssJ;{v0hbKO^C`VFP1|4}#g0s4^94xx2Xu@HIH&%u>hsB;O*^fHy<)a8sCscpb z@4pug#?Pq}^FZ7gg%gX)H`<%*0IA8>Rn{>oT;hLEKJx9q|#OGI#lHdAx+&hT}-han+tRtkJ9uV z6I`v`MrxH??Nf&XeO(l(RhF7knob_nB`JApAiq{qcuJ?nAbJ}c>Y+h@TPiR90M|(^ zkq6n$$QwEOIuq}z32w&QKqGu+=#oza;(Qw*5gR>f;#_xllA}XMYZdTuMGrZM6p1R|VyczjqFo zB^(^Y*I&k#n7FAJ=QdjK3r*(!jRVaJQ*%5xW@wTHNaa`Bz^uq4P_Ys0*t)E^R9Gps2? zLj`Hg^l}H#U1nWx@{>T(L`D$so*ajw^X}~9^Qd7!s0#6fEpXGips=vA+Cx9hE6yt4 z%08g%WEklV2FHKx8&%$t!s8a6x!nZyJjVF|^jp$QQTO`_3IL6mTFpeXWNlwCUEyAj zHyJPwX0E3;AI=?|u{l?%Xth9E)#}SynQ!R_t=>kfL_*|gc@LTFZf1^YV(MoNer4X) zLFH`BB5jCw&)In2SnE9#oUeyY+Hluy8tR?DDpri|!E--*#I%Q4ZvG-=Ii(9=5um%C8hH+%#GqTa{A3Ox`U)k zE9dg)(AD*N@rr%@cE+sQOdR8~vs!sfLW%SIR7ks~-ZfdF;SOl>ZUZ-T3Li}k%mU&i z@OdhOA1&uv??B}(#s#;Qmkr3mb*3$k>B-A6mkvCD{*)CIbh8>?pPP%bsbU@>B)!Q{ zfjK^@rWam8g3I~xjg4`=m=o=3YV?cByVz0ZFJ#javWBuXZjAUi#tdC z;=#f|yyL2(y9x_UBx}z39_dm`Ogvl=sSS~~LQp1xi5uDQ1&PJh644`lSdQSGq> zewT2DbWE#9a`P{_&;~RC^Rd07@o+K|rCGm*jmmFjKPF5~ACyC!fTbU-a8=`lvs(DM zo*c5fvn4>v=HvnLuS~F`c^Bu>lcJn+;mjkEToecMrcGj*ti36NQ9Hppwky7 z!meKzqQuhI1q8FcR4F2;dktWm25b}U6?IHT&cq%4uUePm0^K@#4oV?(^INi8Gc`y# zqEDhjMm*<%?NwbpKF1vdwC*aa2gtXm%(F+56%z#tUJwPDd$#utV^w++=n^Et$rVSd zw;O$iIRuH30cLULwLf5nXkF8Xnu@x^8OWsK=-~{(o@wjDWlRz?hC9hGdp;2f7tVRZ zLLRw#Ovre4doM)?HK}PQ{Pf~}A3lC(^@a&@j~Mjf!=?uv`X4W}Oj@}EeaN`iG1U{O zy+{%{(pjcfA}EWrfD&;-Go!lknD&{!``?X}e@p*2WLeBz8-~u8AHwX%&*F^2-0M_b zE7Npy)pXQGIvTwHkaYfFkrDUI*QHA?ZFSXlfJE$g%J*W;?sWA&8&zk&a%5H;P?$h3 ztT8zu_FhUg>_v=geNc~dBnLV9yi~3(vwqa8(?+g?Lz=?^v|q#|g@&ebvrdi^&$9ii zo-MI!6}vP)(ms1JcINjxHNx3gN5ojJ&YpOR-NvBL8d%1OU7MKAtuDPEIt69Wg)|mb zr%!KUtnLH-bwJesU_c-F7TAO7-OyaB0 zT_XI_dCb}uV+cf# zJ8mh|2_3QHo|*1VeBw^Qz+_|f$VfsGW)B2;7{}}yV>tu5Bm~R#y*jiveQ%*@x51?C z1f9uE_&ILf0aHNvy+gjH^;vNGRQl};+(9@G?yRF?96T)s$NTQsIX1TDfV{bt+ItYB z9~|zIW^V3jBuj0TJN#O0bbVEWlbRyr(2RV4w;cvYQDfc6_|dSo!V>6Z(aNF0}a0EfbQN4A8v=5?71Wq)jH+H6KN zZddX#d;w?{OaF$k1hA=+*Er$twozkjhvCsw@^yZZF<7J)v9FD*2PWBNzKIegwLaLU2CL>`0HLpUZR})FN76)`&(Y0 z&-(V=_%Xo|dqwS@+|qe>Fx8twkb5eV2UGM!49wukC6GT=y+c;g9!?5A&M2}y$P+urDl_ky+7xqFx(SU@j(;e-(X?7Ct}f z7e*YcJb^%N#gbUII^;b}T6-*<-#f&Y^CB;;ka|@uZR|!>O`z@@LcHM}EvNHuAlu<7 zb#KwKXRPq?z{I>jb>sU&R!$tVAS^mqJ_O4C0EbeBS`WHBWq8dvE3Dck7nio$m{J`v zJvn_z*_WmRst*G3gHq~64MiPO6`nOMYhGs$-O~|KlhfZx;fCq+up#kI2yu}*CFO;d zMwJeP^TsW!TSmF#L=})-gT0GSB=goxBu_pVp`Z^|4Sir*3UjXqCR7Xv32Z2X5aLM1y z|ETRZvu~N19G#-~3eEAy>H`ySy(wB9y~EwcgkxCPd+R%@5HaesjcNaUi#qLNbF!r9 z!9GSJuUS^u_d2exE!_G{2}uC>0_aWdHK}FV63oxQAZ1;eNSZgNy`qyIu~jlYSe$GN zr|bHB925+^=-?97_TYG(g-T}>THB)R(U0L_cYNtbU-JmYc)X^t0`QQqv4Pa0d%g=_`Q&NOdR za*7fl=q_|KtHLvoG71XyBc4;}a{OIW)U4}VFl;p~B&vfJd`l4D>Y!;DN{78d8FXD_6vG#gK^cE(3KM|y^VU@^&S?-J7-i8 zj9|J!67DupdRb+A_}!ogXlmQDTAHJpXViNV6Yxe(%J5P2vrnUHq1Jl|IEqsk(}$oF zrms%x5?O|8-qa%95<4Z*@kN{85wXxVje)18s~D zJc79~R<=P+w*mEnc{6r~pLJEmovIa9VC*X=`?_zgP`k|ze#ME!3q3=tf2Y146E3O- zh-ypZQAgcaNhfo0SjX;W{*6YTsyh>}^b7_n>}lvF;>tL5%e2M z%IT+j;C=7*(6z+b>6_N6T9IE7peI?n#@{_fCz7uNLpTK0(|;{e1t^-R6^bSHRuNpu zxMfVfNlX_XJm4Ez`fRg6gPWJGg1-P!C8Zlq<2U(C>@1X{spb2jIWw{Zqo1ZOy@y`j z(2;7L&%JW28-uk7G$crtjF9H%rwql{U%-^h>cvW02;Yk2X%3BEwJ>hVW0k`Z!dx!E z$yGT`m{X5f{p_~(xMQH=>W8RsxDYKNZoi5g$&PJeMe|LXSQyaR$cmLgr5?sdyQ`NL zO}|HN{&Z5BgyeYxD}aL^V<|@Tiy6_|ko2-aK0)S)j@3RB1igtGm{U2Su?8L!%%`KL zoU{6G7*eU^iB@L)D<270{(DZj%&?SIqe>UnWknM$a%4GRfgB!bxef>HKo~uBj{bTY z;Ii~NlG!ej0!&nJRO4Zbn75oC51rXcF=wB506fP9Ol8`JmHi;N(=atgtsL1m$HiJr z{<2}0r%u2=;uPC*lct&RN|Y(-Tdc51!>3KXEW1Q^38S>KRCm6CQ<^{@k>UZc=HV=T#)) z?E$NHb1sl+E8>Voos2@w(ps|dFWrCKVa!DiE%6{>WrUi$^0F~H2R>*+ZYf?+X~hy9 zc={MLkzc$hf9#u}WMn%Obm~AHcP`pscP_CWh3KRiPG4nootwNv8@oEXYr|pY2CS|L z-+;f=oZqcFH|Vxsx~Cp|-#vQ3)}X2ydnhaCQ}Jj3^47)7NgjV;yCX$Zp-12}9(W5? zSjrRU&`>D9o+;zHjswzReV5UIjP@&Xh=06J!64kBXglhJOYCQyKtll6k_AiY0-K9J zl(;@)sQY8ybC z4dEeJtfTSlJ~sQt`Qn*4A3C8>WI>>alsYy-v&A2Wi)-40B87@M=`T}r6PP|kTAd+0 zD2>MaDTX151=$atZ^m@1sT$wr5^??~Tuy2u$y0g0>YhOW5^)*+_l=nzw&XpMVhg z*0n#+lIAC8hCOAQnN1CyG&Yzd4is6wn#mIgq`D-H-O0_gs0}pO2JC$YFrTKoVTKR$ ze{BbMb`=Q~&AWeevD;ASf@w(2zI4$2O<}JV{P}m0Qiq@H!93%SI{b=3a>p;84==e{ zUZ$i=E;Ot%#}1s~zxKF5Vt+Z>-;AB3k^YzO(5jC*09@b^*L{(Te>WEX)BZoS9se)B z$HTnzTK-q|NfzK_?UzWn_c87<;>1;Cj(KftCimlq1)afg@`jKEhBT~7Bes7?&QEfX zs!e%Hp5-62qVYFx(9RUdTEhUJ`Aepxr5BIRM|8Kbm(UV#u-cbwq7q)I3;$iLbJLqU z)BuHq!S<^%R{bK0B0#5*3%SN2zE?K__YYAc<(@-VS)SPJWIc6z+&{(lHw*KU2|oc& zIouo|=5VC%o=r2}?`lOKJdl32ZC-8jQ+>_qhx^Z779peb(_cDqbGmCKBmUbva~wR% zwW*BbbFDeRsz^+{W~>XDG)0%p%h8-=pu}twwORloZU^BADN}Zi81hcJ7%XluHOI}{ zjkk$EYTE^kT^pBy&38sYthB0+eMpWJKXcRO~#1j)>Fj(TJWU7 ztOkNAlZ-iRBnWx{=rrzL64vHt7D~*(}WP9j(#xxnawHl|3PrDJ} zbpzb682Ao$)!H(%yaJ1k1t=Fjbmj3Z_TR z^5%t*YZ->&aTS&(zd(xRW5=5V9a{|G;E%kNFPeWuCLyOpO3NN%Crj-u^%0XQb+LmY zV{8uJ5#+o*PSm62famdk0_$*vuPJpL>J_fLoHtEALBnrQ6!CINancsy`vkYL@%iH` zU-nC?69!SHmuZTrE1=x3^;b~8`U@3~;7T;^V*TtNDmCTCWONkl7sZrl>jz0(?wd=}=F5fTl&M$F z>~}cHD0KuHlIA^n-K1cMS!G^4fBW*?1XI29;X&*Cl|41IPBg>>qK&R% zrt565i^?qYaY%v_f9>o)jMMDVFu&zqT>{qot{6KPel1{zcH3#okaO|pC3zSeY2 zPmyfnF8}ySWXw9fZ=$OjRydes!zIeL;|nSqu+5sEghL}X(V%rRA!sgd17KtNFEhbr zN$bSpDTyRZWoZppd!_mue~axtE&Z2}GiIn~IQNa?`qK3|VSic?TpnrUXd=UA(#L3K zG64DrqGO0r7t@aqU6p+3{9pl?+vI2qzKm)iodmDXeP1!w?e3RTssBUrcZroxPPBEs zTA;qvnnIu#9yuC*2QB>K@ z!TjQm+0Ut(8i0p~4~{dTg9A1PQnemoeMM+sSG^rWeR&sw{d2Ys{T91LAlCvbi&OkY zcN6<1_Jg!~JhHv1{jVKED#y34V&W2XyV^q4kMXEolvu+`r*5vM%q%4^57^R&g2$)qktP!z zxOrk&QzqBr#M#K_O2GnxnNr!K+#+g{3ou-G&79TuxqySUN(xHRoRe&WMA(A}ltMRh zr;Hd(uAa~n#}^74mKrnx4h)%ozI*fX$ubn5J5vVyp16uV z-#sFkIK2=-5=)*$V5(XME_YBV)jD~@adhkXf5Qi{xJX-!4S^8zC zS~C=xoS`Ok_c{R^t`W+v6j=l{7|q8A@mo1^c9#jD)p#h&`iM8wtqkv9k0h4ycrd?? zmNm7OE8r-CxS-kNaD1DTifbXHI{a4xL>`kl!)hmoDEydGv7VVs+GgnHG2tV*GOyB_ zRp01Am!SK#EKep^Ih;=&%nD(NPLQhw$BT;BXKVaIO^-vHJXs94!c3O(=U3<`^6r-%J!$aPvcNYHQKbEWljW^TQ+H|y5 zlxz(ic7v!_@p&?~z|311bPx8d{!Ff6mEYT3nPUIxxMF>MRbE{&YCyey^s3iblF>aL zEtm@K75WQ&UYf2=pe&Gfq_Hz}On1iq=(DkFinQ!3;m>O`IQq zDaub`iya^_nQ#+Ak$cMV`rii?-ju;#W+lI(<1dGwwnhBEARPaH4uSLfw=s4@kX;v9 z+li+-zV6lZu;phW40f3xLL_Q+xF^$JI0+hwE{a%|&W_D;0bCV+ydrtcm@X}b(>sXr z#Jzf+(G5Bi=FBXJ8msS={JJ=`T8G&w&abh!tq4P_Mpqlto4G)kF)mLx<}F*bnhi|$ z;>LZDN=>c4E8z#f46NSzwv*ge{P6Gh|6kw#M_@g*<-ejIUIRb<=PQslyZ!pjHFqUS zejMfa>eB*$tJd(-jW;Dx%Pq9E+cPP_a3c&)5OL7$JO*Pv4(RUgH$6?Bs+#blZ^+p; zF&U!~&s(<978RkA-_WL!ZzFCzy80Tk%5xF=bkq%D#tFn9WfTG99F@i!b~xI1%x)m` znMGHF0iUU5()`ITx32~_&zQW^uTUQ+HvW**E?~;ixj)8G-oBE#Nvd0Y^bme~tu_AJ z`s3^6((>ZUaX~e};PFK!mkk@?dz7+R*1hgQm-+hL=^!T+$rZ+HRzpPcVai25&uru7 zvwJqV2R2E5i&7Cttg_r3p z`j3xdGW)>~{P5DgRBMihJe;B!OtAP*K8zN2!1i~R91>lscL@FYEt%`lg-<^%BJe? zJ*0;s5;*zhIshMLUxWm}WBtzefb65ClTd*hicH(Caz^=V-Dzdg&cyfX8M{Zc?PaL% zDb#gB(mydt3_9h*LcSR!;thg+qBR3 z?7yQ_9HnF%m!ZVy2k#6@Ah6cpV@VgTCxVacEW&7z{GSr^I#G!o}LXi+!Z9Kxa-T?Vel{m+FYIHF+o za1Hh5&>P01sA4!&WqtK8wS%Zf+U}xKnE`t|4*GXzFa}-Hd%5F5*?>OduyAFiwIa|a zmDD6n!lX?yilCglO*f_g)Mgbkhx4~~$##UR01jY_N`vSVmT?(%UBefZaj%JtrJ6Ws zQl7+s&UUV8P#%?&ug~AFHHUgK{y2wMpB*5J{D?cZ`-T^b2ck=cwYVSx5PvMtP_H z2=KOhkI*->G$iPc&&o{lp;W5GYF?qMN6qcV-GfX_ElsouxW;g(=^W+pi=pN`9DPE{ zqhM(KQqWcJI>-IQPHf|{8Cunmn3!$E#N)Iu((;-)GQy5Dw|Dbl3nIK8{WNmQrpksc zx>{x%``}wdUD|uN4s3?>J(7rrW{a*{TFHzNkNvqordBW4+rz6_xYF&qzg4pQR481q z=#sDRDn|{>Du9?{TBFbcnpr2KD-LSd!Uj0bH!dkH%KzO&K#Yugl2y~BIB1JJ3N3zA z(KWC(Ys8wr&r?$J%{^PZ#6K}9sRx<~LS{ZUI4Gl<&Z(~VSSo5ie@Menrc8W_22o)Ih-R+0N-n&=7LW02 zzVZ(4p76hO)vPh-H7b)U!pUaOX6Il?R$-;1<^p2)%oZZ{Q7ZE5ws>}V^p$KsK|c6B z0P=@KXD8W7Q&KNTN@bM=%M{1H*1+&)Cb)ghiL*`$-?)BtU!01$e%+VBYxP#XPiO(d zsaLwShd==Xt=P6+T@j&%J%KNDsFp?hMr@HBCp@Bl~< zoVU}GxtW(4fF)67TvA;9dxu8y-<+FxMsWfVDN1pT=-h;r5&~=zfIGh&ExX;u6Erh^ z4fXsX337++h|2e!Uw`!unJH`x8yptb%5gtJ(@n7cAyIQ-h&Nl^4s|NZjP>J99Lt3w z6F2}6V8KmY>$}Q{CQ-Wh^%ItAe0D!x9eSGID-cq6^m3?ksYd{Q=6;wRk4C0koR&d} zo)N@)p@4MDNq2|xgy@O3yRdH(bX4tZZFd`l3mtm?dR30S22OwaCb#)r+Rw&%3W`Bx zhl1K2;GUPdtL8@DA)~UOL?+2}cgkAa{Jc&WPHAJ_e%v>tZX<6EjHMnhkbK|em09VW z>ZIzKbX%+PM6BR|v6odd>biJq`>Au!ai$n}-uE7AGd znLS(HwJ8Q%Ur2Jd<)43*GYLrm)gB`D`d=q*g8Ya4QZ>v!Tgq7!=h0vY4h;{=nu)b#UGMk!`guVH32=_ zzY_R8aC+$T*5vcG&6x!7uCh;!W&zVrr~ymu+sGtA5JDe-h^NaQ^S zHd79R44Nu&_f}tT#Np1oYWv$e&Yhxr!CD*zepW!Mqw(@(8;|;&+U?fuVZW6*wi&no zd=D>t_Z;ZSil-mSrYiMLn8(9CWb{qD9AFafMdeS|e#YM{%yU>`>Q!VL{k18^*?K~O z0yVT8#hNBrt5jCJ2>lMjJ+!;>@H+4dKg@}EKM-$FRrKB@;G$A2T;ssixL7~FTUbD0C) zqo0n8E~4tQ94`+z&?XOSYdOS<9<2Hv2$^AQ91JTPGSqnc9mcmxO$&kUc^`vo12H{s z@p!7j%djTwGy2aLQkPj4=Qc&ZYvNCK&$8b71|(ng7hUxQoLq~X$LpTQ-5|bw$wG#o z?#f3>3m6iBCe{Mw-u%4#?FmKXH-?-;4JshxyA0NUEQxbrP}$~X<9NQ(>n&tlzQp!7 zfMHH#|9q2kp8`1W2y^=UU%yi>hL06#E}TffFbCd-crir}cqo$lpKx9DdTG;@H~Xd8 z8QoetHe2C_bxH?6!LngI8a9^q=Z1)$Pqy}$Y~p%8gyIYb8gZi>C>CMr#1#ivnE<#2 z%O|?@1qg*IXX!c^pQIXlif#yakvw|!?q5~;KMusUbGcirr_sOt9M`bR82+pvf#~@h80F?2w|yiD zUli&DOrvg=XmPaBtynQOFYkcA;vRiN1016pRm8$pl;_Vg75uu-O0x4U26>728ZEMR zjiguB>&v^-q#15S<7($BMP5gxX}`Q;9gH;e)!tyc=i`I??0IDCA;5#3T~--%@l=Eo9 z&wv%wz?3+NAn<|Xn4ITEM*o6U{*Fs&voMV_W_@+Yj%ruipJ+y>){TIMY2VbwfMHy_ zAXoVYwrjOMCnvEvj=I2-1F)4+-}aS+pUkq&XWVxVDq*X2e5U*j`K$?S&}DI&VJ`q~ zJ^dNYSq&0CS7~@w>PIK;V=2V@$yt-=O@x8#Rvm*me8_^gdYyH-(}CW1Ammb z5V|NDVwlJH+r~}kX+E#hxNfTE^U zi^91xYCik%PJF7PlJ4&88w{{+oCfN5I>C|o0J2#8PtY;z7do!K z!O3`{8y77E43=QAl%8L&Uw2ovsZ4}Qri6)KC*&!;#@Vj@)IeNYky|oG zXANY3%q{S6O>XzcIJUsHFwo}Qp5y~lpffT2L}$=$uJSC(+2z+jk{OSoA(>NRkD8tv zbCZYbw5M^@7-cG2EM!hpRE!g-)O4s|7Rwwo`PRK)fzd}+tVh|a$wb=R$jXi{x(y9+ z-*5EAwa;N|T@^l$yt5C;-Xisnuqs`RP~tH4Z9CNOif_VF^~t{o2f3(FwQo@jHvFS( znq%#PmCYuy_D)z+q$$ykRtAlFsM&cIVKVIIthD|r0k{7nL%jd>BBNMVighO^O?kFS& zG9LOd$el6{{B{7c{`M939X-zb>L?w&a48&ix7u*7%gk!s@A`=tHL41g$Z)2BVBa-T zFzMDG-}uG-A(gKbpX~IeGH2KDL2HJ8J|fAs#!6CH75)9>m`0yfd(@bY3AkPZgA4X5%lG~6|tWHp5SEdSuGAk zH8YHXW5GpeN%hfea?GdFNM`jq&Db%!7dQjlO{dQw&cwhq|DW*VcWI0Q1SSGRM&Ek( zFjPS&%_EM@queqVG4ISi{pY^xCPjK{3T;0uLaE*TGm1I7eQ{u!ruq-b;G#rdBPAk^ z$%h)nm9w4>qP;|}Q(|dQ*&vSQdwmTX^8&lWy$p3+_30kz(803KpAHF0F){IujYUj= z?pO8jNtC2WoBIARs8Sl-{6l10K;nCw<*L-4LULTBC%afKF&1~2G52^}Jx96jSzhx8 zV{1plZ?skO+pnuNSQ7cPFexpqmyEM7(i6^3FcHlR5sT}`!J_2~EuB&)TW0|qp&2U= z`i5lS+%Iw`DX}cb9AIYiJz~S!JiyhGP-jwQS5eL7yFIsk3T}UB9{(xT`a6Qs)t+0} zcapnG7Ha@jV7v6B5t*AUpLe6Fsrqm#Rp$(J2WGIBKPS{yJWpbH9o}tN#70q^@{jfN zd;FhR-VALsqZ)ygDkpmk_U%>iwH(GZe!aRbQ36sq6@}FWA>$PX_cS^bL^Zom0l`#J zhg`Bc)0m`GwbI6T_?PN=yvnTd3r%(o4nZF$p-tO)=Z_^5A_DALKSPUuR%>Z&V>Fco zywKt?6wvM*jQwI9bGnw@K8MbuyThCA0OoMt=i)Ras|Dc#&ktdzc}nxv>-4vZwFUl= z`0T5u{CH{L_>E~WWJXE!l2y8GUcvI2b28etW(8GmJmjKWW;Fg=6Z7R!f4+Qyz zmiU|~r$;!PJQmM9EfEoU_O|qvsjHxWYn-^!3)^}uG}DdxtCzHoSHqHXVL(+v`w4ta z`>Rp?C`Df>{a0FdN5p}_<&8r+dBfl_-BxMYr9kg?l9CN&ePewpXSSNa9Pwt|fC!Uz zrPh8&>6qE6yr9DVO%3Xodkg*Y=XYYf@kPx9nd0lrTOtwZ`6}>IU)rE$q=iTH7VOe>APP%z|IUDfz&N z3$te$20ni`aW)$hX_qgt(3NjDZ~Zt?w==zKgGt2*)ZAeCVe!xj?_5RSdrsm&pKug3 zPG>WXqSvp2=M~mzo2nl(BZus!@4L^a8aHL$9#ZODnkaDvuq5*fnTbtM$c}bx@LQ_S zI5Au1bj)b3o2DLU39alv*itDvpv;U0u--Cc(~}cDdLw{;hRTrjN1>6;C-g7rO42%% zG|+DKzqXCG6W<%>bR+SrZ-Ftv9ZPYCbDIG2oem{|m-LJB4oH`{p4?6(-qFLwTtQ{z zg5oazuE(ihKuU+8!4I^c&VlOC8J0TL!FA4>{VH@f!{FNg%4b~LXb`=~w(;~U3+0}# zH+-q3_MD{hUHQjEg<;i*?5tW$9xu-0zHKXO!@jt4d-oBAY0(1L?$M>rYb(qfU)jFV z$0+BX=f8N*T~#ni3fC63t-&t){43dYZ$rrog6HmoRCC-_2WI>KP091RQ5YuKi1e9x<9%xWZ%|3?ERK`q5FqqD)kRZ0l=DsrOH63iN&?31Zi)~-k)|P+nntu7VpcGg z$O>8OyP{*(LcJ;DwG{s4BDH$jkV!jK$;MV616KoyRxG|=3 zc*ap#s^dcMJf%)zt?AB=`&Sc#HHV+UD6;E6=kqjE!i+Pu-#++$L*nqHI`7$BmGq!% z5&}OEwDJK36%Z2lZ)NHQ^}2Uutj#ok!EJ0zzj|l)Tj|R!Ca_!+4Wz4rH{Q=MGB)C5 z?mbGol`-`vat9|=K_Iv;uS254-7HGJmUj5BdQ%W3x1Q)%F9qWKI6n9L!Jl6j@y;tK zO)FIn{U3$?bVGG#iKin*bC3=V;W4(R_}c>z9j_WYn(VI}EZBDf{t|x?VnwI%VvNF9 zZ~OJ+mlJmgIXavazkPs-wuPit>93b3vX`0hnr%BkT^O*Z*e zMgZK$tyA7j6V+(16|f1}WwYGW$5%0*bQs*36Ze{z(k;gt+)0{-hBSA=A6u{S-ECj{ zXy+(MNK%SWj~3#L>2#u{fMk!C=XW5uR|(ZRzPUqV3f-!iyo;L48?`^a+pNqx+gOg!)G zvc;%XN1WKHwF&HVVUx|4!>)7!)ICPh?X+{kl}TpLNEE;1{Zdsi?7?E?WaF5aGHnN9 z+%TWa+fAEq8cUHtvZ`9K8qC}78zTHUdsmLo80V+xs55d5hqZW2d`ua?i?*6;g*J#A zpF7pQjsr2FH*ZBUZK~Mrz21StYt0Ah0-6O?dtWP2<+YAQh$^1>N~*~BsZF_+H1!+D zn5%?jT_q%#mG$W<)0CuS0%f9yG7G0swX>x^9 zCF>w{VFsUQn}GO*Dvoq9&h6mm2(gqh>z8klx1L^A&*W2}h*>}KiPr+p>6e*0fH{SDhsY%#`{G8ehq#q(>N}px2!!Fu zL>|^)38J&rp4d)NlaN8A;K(q7e<_8v<=jF5YS8q ztID;<1)$%RouF9ZnM&k$VV<-t&g`MGVW;r6<`fWDs0J1GMOwd!qhI_du4;k0Qv}5$ zY7QS@Out{UTH7|B#0b#-$;9EYZMeqD$R1?TjP_dVrn6s}mS)Xn8CGbZAl!tL-0iZd zqRQmD74=IYA<;wqrSal&)#k6kQ&7)8a;4NcJjpMjIgHa=gZRLLXHB~eBDZ@rjLOUX z=zrNBqQ|(a8dW2~$gUs_f5#7JlR*S#y}}a7ikb^!flh{FY_)|&aeRVcY*~X*NfCr=@4Ij_)1wDd)6F5?Q9B-W zgFeg;&JLIfOi=rDP~Qfp_MFhr=qy2aO}gPtr8TqZgHAwvlAc(X`q-z3e~IQ|iV~hQ zHj+(_TKB-V85$0vlz-%hjQ!Y)gMKyw5~I*<-$$sWKmTWI#&LN1pj1iMdiV(Hw`c*-oYfVjLv2)vn_g9^j95= z2q(X3QSM>cdx;%Z*cc=Gr!yqG{&+aKFRKZ7%9_% zIsT&IPEtRM|J>77uV7+-7|AV}m%PqrV{TNMeZ-^-%q zEo3bs@yklNC|!qp602G`rf1ZqbAlI-a9P`6!nVxC!zt0_eQ%KBVX z@X6!~uX<8qQijfBxOlyCG~I{u-Rg$v29eaeT%=gbgqaGghGCH*A0k?8yS&KG+Chk} zh)i^b8;MeBNHhKPA7kf#=WTW~glIGZ& zCYnZfGj?pf>z{+GOn?U1CN0QU%OJ})k23;!<6=wqK9}_UBe1XN?ogbd#htXp2^utLi@OsXzT)n##ogU+*7~e_?{&`J`#$^L^PK#Vym_X+Q^p)) z%rSn06I$P0q!)Qbxws|~c2@1UbqF;Q{=M99*FCMrM|Xf7o9o`oCHYyx72?Su6|ur_ zG72QSFv_qyY4Q%>G|)`W;$U_MaF5k(q(2oy63qf zLdwjWI->2FP-oYgr&$@5o+?da6v>&~rri-15uQwjZ$%JLVQR}7y3!i?Xa@!~K?aVr zv}MvyvI)HdV8nm^xAXGf0KI)pU;jjC&3}RZetuWOL8Y)~;FACU4$1{o9uR>;L{cmJM88mnrGW(;&L0 z{jBe}dHc{OMkA>v9FuV>J zC$Z^4{R6?7J=iFS1ldp9mVCRfON#xKj%rxwVhWahLbSoYAb4JI5?O6QjTW|o6^AP;+;ox&JL{!;B} zK<08y1Hs^LR;+d$S5_AoR+`gm64NCn-G0tN!y1=adOzTvo1f<6Pjs`xKDv)AG;(d$ zE0F*2FKLK7vD_5vVHV6epj%1vYqp9Em4PTyfDoKn;gght3D_vuU|LpLPvI|;_}j=^ z8pgexl^qQnt!>&UAQRfx_P76+jz2C8_%hqcCppTW4G${9ZpY#YDj+WX%T~9W0x?l)TAf3m>7wP zX~m^aI@FgHbPa}08Em8Ak*t$!D(o5M;t9D~F@XMZ(-}QLziJ1E%a!Kh~x*&~bvC)-3#3pOmjPY+Di15WIXGZ@h z>y!=xV@z|QBE74Wktex1hpG+86fnzt$$RbJx;99F1nc-5P=}Z-nT!)+)1Vw-tD?X& z4Ntd^d2WL}+uJXonUy>8ixbasT{u5PVAfE-N&VI{)cu5prc$s!0`up^`_2|x@rEv7 zx7~#2^9XG0Y|)AWSCj^y9RN59_;*wPpV}lp!ARiEY^g!6b0`Ic2OB=8(-g^P-S)!WR%As7c4g1;XP!@g=d(#Z2k{?^Rd2n2un5%Ih z+rLfBv18tVp_*`silWqbggyXBCrF&I>gVo|xxTT`Tkx;%_3yXzKPrIwvHZLrv1hM; zG);NE_3siM5b8ksq<2?^fAU5j=3P_HJK$waQA>M@CO#Jb(Z8Mes8VS5%4huVy4W)7 z=ggn0mTq`pnAoXjRm325ShAyJ3mDZj%~w_k6)PA5&Q>Q=K25yB3v*AvN}j}8)4T&< z2<}=6xf=;5SzNO@1O7)-h<9?ym|56l#u?IDI5*{NNU;3^Coz_%-lC-2^!VP_-=!4l zN=1l+D?xVCpNORAR-4z5q8DS0%~-#5?*sFtCFJ$w=m+h_2pk>nH*+GqTC_6dig579 zL|0&f+2TwUHVghCPoMS|pPHDaT z$&$>8M_6^`>g$f@5Tib*R}WDEs>w(ybn`4nI&qkTD!T(_1p`nHabe{gzsxrI88?Ms zDMeq!?aT35y^YGZ_rGuj@tyNNs-3Jh2oHUY{uDz}MAB-7Nm6xe)7=!^7a4rsaVB?- zKFt7;1oHtEI!>svbFCl|KHYW#Ig#ImrdhfD!Yisy^D1w5N*9%J!TO*?gRgm~g`xXC^CDFya6eANgz-_S4Jb!)|bh!$V$*BvGhYmxr_ zVLH1!PbW`i=i@^XAzDt!HXT2$h;VfIzgPS(==xo6_!_pzN`EEj9N22-Z3MwCgWPeTrfzh5O+WsdbUA@VvpOR z(3|DoQl|3_p$uB66ji-Z`s@iK&!vO&9k8*iz?2TexHPd~xiDsopMDnJxF~z?5ju6&YF9V^E>#R?h9H`<%vQ#oQkwGd z%90%&Eq@^>{Tk4W)!-LANI?DPe@DgvSZ^=I&3EPmhp@ELR5o?x)ewqTelEJyT8!gq z6GGdnQCpHP0tU)+BqY+JTXjyw|i|zS}ll0QV zTs%2Ou9tH0LtfsZ#^~Y^MFPG=9Jjf#MX76(Qg5u{aFP<^qNa49dyWTEDE~IP`bJl7 z_0x`rUHoc%Q(|t+GK;LXDOokoW+DV)&S4opIlQ2d4cDeClrS8?7uA;qrWu!6nbw4x z0i(R^U+qggL2hbI`+Hm8KNqq|8+ag6P~*0S{b~uq8)s$DVLaP<5M#tEsbv&kRgBaP z{rV|JY%xg*`c%9nS;F(HEg8=Vu}Rv`V^g1ZnbwP?H5Q|%elm@Wif^|($lA#zW_4SP z;8x9or)pSc)36Yhwj%r2&Ws4{SQfU^^fKKAAV^folTkX|jCw(J8NNe3U8`@REkSCF zt^~|6J7?mIi1!M6gTXaup?Cz;Z8ljU;>#uW+umS%)mjj`NZ?aO+14w^D6POS0+Oeq znl{{-k1?kJBK;zpeBij#rP2lz@BaM+|KtCCSl$#!CI9@!Q8vHRmik9|i22k4V2Z4Ejjqb*I6)!$hMgHoWSh*!I#yl4NMxX1K1SX#P?Pl=Pmcw4{$6&A%)!r+TFvPKZo4sY`=B05)9(#eM#rCWy zlf@TI(fK^Evq;Lb%EL+yVYcwBvH|&r3OKktuMy=SDhu)Qf@v zJ;>>*{3U~xp2J+b(vbKU&O za~Aw$RB7|Vp)1GgW8(J_uxV9uu=1)l9Yz(mG|fB~#Xd~bJuPNA`6G9w;w9I}Ci+Bg zuz=qYPAA)X)?@q=+;wX;?|7-Gp>ue84t)uETe4&i65k!u_vL7Pz6CQz7F|j5c(p_| z2QBwe7}J9;*9st51Q8#VMr*&lJz>#6%dGCuPSj~ZB|Cz*hX9>vx~nhJb%s4Mso*@O zAt7Zw30hd>Jb+6V!}$oNZiZ;p1{i}RL`YaA1fSV+I#wOwU^9W2ffOsXNK+za%Swfa zbh!Hw5P2_o>ZadHAP|eio4Bd#^=se9p#{&y-%`>nr@jx&l8yY-G4>T0lR+?38=!tj z?*DRj#$BboV~@#HwoAP`J_XzIB*KLA#~Zk1)MWaA88wOw^;}7iZp8YTxJo$k>6i%G z8jfuk2XnEMk`%pNfJHS0M#Vqrm|>?SqA%|V)AzLlvUbryFtaN%FQJGf7o$XphSx#N zOdG&PeFsDVV|!|g>aL|MHar+ERwOt5`2<)f1+qOr5VWQ3u$^+{ed~Rs%AYkBk!pxj zGR+YMj;}0%H~>3@C|(VWL>EX;R1*hk*{qT<{l}&IpJ2s3vxbKETCJ~6y<)~ziQ zvui_|A)k%LuAi(kW;|nZr*iC5CXEYt@+if(fHv+52sz65u@;jwF*pMD++-|;Az-Gz zhH}b@D{Z%QDOinXiHWPcf>y!|wJdt+XWWTmS(ezk>&It)pp3xYTzRq;2byi=ZL!06 zO!tjZroov}x^^5O^7#)HkUM}>ukc9D+}emPIOUr)O6}qZ8u}p(Z0-u`?N9>f4CUXA7*2KM#qOHH1Gk z7!`fHY6)1M{=ry+UkN|*QnFwfV#gW5+{et&E=#%>8@9kuSCm!v&ujLNvkL1W!JA%M z#^EmQQH!P4)KvH*9(Jv42&6u0Wv36MZO`(58*-;G>wEV=IqcguvyMRr&{R`8fB1}g zgbR&43o%+r|64emrVAX;!HENx&_)MdT{p=JSD$5CQnMRSkA3=XG&mQ!?L^|=ijkxN zG->V%wINv5foTq*@Ogfey?gL*+CefSR^L@$Eq;}!=gReUXZVTO>I(@gN$H2)?ax__S`(&&lu44DVN7c~+wfk$sPTDd(PK zR9pb0VT-FnaC6#?4Tt000aX2>HgQMc9V3225@3z2N&`wb;Bl4eF=-o9$#kNLkZm$O zW|A6rDmQ0dRPG~t97Tu*ySEvN&s5lV%v_+a@*m$LX-?=~oR{ph%g!M0!|$>ni6&hn z?I`iU$0An#k-TYRO|^(Z2FG@K9s(d9R2ZN~OK-9{Sg*XmWv(psx6I?l$K)Se7FWY% zf*{kr!v)V46_Dn-SJ^$M)5v}i%X^7!SJaG17{aN!BhH`Lr+&^cIj8fftwQezPwF8O z+({~7`lDaJN6iZxr-b6c<2+Xbj0+r({AqHS9Hp!t`o#Crh?!4RZ;R=E;rqvq z;#rn5r^XaG3WUTAS!lroQ4uGsO-yG*McpL5Hs=^Koc4wd-Vk@tv71Cohnu6KlAb+J z@93$gOTT>5dIva@K5=u3aq#raJ;Uoh?PK)i@t!!x(Tew~?>@wOWGd^{rd4Xhlcnn@ zkx?L7jflenMPF68`mG`9Wh*nOQIG~W6vv~h!kinP3MlPLAUOn`C1ImMJkK@Bi7`lx zY3%X`SjH^JmSKrAeZ)&K7_JTX%zQ8qc4>Y@5%18Dy>|y_36;77)T+fKbZnLI+1Wsu z{ithqwy1kG22u**uR*{B#h13SJl;FtfK&GCw9jY_p7?hF2kB=U_`o1YfP`=apOIgJ zQld;l**1tMf*8-UQ?TPI%Cy5O`&N5DvCxDF{-eV0Dj0}znBpEWGD+(8>bpHWB6@Kf z?#8`|7os?TLUM>J9~l z<1_z5(ft`%=?tVb`=2 zsTrdcn_&Hu;ME_^|LdQZ!>W!kba6&69W6G%eY>ub4p*|7NcN?jm6{OR>Pzw2M1!P! z-|a8C9f38TK5cfbud|A5qY&{QET-*2M2HBRN;WLS`}P!Md+6$0E#PF5HW!ZbY+r#F zxN%{D>rhAwrl}|eqNNf0M?{}0wDo`8izhv4i=&luVP4+@+5QUI_Xg+V@44@%YGEg1S1i&tRrBy2J=(vACr*)e_ zH>B=yb-!2gB;V$S><&<;cM_<89|}!tJcHd9;{$C*jUe3QO4Kd<(i?l3kHigx@N%S= zeQq92nXa-*{l?>X8?;a@RvV%{Dj;#CoZv9XmFI54SCjSRqTM)}+2kB6=E|np8N=Uh z%HBGz;)PLGzIUKuS23i*9t8I6!vxkXe_h!iJ{mlPfi3d*K&x&Mun*dqk8@^=168Ag zyF9wLCH%Cw();713g9}crfc}Nc)jq8GLI>_G%uaHwfW7<@|_*tODGB2gDEW;9}TH-pp@+K9kND<-WAJN)Q!cDW;}y%6i=tmQzzZ=&owf3)_+U zF%uPPslnz*R&6_?6C|2mNHx(BX5FXrl|YUw3pr7b0u8!YZ zIWHlb^@4e?iOb5ZldT=K;mi~3qy}rUF-Y~S6-fY0*2GP4-hIW@adz#jV!>&Ar2^&`&*?~BzNmK~lC%#6LA*go(# z07Hg$eriW^FA90n+Oxi!7gb;^q}pbm@ebjT)_d4@%{N{HeGVGk3XAkdQ)*YwUpWv@ zrB+zVtYjlsAb_bllBJqBPJak(7aq*Gu7IjXVY{qdayG^kZQ*>i|4T^xw@Vc*Wxkf= zTIz%6>E7mxgOBd=S0#-tzG(B-x8Jb{0638f5!ghCQ|#yC>yJ+w~@=fnXc|T>j>kcr=C!F+khhh*#G%s@&ggc|}0J#p2fBXE$Ag1bN z&3LM`)2DTcM>%iJu}>Rv#;S+h@E;PENQ|^*z%aO$&UJT62ce6!FVH*wDD+>F(lHs? z5q@1so+Io-(vHAZY;3|IgW6aVQ?Vobia`|%C?lQi&u^zxEN7%@Q?KgQPOPaGAVI+N z)OOY=SB#tUNPPE94ErO>q*K*-hX(3PfL(3AOObA-ey`U`%b|jjsGBp&YbgheBxN35 zhbX{p>k5p|#6=h?ud@kbG_3hSR+&S?5Fp%#phbM0^7Ov<-VegnK8?%A^4n;Q5A_mw zP1lKpi&=mBFC{$Dl+{aCBfQ-(Ea19z?G0X-*ONwWyZ!-o=tseG zs_V&a;dh-kU%V(1#ao-JZJl-+-{7(S>ns2~ivMN;e&U&;?JP&Wfs?pF07i#HoIYKo z=G>=lFFZ;bleCH8ix&PI1?vu7d;+&6qB92a?RF^xlV(H8P_mPB5+bR{u(QcXMYM_Y{dM z+k6UO?`xh0%!er~wL`K6++? zAj1unYs$cX*n1v*Pt5%D9Uw7k>{cXYmf8mNGN$=RdD$ZJzNN9sT(e-tyH98AM?kdG z4xtsoV?f!T#OiPv_A|jz(Uz%u)W;$1W4%lks-KpMtU+V5?_D{}^CR|C zJ0&@p*M*u{*@B7ulVfz2*%?kOAOTefiZ+~P62LF&LR!6f{5Xa^Hr6zkSTI0zwS(&O&j`QZ`r)}u@XY} zeayUnBX1!mrSvow-m$swYDX!1T!Dg^>dW(2?TcPdBB|^6|c4b1Lf&zDaEAvf= zlZz_D^-9d(mVs*NcI@a()OXrOHsq&+eaZAnE9X67WyyUX;-kw(rSbz?>tUDu&5D2- zmjc(&LE|m@6?0^pAG+lk4=~)dg%yHI{e9f~Uo&WU0W3wonlq*OYXd@9FT# z2s>2pNsC)=h;fLY*c>!csVvM9JCvL^g#D|uBO6wEFKtJn8Z^PSnECwKC0dNW)4*|BYR3MCSg%{{Bv6D9 z#cfJonBSH#(DVMe#Qr$p{`_&h`IVc9ryGF zvjK-SBAl$hPG9;5emr{piJPA8XfhTu>g46b*?W?4A=7{F411Xp!1+Ae1Xa{zi zAZA@A)JT3N*->vN-w|+yVv_?dYj1)XdlnqLxOHjicnLa*liH6RQ>>iHnKcDf!!V$| zSpR`Nf%03jS7RZEAzVBaIEwH+6cRpEd=`y4C2zgjhmG>FZ-OEz75ff2oi9CQOhWPm z_7;h%paWqPor$1)mH)JmilqK8Wg}*1}Gu&*tEz4TA$JlOZoVSwnCg8&e)H~;vm}e zq10JAEutrNhWI&68;G2=&%MSLbZOQck4kZ?cilbdOg?h@%;6Pr@lnVv~ZB*!s;S9BjvTlLmPDr@uQ_g z{oAtP)cN7VLGz9$K1NzjhvccBvG`v`=JBYCrjXg=QnZ%}lmQJ?XyWIpA5W?QC;D;y zs}YPNv=6mDA;@Q!PO~IAdK#5-Y`~7b%Ih_>*eaa{1?8C-Ys8MRRBuf540NyZIw!@X z7tU!!2VdPwvy>bmnV;EeOC9O$v?_hMJvp51qyj*uOt-^vK*J0~L`e$t|N zN6fds?R+I;AxNfi1)li8H5g6>k=rvJHa`Ia7!Oo0#f+mkU&|DGVI*L{YaZsn(lp6G z`#;si@uh@-m3@c^x~Ny?JlZbDB;4>Vr4EgmEd3Co)*wv~ldBkZA=TxO_zCO<<`axu z&X+r^)^yWgc^h7W=ET3Y5$Zvy0-eaq0%J}Yn=5>DcL|;K%E+l8+O<>s0!!Z(fbG&J zLc;jn-vZYwQ`#sMbeewkoPL9vnl&5zjP7BItzCX;KyVko9X203_XONusV1QQVd`y! z`|mCM?{QBaQ7bhNFrGWnZ@sJylF0ISt+c-_KZde?_~+-v=q|pM_x{Y`hs}^SWl#S` z$kT|ignNjcPkhOJ5CuEVly7K8YGr~#?Od3}KaCs)fr|(t93yTC$o#I#l%7K9l)i?< zp+=0q7E51+K>PTk%F79q5hGhj9%0z?v`^D8ZW+ShxB{+nR~flXp$Yx9OVXcR4t=k$ z;Yi*)z;^L2nW&hqfjpb-4>#eT@y;C0xeW3A)YGnqBQ}lKCwrA6ETjbUTZVT4&gQSv zHxdERB@*BuVF zrcA5Y?jrq1Ycz2yz8~WD9QSx~aD;X|=S7~H6LM+G-}~H6zYo{Gjra?9%^bd^k^AwI>kRC9%V2+^D>OQuscpk#L~zcTi$NkU8vi6fJRa9)Y+hLGuymR~Y`s zWQLGu$!r}dzyP+6PP*+_#i_SKO(v_v{V`0FmV|## ziu-cWY>}h=$C;K@!(8%j6upE2L=+uEcK~joso}clH5?zsZZ1@C$kDJ3Cpt7=bxF|! z+e^mRtnoS~B+TFjZ>kAT$={0CwDpw26oFREzMk6LG6N_$0|OLfJsK#nI7X$qQJ-uidi-? z?&#>QLwohd=s&~ELs{PW-o8Qc!f|@OXexIA|E4>@OM^Q=4@w^W{qtWyRbaSsj32J4 zj&z`J(lvbBZBO_`%I18{c^?~U{f_15AN8OvaH~TU_|?K!lnB{y6b#w22D+3@EO89W zC)2N|F%O`rQ>{~s{6Ab)}P7mK5)5fZyp!Xw3@HhWR62<;0w6Vba#jtg+Q^> zw)-)25m5=BlC02yrwZ6HD<#3E*;O$V*{A0fzP=T>oQsa^%M=Yw1J_ArNH$1Rptnowd)=4yMGLrB2}zf{%}AoX@-^g zC1{Ctlh9Vm@O&-${3*qBT@4n>*GVod_~NmXW9sfm_4@ZLrtL){RcjxX9i_}7OI*QPkePt$BUWAv zSLuN)jLb2T;1$e5C4$f{_zq|0izSeosE$(i`~EHvBVFpk_F-`C!{l%Wi#9g_h~O&4 z1Dg1+^{!ThhxQ#}sY)Sq{LjBjb~_Ek417glgw*|fRbHDp#udMC>MAGxsP6q&`Y-#k zcC9%+6(Ei8$dMRWAUgE|H7%iefG(Xw_RzVyH<2hp8l2`NS|Y$$wZ{nbSHD<2Iy#W-)c8!vz9SS1Y#4en^oB(%rZ^l!hP=Dw9k!hMCh*T? z`yao!%Cg3=U9&Z5HDtqnsQ9eefkzorEoa_q(qn~4gX}r6DTLpRy4AUBm%C%t6t$;L zDHLqys?}tGo=Y3>iw)u#i%G0{g8So(qA%I0c|CDOVn z_oFuWrZQMW5C@bX1v|Tl~61}PPT;MGb*X#ShOZ}&C3%+v3NB$kCBm) zq1|Kd4YXX}v>HSG@K@p0Ia-#j5Kovbc>5N@aQlszUct67-aboHLpF45H{ht#A6${8}3AOOW!2I&?YtNrFR#H0S`P~6RouT&#PnYlS{@Dnbx6ev(4QTPO+)e0kcKEV@xS?Qre?qH#Gp_^%`L z_by--q%V(-$;35&A&agnxb6g{pe)<*Wc;biyIC^_3=jK|q>pf&xH~|Mp87F2qWp)8 zRP0>LwRTjiJgE>rUeaFc@u;&A4{0y+k;5_fz3TX2j}4heKZd*Th0^RSt2&c_txkoa z^Tv*n#|7q>NP^!`SFSJE8pD6y>QgR0etSqt(L=B2lVsCGJe`hQzR9Zd2^h9P z&9>YlnZP$Ro}|dV?(!w%+8^!cDb^oFsVh)8!pQDy(7bgEWN3qB}vs#Z(PEp_lCLK0J4W5_1w^*~l?VZ)bHhn3>N72j-_vjZKKu(N0d!qAQK zM)t7bF4{B%n=Br*t20KOuB=icn@+Ds*2j9gl{ooZiu5D19G^jiwn(E92ed6kVcs_E zP9&b;#*C@;ZOn4KbZp4j!c8ri=pVV>hq{Wr2b`@`gI*$S0 zqw|F5u;Le^j0aQI7$3?Dz*pgmz}m3Jk2rXv5sbM@?>~3jF0xP3BB$g!siv#^9cEYC!t?fi|OVOMsjCHL<*>IW+p(q@Vy?kFV+;_SsGHiEbpUyfEF#3ah*I1aN` zw$aOE3#Rm<@{k0yZNbVLI%W2e2DxvSJ*U2yyE z>>1h>Gc^2Tl~ep=-m(&N5JBuxVe+<%;UlKj5&Mae$U}uj7M+Yaev83gO&@PJ%o323 z0>N@%MFG!XN?dmjr6kkXo(xLNISrfTz**lJWYbTP>|8a-xUsFB*1gEE(yK9!_iXxd z+cm#TYP~dwr&S&qTDq-difn%NZd^N0hJ|Xxd_}T_9+pC>+|!nVeV@o+cn%NeYL)gy3iD^#G3f&)v2ZmA? z7whK+#Uz8=6M?NoIni@I$y4yX2fiM%->;g4Pa(MP37?%Oh@SaCw}+SW?>IC~%` zyuP&S3thP_x}1}ru3T(QsfWrmn2TNKp8D_U4YS5a2s72EF=mLB!{J7$Le10^-+wRT zeRu~@yRo6)nLe;^b66uE4vln?@vy7Lx9rM|s|+7Z?q+A7=#QJc=gVPpIRrWT8Q3@` z$`zsw9*c~r2$6(cedjMS?e1)E>W&|_kYhRGs7~2ncUvaoN3^Rc@EIsdD?1S!&|G2p z(z!$5?PAvZ&~1MM1?=7^?J#ESOqT|9ps*a|kSBb0=c^?y2^TiCT53b|A98F-ojL1M z+YJo};Axzdah@wukSvvbl>0i>%;_pS>t5P0Rua$+CD|4;y-mV>Q3naSOdPtQ*Iu%- z(C>>J7ln+Cw#@AyEFsh@m2s`LVEYMI!=aLGzhodcL|vPMd1S~%N8Y}dd`=O6dVL$u zrJ%bVDhZ#Ot61EtKG`%d-1X#03Hxq$e$=&Skl;c$$IveHG^nTD&q{0TNT{=1Mog}f z?B~s~oBmNokqYz_O|w5kH9@=}=GY@5Yngj!Y+91&a7h6hLGOimw9Fn`HHuQyur;!$ zd~A069>vO31C{$E^=7WlvrN1or#PBTu?c3#Ww*K!bIo|!1Mc;bBeu5DxlC*3M_e{2 zMrPSw%JT~-#<7bV)@Ry{H*2;hPI(f>?jd%_R`D-ookl8#K3>ow1;e;Pjmo~Yg43iN zmbnqkH0kH^G0PI%@AQeuVfazKOZogUMltxucGO)k9XcC4qk?FhghTF$02COxHPQZ7 za*0r~>F64p&>esV>O)kcYZ-N&pdK(=nLAZda25}x?A7`y(PxAA8&1R5`pfe(;h!CcPLbrr3f?m#h6*jmJiY>^M z0-e7Q92&(dkK}m3zC15YM=4K9U_!>-A7=xu$uY;(_IafJq1-e6uSVT=V`Dd~e#BL^ z=Wa0Xjs+YVyVx1;9xfO7;6flK>9igUimK^3LV8c*W$6sff9dd}f1}S~ZTjUvy65#r zzI(qQMEP$25~cYgVeubXe&N7b7Q^Qc`u2WO-Ohae-3|PqD(E(p^{NGmx=PiczvOA$ z!@l`Qi(c#(xw)I#g}_p-sQ;8Us|3kadq9+z&|<3_=bORWsev7HJt7u)?Dlwz)z0kP zix7)7-Aw2sNO&f%F9y1XDHdP-*fVpl2CcO*{artKq>zB>;hs&c<}LpfnVL-90iO+P zLRiI_jIB^raD;1ld#MTI!hGpa$H-)*40D$Yw?c^^1Hy6R%dwS&BvS8)4uncZ=#Q^ibffWsauYbXqg=G& zz0VOl$l0tLGW5V`UP_0r-K+)M-&87$iQx}}S{ah9)^mV?sVt);UGgl=^C^^B^n8r& zip(oJNv>jjH**D9wn7+M5#9vu#yc*7Vb4uwci>fv4Ysw;39^P=RMdU$?>FfRw;J9$ ze_bN3_1#!b6y<3Q3lUHrGOoLPzRSuG*c*VP$?mu~nlK3s-pKvKn)`?a#v%yw-CR{F|i4+C{Es^HpML4w!G?21odvDR&dhoKUx3w>w0 z_|l||!jJZOSy)4KW;~>Tu(GmG1aI(jU5V_Y%Tyqbzay6Qma}2Bt>tOcJH1jU{e7_K zxNtO6jAE8pj`c)i@3j6IURPsG8i>gG^ac8hXGg5Y^zRpwV9II&x}p%z=1|CSo<9tl zP3q40+Or}Rs4#o5jf}cV#ssbPL^roKntnom@@y7X4N(+VesY3lZF>7*#$RXiN#g=L zuz!p4!g5zTKIo{xAj!!j<*=p8cE~8Y1p3%W3t`h}^X*ZT$eJFG9mS6!;nELVdbP2X z0g~WW{xD+Ut|0r+44Oi{2R_xYHa6cr&SQ;B0xKTny>yiHp`FHSaq8)0$K>AquKdtL+OeU&pXY_-pWE8*Q5>T8dms|TSR$eeaXA12aVYS zvFAYF+Y;qA6y?i<{Wu~A=gAK4%aJ`(KIbgzm)Xt0vORHs9Xm=L5a#>XctY1UPl$A^ zzDVv4KxZJRnR0N`nZ%m?iS3}o_5yn$u3Jmj;O$bnTc*cS^SlCCLBq`=UScFxY>Sk<+S;e>G`vg6$qph5*xfp$~0EF2k?!Zj?n zxI4i1U5g9{jjQs8ZRBx4v^;L_dwPsGTQM#Ad`Sc3?u=G~6_7iMlE#MyNt2GGqD&?w zXpJ$UR~eD0uQ|EV?JuCnc9apA!?&Fh(8xS3pVr)O%`|N$ueCO9o$u}vSn)0Uh06(M zOn{C>$~Qigy)7lmH*UMQfUe%ITWnFRUQ^rw2=4&HulSrfqSfSo=W|G1;GTXEQ1bi$ zzju)kuG1T%7H6-V(~+TbdBaIDBj^~NmRl0~-aAL70Qz2;Xadvfd&{N%nb2 zdUkI?$DU9QkaM$v5OdlZL@fxjZO*%_&d^9FC^RoU%I)4%%*wNz@O~8orkc;yo48JA zF7CLnCsZfHH~XHM;m>ZFY;3F@lOd58Y1od>WTlM~m&lORyU*o%XpG<5Z1g=P=NPwn z`@_OKGtg4GICbJ|_pMCjg1j>%)aiJ69Iws7bxT2*$;7Kb#^S1OUgJkf+Un|3=kP&~ zEZwk$swWw9nC;QxN$28DRSGM?H6J3*&?7;+LM4PIqjN2bBsINW*=)t?#k`ZUvLFiS zbrfM$O>aL@d5q4DUl7lq@zN-OQv=Ocnr@tOd_M5iTmrQrhZcTOVS zQEneQd!(wBRDp>{sT+xWxses3thw?jk&`nvOf}Jzjq5R2A!U?GnIxjgSVnyeE)_U7 zqAlo-cG{cgC86Wr(H^(anW=spGn&bTYZE;`o)Z}2IeXjY(I6?g{HfqT+ul;8Zt5V^ zlcICRg>JOMhC5)7BrqT_g#i&5FUwNBX^>rP88a3}Vnf<#`dd)SBhMrFlux;4%r~hjU>lB9 z23uC~I86ixnD|mM15p6uM?~myWE?~E!~JFBjqb2>k(5l+diu;Us<&2~WxCTBY;&ge z*lP?=U2iamc`C8`e-=g+KGxmT7NTD3_KB?az};b4xH3*2R2pkeVR2T%(YJ0+k`7v0 zY1F6bX(-{ddNl;%q#4rjt$f8Do=U|@(^31RyI8%NTVSN8DYamRYf|wtU8({`S+*JD zAw?U3I7EqU9T&kc=WM)}+f^m`^ymDCieQMKbkHL8vi49U)y^J1=|y?2E@O?E{p z_bodSn=GY?L^`d6PD1z-W|_mFn4S2*$eIMSDi@+nZPlw&dm02(>Lwy-u(cyT)yer$ zo}XC2jkJGtifMFn^9_~5)cfcbGf-ZianJK|;;InF#iN~`#~dLsBh^KAc}mQ#8FDdh zg#}QIWG*5bRZ+LHz;anQ}X|Uhs z4&c!`5--9W5Dr&MgvC8!vOI-RmPY95@fXGXAgY>ITy${~42y@0K|idub3F5?X75Y2 zqjRB7l3tF=p_@9Tp6C=qIlUTa04eEAqEHc9JA7?OYYm)qqSd(Wp(3QX`tVYc*h+B~ z`$pu&+A>7WPu%ZA-FN*JCBr~Q!kB`fpyAm41moeK#>>C3v9?kpd%i(lWPz7$zu%X! zDIqx=b!7@u?N27`G4Kj_%x>q3BIM_W7h%UM&{G@f?*#CuU?miM(^e}tu~-H z+R`bKs4XjmZ(J=IK=~LOrMRx%KQ5BD4N`T>R==@yIEyz(3zS?_46anTlsO}t$KPlG zd2e={0wT`ZT5cVbyuvL`z2`}AF9u~VMT(}d=a`2*BGE=8wz2-o8P;*4nn@=TRH7HI zRQp(YO{?TnCTA)H9&|aBCd5kbtT>1oYl7Lu=Dm=>>-Ef=SK~Vvve_5bj6SPzj>=iN z%wVI6TubLPPOu$syGH{1EdLzC@uYu45-Gt>hq5QCq|5xXHlwE~LZjBr!x_x1B>f;=AJWuoE#K;|OvBX>j zJ7VM=yO4w=5ej1d+oOipNcuPaO9C@~s#+OsQD87^z<7#Rf(4PuRibCtOvK(&c*-Hu z)zplNBO^q3xG#9KqgBlS#>c&xh1`3w%6F zuM6fy=32R5VQw%(|IlkFvHmJ7!8b66N<#jaRQe z1Qz_*)>XK`D(UI}OpJuVVsrV2M+15RYgKpegaJJ@S;0bQ*qYkemEq1UpWbny35HN? zV9)&bZfsn(ISV%af<-|uqN6Y!lR>go+M*~FyvD{lv$^!Wp$!a?&=Cj5X;%qzx*;(z z>bf}MXJo*wX`=gM(2;KXgxB$13dFcJo%G3t+@0&*pd zMF$4%x7HO{$VjZdRSy`E*>xQ3XzWoy;pT>I;iS@23emkZt1f7Vi|Oeus;l$9`gXs~B7J$M0UARrR#ETeZNh22#?}wX$ zdR6MMI3u;VvCN4j;B1Es;>AQk=p_gu@~SRl*VFfg3X6_)sohPjbpdfWUTQf0W}`+z zj`QcrK{ZhtMS6h&oxv~(Y;5X+tNcRj5>fmZ|155#=xo8}l7OOEaP|LT@2#WSTKa!s z>V+0)(BcJx6C}77iUlpfy|_aI#jU5fLvSzd4#A;C1BBu(EfAn+afjZVN6vGutlzud zd+%TGdiOtj*4o3K*^^=BJ2M~QW4pc3zY7Z4#aB%};m6n>jYu#Y_|L!mjb$ZpTWEVP z$oFqqJfjqTWBHN%#>z3delW3qi`#Ma8;kprGvIEt|5~VXPvY;}cab7t*2+s-In(RS z)SpL&7_|G{c-_puSPDZE_d5B{zE@k){vvOTp{g=;bgx`S#%L|OmbZD!QQ*_Cn@4)w z@Nkoaeh;7CTMZvc=r5*XD;X>+%H+mos1(5yg=m|kL9H53@P7Wt#%_Iy@l)&6xVE+N zVX8SOj*M{{WLUt&z&HPjflHgCn{CU;)%24TRoG(IW*oV?>`h7)CusQ(_oh#6Rk735 zn~!)-o-wBHR0i8b3x~I~)v)O$I1Z$o2(fyp7ae5442?D{O}h=p-cVN=Di9}Z5VDo7 z^KpO-%3KV1K@G51EtR;K5STWtiSV4^@(~A~1S{tjr}c8K$W4`B19wG7%w$XB8^qJq z!$UnA{&dy6lwNX-D+*|}5K=!auoD_}7WHk`|9p9iX5?|Zg6f4C179PxKxL3td zlPptM<5siW8Z6vX9^qHwjgUsn%-V<>^i8~H+E+ZnYmagvNj%CC&#p$=C{=~T{n-R& zSI-rO0^*Y56^-gJZdJdrz{x?AEvirAlrG*|gvDIcXDv6SoRFua>~V_~iYOWuyEbaX zN5Avr8+=)4a(~d%iJdJDZBa0CWpc`}dnL{TFD6n{s`BAI zO`0wa!^o&VH~Bcq0vtk-%_QCY&znAs$Y(2yj~LzV`9Bq+;t^L+RW6ryT$r+~Dano} zPt7`{cE-Z0c2-F8zH<-1qX@e~gRT2?e_bxKT{trB#14GM#5}S6ha+qU1QLzhn6q1n zXiwvhaktGF|?rUh5_c0yJ&s|AkkA=z$4k}Zc}y8Znx{|7c-5alE33@T+E&0%#q^NM`tWtt^Da* z>q(^uRx7-I?k&E0MD$GBdRso{hiuTfniF!UI#B5*4YpBZotkEJsd%+{h0&x(SV|5I*6+ zgRDOIs@RBcYjEkY=>*{C-uK%%m7xQ7MoPB<(Q=;R)JcnS)Xo%OC)*rE9-kM>htK!* z2>WqDbM&xAQ2x+`)`bT=>zlW5@JO~hX@bFvv`UZM zNp*&eI}Nps^O#jxXD511i(Iu{t<6vu`rWJa<335cEDHCUyPMq_w=*hRyJ`=`D5Bs} zGM{h!(n3GcXo!0i;qo2R=ThB5e`C?)D`D(Z7JJ;wcUqRkagLnyRVXi<_$n$&*UP_^ z5%Ltrh)F9U-eEmTkHj*~G#PP5`7`z$-pI6Mj~XOBbD z)EFWyk?-BB2fw-j>z%zdSX`UG>Ly|m*!`t3W9M@T->eLu9J4Dr?F%!`7?{7JaR+&) zfbht+r-bSE0Pc@7-rwNkbxPedef7PwoT?eB9B*b0@@p5&KhcBQ;5;!M43aOhw_LMQ zBJzgjXXb4f*1+H9wfpJbnWCo=2~8+-{Y8vLC^;eu-|t+@@Hf^0X@Xz$C?T1wpyF&|Gx!zWWoK7MY(@>pK}js*h|~FmihgQ%xyE)KQVOpA$p2?poU}uB3W6ZKhCgd z>4%&{2YG9ud4S!?PatFj%2b8~Myav|A4@($x@o|5$;be4=P$Qea|uwW)rQU-;A@GhG`Ti@*d->KNGLeixkl}#jA=$08cza*K z-Qdiz92?)ESy@=|UXGD7#PLB6c1SrR(H1j*?5DY;#CfB5f*=2G7R>!VzVJnS>>Xa* zEj;kYw*D983Z+>Ev3Q1{TB{8P;1C8MiWB^|hzBPtVYYNc1-9N~K#&?7TB|QU{oT5y z2xdIq;wnu}jXo4wA>;>JY9Bp?nXwaWPD<%~ie5!u+BEvdQ8J)`GRLXaHRC#S0y%zx5_kxuezp9}n~ zxxzW^kC!9wwdLqaJn;_Y zVLqUPIF=4+g~bTddyPMM85^~(B(of{rPIOEW>sPD?5CSG@)*Z48^7_Q-9a>eC}`Wj zGt}7pARq*V;n=M=0#;Jy* z$_4p=3B{Z8zRXq@K7?yL3t8)p+`x%q-+ICMf5?daxsHmdM)rL^2^hl+Tsrlbc!&2} zX(%Rf-pkF6Y`lyKgCB^@^@ znU<7az#Q8+zOA_qhD(5OAaU6yAD7v)SxUNDXd2tHdM(Uw95@tWD~VoB*XKO6+JeH3 zA<-uTV>>IY9O`@h;p4dd)UMK)onigd)NDH1oEe?qj*ckZ5aP%z7w6^3Q^UXkL&Z=v zz1jxeA&9Fr?}dc!dS1YQB%sLk<>hdFh|cN5(>2PJZ^~v62TjwKFv21aRy;{@g(zYK ze0xFPR+2?jTK-5r%%bI^Q!B<8F!oPl!16GNo~pANjY(WUWlr$qS~!sZ6CaC$7-o5c zvX;d-c9`+-&h!i_E$0#|v3AsT?|Cg(59$r-JL@ivd-MBXM9d`f9Ga9M7@_Ri#cxs6 z%L{A<&!)jCZewU&8fdq#O=(@gzF_Ex*>~S&XO7nbpNMA9)W{M6XXrjpAY$1@Iv>O| z`zIdUC6Z}$fw;a)Sj==%TQixDWNMelQ_V4p$v`no0h(t&AXlfK3^(ju}%1+lVot``Bte#~n9Pb(95_;hVbY?w`8x;AuHAYf{g2%`;l`5>V(pyXRGmbZ-|@-cYDB@w7e;q(>B}Udpc0yNjd6Fb z0e4kDcCWOr2urSO{2gkg(n+TEpSAuvL;MxP`t+5jDDgOMYy$BT_CY9zgjLN*QVFWW{O4rb3DW&~Hm zoupkK2dI(DpG#(U(b0p>Ub^E;P+d7%^6RI^)#z#2rHi9mHU=f8-*@C-Pso%$RX^Op zkux^a9F!u|(6ngXFY+_rBAvgCyo>%yaD%Gv^*OB2njk;xaoaon<#nBWA&&pP?zm}* zW0`p%9@l|C`cQviy=PuL5NZl4iSqeXf_R|c(0SKD&T^vs8*3|a^OK0j8HGw)A|kaz z!<=#w><1BQ6bpVnjj{Wkv@RMG_>ENvbg5Wfn_iwPI?K^LFr&%cPKE6=>D2YDwW-t? zx2aZ&Ma5y=vtCJvk{f3=RB@}>5hCG2&wvzT=xX)yngZ19{nT7u_W5PcpSU;qgZijz z2dwkqzKef|9mlf;iN)|^d`IDXsww)X0m6gU&KZi-y$TJ492u^xUIVPq7xJp4^T$N< zcyQHKPZhZ+Tj9LY504N^tq^RCktM6jW+(l^4pBw~teyR$L%H#|quuF3JZ`-DG<8Lb z8)ezemzp7xnpzao%>w`|th4Nml^hJh4D$&ma5F;6oh=+DnbJciNpk zwuo4<;^zZX&4;dD?!j4th?~cVv}fPC^@9xzZ7<2vBAO!2E)_({_u6;-w||XzpS#b@ z&Ru4EpgkI$$Z_{7t@JMr3tr*90~jPx$`PB|Oqo$SURuVi$y`wfwonXoETxqb+U zV!144Z-lpaD1AYyaynmg zg33_{`VDD=4uIRo+K(0Aw{gY6yRw^tSDGx84yX1Y@8lxW^?C8iy20>gDW@)!p5X7m zG=gBF`eU&OSAf0YNA}lj6LqC&yJ6vSNr$RF!ceXwDXOGeJZ#s>^{^ zr-3^_l2P$F#c&lyO?FNs)mOy+0WV&AP@rwG<50>qERQ!Hx&$*|f%wuI_!q z)6V2&-{TLCTW3*3!%y7KvguwVdPJfS(%wE5edt$_nGq^o7kNAz(2=__zpSqP#xULS09+!SpRouN?RpnWD@Ho$PO3ZgO-~FG!_Z zg^oKmt(8H@N~A4N!wO*WvA9so6=BM;663s!bLg21MnU^1MYLZfekf#5gU-3;>Q>-z zP@Ctc#f(}E#p1X4x_5z0G==wW6z2Fv2)V3;mBkg;c*${}pdyoGv-3kZa<=1s%+&La zSu>(Z+;ImylT6yWrGsa;uX=E%laqDY*^^sxK)z)dw<<*FoKP888GP6zr+6wuZ8IRXfVlSn6Hp$ zCjqWl7yzsaJm2R7#tb*($U2M_Rx!lH$y6EGdbkmvRFX@u*$8K+~XB0e-GMyHz}Bi|%pn><`$nPO2`VSMJY-f{a8+!8}Hjnkc| zY2CB=s>6ZLM(dDbbeV&gjKQK~u}Nv%J1%J}#E-t$6DsTzXv!TJ2Lo@+AF9^xCr2rtVuA=U0Nm65Ht3yw8m2#!6 zVjY?$N*EYyz_Uo?wLJ0(tA-+mVj~+5!qCwCB5mv!m#9K-6POas!oad?r?pf|7eOf0 ztX|BJDww1E@^&Dbjtr;8o4?`EvY$u86U4AF4B<^Jke1btJ98HQLKXYx3g+H_z_rf~ zK5us(To|uCH@^I6{PEAHnCow_CKGQWWs8d{&5M4>?dHozyZ_99L8kqC4fm|+FY>{w zf1=CoL1wTCR(@9c-8?Zebfl81=^ckF!zPq>K7qXpX6QN$` zS?GM!{#r>))VZ6n3TYI_DxwWhz^$)s^r;lPT4o&eCRg7i70l#5^^1YzZL?p6-$Vx! zcb_`XmxbM_(^mxb5?$2`Z)#a5cg;6vZMjN3MD}TaZLvpoAX(ZMzZO7OwNoiBDjZ)g zB9^5#5o@6el=d-U59@5C<*M753bo=B>}Is%PH_s=u@kz-GTszA2|MswOD63wutxzc zmLK4sAmURb$EjFd7*J|&JQR00&14LaT)RL@CzibUmk9`HP->dHSc18Jq~`U(Db0)g zp6!x_bK>r<^p+@PxQyR;*Pw9=K35X&QqzeZZt5emIoTMhYa*fISHE(?bfeNC8?S1Z@B+(=ecAmzF?hF z)B8mMrKT8jg2;=R?gIqJ(nmA9%9OOk2l>y-9TAb`JMhIqV%bwq9)=R!nF}dw4#Ghw za*{2X$*nsf7EzC1b+;(n^4`S?8iD+J?TEGc#xlHcrJWPj zoCF7x8k#os6yA8pkuhTq-Nho$DZ;soQa@DnH3aCQSDI=S`WnJD^{%T8hF3(!jGErkDRRxJkF*4nt9HS9382Q1l+bL= zD6|u=y2;eM@V4Lr3oidr(x+b_5<)^GQj`V_Jga{WBTM-Sme>&)3%s`c*edoY@` z`s}>a+qi?ftd%%j{Sh+_rTR}xDnj=X3K%wdc#rN){*_(SyyQ}WX)}J063*3zN6YIp zx>g=v&}xb1C+std8cNw?Qa*eHATyl|mP19B&23_wd!ESs*QHJv(R|ER4PcF@P>Q-+I)=JKdM}tt@Rdl&@m#+W z76)tG6c8Q{oP|UtO0Y=Z9Qo$Op*SDdutI;rCK*`Pcr^H*nI50HJqh!m`E;S(B={@9 z&2{;4|C9BFca(nr^B-qXKMR=Uw~L4|J9;ZuQ<9>@9(uC?m+Rtc!*z~kwXT_h(1ZCE zC?3~}-|0I(i21KS?`R{_WSGF?Fq{`g25c&u1EH>;{rI?b@$ZkK{&)G&@`75=teq7Y zC1M5yM_5JBlO)7bEz+{qS_A0Z#bz%#yX6_cWf|m3D0}YS*kMwB=gqf2e~A|pr>vPp zZ3GRyM!Llv{g0bkriyPF4d4dXO30wRx$wXjr^uK;fP>gRt&O_Oy&tu$OEx73oo#9vT@M-C zy7IWE$y?xx%nA`^=*&r3pSJ6dHJEq*Z%h9a*fAZt*xeLLG}_De|HcxqyJWbP648~2 zNH<>RzHK#?mAGs*9^l-*hvaL_nYohF@+Z&tE0rC2p57Q6>MEOW~D)%aP+r&;3GJXgDm`sB#~JSh_H>8FD*S%%)>UHj|w ziT|$)kAkH?m$7>ZD$eX5IKB4wGQ6O?Nd;RpLc3Q7bSetLJ&rX10Acablv(5C?AMC< z?-;gN6wwvqZ>*Dk&Y%y8zp+ev{j+~#Ey`-z1Xpg54+pJ#wtgso(?7h|c5j4vz+C_6 z*9QFM5zj)LPQQWM^3EnztCFI>+*VVItWQSy4T{HDE{RXS2Q^$XUjraNL-G4yJ=)n| zRmrmsM+bfIDLsmhPsV)eVLtfZ|Vg@-(sk2{nUx`7D3@bm>*8#_OOZcJ6=LS$k z^Zdxm)N=qHqI%;}lTkf%Z8mo&CH2EviTW8cbQE)n&`O9h2|51ct!?#Ys@YL5OVFW| z2uB{T8P}1yafd`&t_8DVf;#iO>rHYXy7Zn2_i3zhYCZ6!FZi&ZH@u=vt#mBv|9oL= z+R}+HVyD5i^89SXDYIasRoTAf^+H|p`cQR$sV*cDWX=>7_Ehq$WpyZ(JYvu1tAr}M zm&k%d;_6^_?C8MCxnql2;Y}6+S{0G^&qb)NVgUyMoQ|;)84(B2Xup0{nUEp{w^7%& zl+`Drdb9W&>nO@=Wv6$GzJGJ|yYFwT&9d3}E%6t_w?v1aLthVH(qJEP8mmU{U#rK5 z?hqrBm;UlNEg~l>Nria@ieVg;YSpVA$R@a@0rJufyFzf>R+&>dxtT-Rci{1qsUXJP*a(9)=~Ey_V$qb_=5 z#71)BmKxD(dG*WCuHiQR`=(7I7M5X$BThsAVRpbRVP_4k4Y6I!n#7YUCV$h{VEKQ{ zkI`0?r7^ard9Gt|=7#_w_B2(!T#hb`H@f^sJQnWMbgDieZ^$f+F%g5vhp5rO;?Bb9 z6Zo$@R`b`;>lGw7D&rOUO#|-fIO4x9@VCjZu%y>KW!$ulpgY++Z{@lWlLj38l_w=k zsF*E)SA_{^1E+C|NIu3+=|BGZhuFR$E-;je#dwL$qD}WriPzTicbiEry^87R=w_g)yaC3-*CYXsdaj z=4T|lk#%f26GQUN45j<_+1A61+W-<#OX@Xv5g}rqrH4jKX3;W!y(VQ=l>oL}i*x(6 zeC?C8n&ywFUzDNe9C${m;yxj+rCt~PRLWl?$9fqE9|uFTiW3_#Q~)O52;Ww>drQQq zDuIxQQ?-Fcb_cYhnjKv-;RyAzQ-f-z95PTQ+C~JJwx>lM{2kWt9@6B|7)9|7g4sy! zlyp8$N$Y)IELbCYMTA}u5=GAriZf+NaA!OF3~)%_r>?_J_Z2Ji`P#Q3?_dD4)e~El z7P}UeqcuX&b8COYSZ%YkI%&TmKrm!&7;$}OR$(osH`A_ zy}I-CBC*Gt>`43w%-!($fo{f6noyw#)jZAhZj^F=&RV4y?KtdPsp$H!!bMEm;`<@T zE(n11U2MQ;9*L}l@J8R!>NG-TDOf6vX*#gS*1n#lkrEUUdL}gq2c7+*Ai&~F;*?}} z10c)1~^WQA^3_9n*9mP5)+QTXMmm%jO8(=^Yn$Ged# zeP<OU*rMO^4o^$zAuQc(&b6r|!vnDkT_uzB+% z&v^Hm_j&%Yy!%&6Vs2A+X_F*c2w@ZcuEOuM6FAqpcovC7nI;`axB<*%Pzo?~{!QB? zd~dP+!E@y*Bx%YntPl+t)Zms6fXaXID65U{}vwANIs^{(veGJdZJ{PfMJ5 zL(*Gk;6&0NCG;pN=+;%bAD)E2aE4pe;)HQ4Bowu6% zs;~&feVot|C^S{!kb1)lZzhYywcKKh>U9B+x*wJ6ozwN^ti`{~#>)v=)B~3CK+F@A z$;ik!9U`y!i3vKkXbWfgGt~RfRS?gR39D3obVc$`?n7Vm6fkZI_G4KXFJ+{omhU(Z z&IJ2>841!75@2FpINDuACPx$2Oa4~Dr5dwGL_A-a+M;deAlz0(9ggNN6ke-ctw2QP z7Gut5&4n2o6W}6((mLz~^6XmMqg}+FYf4<$ zHIvVkV2G)IQz7*BP(wGmWfHObcpUEa_{p zL*PNRyYYHN6;U`GGPR0kzZOlDjL#wL9EX>6%iZr-x^v@Q0bJ}YM@}VLeT#6qr9AET zPfN@5@&o(GVP&myKvYppHS-bMXkhh?jfAO{Z|5EbJ%?`_f{aldR!Xw;!u5$z=fS-9 zwd`X2{ptmvm^NV;GZ+-%M<+*qE^ZDE78`GUTpwfv-Q)nC7jkP|8!ANMD#1Q^jP;%a z<0aTeS%v(SGsV)Qjca5~p-lasn&mYUNPTe62#$1N(%zmnOR@&7ukDOl-|?&TASc4U zHZ=B;2C6p_|Fnf2VaV027!zS0KDH1H5*YR%32WI<^9W^dP}UV2JZgVdYD+K%DAKj7bgeIQXr^}s zc!rqDDV96?5#saoe;#V;RkIy-$XKT@6%73_o*)X52##j>9OKx9)M!(h@3Q4UP*MrCn8w_Q2$L*i_`k?*O|C~PRi+t{MwUAkAMrz%3$8&{+gK;3$R z@>dn-kb~PX&*0s6slzIHYuvwnE;-8>1hoVk!Rq8_9)vr{O8YKC#A4c1LD(=HsnCQ( zl0LcRHx-(ic+oAQI2RRCK~$o+TwM@mDW4vh*$0R}2hINuiWOQkZO4Ah(YK{9)e-M8 zauu|QrS+3}scH!9xsw*MQ0*C%E39EzD5xX#JhHWs_t4iA$)jNBffjZv@XUcIP2 zIM`^6J`_1e`D1^`pa~RvQ)Ch?K|K}Tk4UILzuMEyUY`EGT$lPAtHU7TG8F$mNebXA zCzm$wmzh7@W&p1g4g|`Dixz=pU*|we?oQ1R7oL6cPA1rG{I&W;(VhFLf062T4F=u$ z=s(5#{Z)~nru~K=o(`IsiTwKN7C1GD@@c!C5H&d)pk+aMD+DO*bNyr)zdS{}KsgSd zWRT6NK+dVcwOAEC8R#g}V{W*R`od3T?V9PV=ukrBs&k^~ROdHVTnGlU`pxR@(wTGP zmBe2u``9PweTiCsTYtji&azu!f6bUnVh6q+rXLq&#QfUhwL;1j3PQ(E!+J6P((8*- z;}5D##Q8KxHE8t^lgd?I>yA z@aViPT)Yz2X7?w2MP<_k{a()#x$4Oy%#K2N*W`;ov@Q{=XY?CZ>uJ+&jcZ&@+JGX` zLd5(kSCS?ZW?mc#g6%kfL^=fNE+6DST))Ux<@macJbgmZ<6z6i;b^Kg(&%3N2lbl_ zw%E>Q2(DiMY5qya%R_H<36^6&wnI)uCz%uXg%LlaySvkXz_uPrh7H;)I?z$+m7-Vp zoTFf>TP3KeA~juT9qB{*kW!e zubOd5Lrcf&_1q|a{KBkvPj^IpV}x{7J!v`O=dZ@BoA%q=A5V8s!FA-Ji)Ff~FVSnm z{cmEb0uxpzAyUuAG4`5LuVb7kreemln(lZG{4pSFAc(F4c4PJ9HRT`}kPJC8H3c%wWY299roN3g5jxc* z(z+HOq18Gpl`$R>Fy5+qu@M)J?t8v%9e+e?H}m%?(DlnLuwNEmhep(+ZhBajAGwrR zDr6-8tO?jiG4~-TB)N&|=KL}4?`$yA=FWY?GB=^yF<{O(KI6zwMmH}C7Wd{4__ezv zhw7btqK80$q)AM=q8D|_^Wh81Oy>C^=6`+k|1GMr=$wp9x@lwSky%MUn>lAwSl2c6 zT{Io(SHeUy)Vc;C8x4CxPK=TedG^yQyWi*-z1UatmAtLqGwnRas^A2R(!3M~$BEiiUAE_4(`SbVGQQcB#r zeGDizk?{F5(7Pq?z@ASB41?i;w2ioo`o~fP@f*S#RQS5%3!FF;i=PZ7A=Nf21$V?A zK%Q7td7H%dzgo_CkHw|UEU+Fwo{A2A*;i#tDCAJlUPquWSb899v=q%qxh4r$t#c9g zb~`%8gI@vO)#}7h>mRSnSXYG3s|Olbyt8z2wAkQ5^S{J0@#XmWRke`Qyeju@^+@^D zMH3Fx6IB5{M+8V=GfEqViFM=*sErxpN|;saJHxQ7ax+q+603 zX|sHNuaf1bP>4WM-G_AV=o~__YD3*Nqvtz&HIuMn#P>9zm%T6jU+)YlA*RJO-!%y; zy5g&AeX;Z$fE6Y)Z~|VTBK~0LGKle;pcusBBK;-U6Yd|(*wu^gjkw)nPRLSirJPlB z;pl?w8^!6yjSdTL;0plyIH;Uk>6@a!cgo>yxsikLj>i-b0%!q+4ZT<^s&DKgp5SgF z4{m+T$9V|$)lnmuF~9n`#$?v1#R1y`bocdv5_5uclR4& zre~ywYE#v$Jfyc&xk}=GV@@Rc_7(`(m*X{(lmF&c(fBQCtZ zn%$jd?=Rjjo-SD}IjQ?ezR7B)!fV_~goXYN5vJ zLovE!)ISaQa6tviIZxGbDW3G?|5|r!A2C1M=g0s42LEzIZ71sb&0ne4dRUsb|I2DT z`fN&_Fk5I?Os*(E)D2y5BXttMQEJqzrI&?9RA9*Umm*2Mo1{*EeK-DJse6S@{xX@$ zUzy&nMwk*;aw{kIbcPAjo`Mhqp9j!{mX;PZa8)Nr*nYe(`KgE@pX>rdS^VQlMj=4p zTKOQPVgSFu*LFG)kgVMt$Ki zlX=c#4`G@7*rMDBPTEZm|b*%VFzovO)C zo_0d?Hu`nD{H^`}mV6wWm%E?8U{fiue<$9mh(lS@JS-(5Ua1r+8kcrjcebC8qYIfB!`AnZB<=ppj-nq-6e|Uz>H4q2)v0x%=T@2|1|lHCA2LD>$Lt` z5|?@Sir#9v=}hin?r_Veo^1--3j&7(ThHmy7<>f{OCQDpz^>I$Tkx6LpDOS=UwaJ* z9eJz;zX?sVRKSfOWqXmW0sy0{m;T?pVho(-ssE9RKOyQvVK#Ukeb#(y;Z&~qoY%a5 zLQ6i8$*Ep{#{r?Z#f!MSHHw>dR}-7p?djS1-kDYf%=)4Jc3z@IzZ4W1fr_TT5|OEU z&^;42Bd^iSU;-0cix+po$Jbej0}v=SV`D}Z!yY5wFS=j;K1saZcRjHmu?cps!!Qh=h8&Ax;})Kqzy6CYq*|qx&r)<@9~g94BQeD4)RB3F)Y5N zfLK_+t`Jp0+$Zi?wt4@b*wKh1f3xz1kix$25vqAoKBAL4?OC~f>iPx42zqSfUs?A;E;LnAs{sdFzCNjw#h8U}l z<+N&THS%4$WJC)jB3ROTdnmdzo z3VCj~OvTz)mBW)a#ItS9kB#L3DR^ze5W}_pCwRf`P*r$Qany@h3^;YBG5LhaKR*yb zA52fBeq4amOhngm;$;1@?Oc)cEq&G$s$(W=N%7PoICe-L1P>mE&~Zw>SjYU2-P7$) zRPprFcnjUqBOJRO%TUZ$5E*U`vKeefVPG%yTb~KZKJi^E*5Bc3p_+6<1`$qVTcijcHOf} zYb;NGbOKL>kEiLY-=Lm6x16pw!3v6mP#Dx;Ln zQ(hTa^kP7av&`mkLoqyXzaqkwEDqq7n$;tmq%9c4mrh)_36-qz2~(DaAA{Of$Wj3X zkIga60}Etpq_u3(s_<62N`KiA&f?%Wo2u#fN&~hMl{0ia#o!c0uO{eJu&TQ+?Gw?| zNV1WJp9U)s+d^a9LY!h+8J0fo!>k%$gcww7^zXwmJ;gQFLp(8|66yB?n; zh4s_M!;;pbLCNtZ)2??=cL-C66MNu^o(f!Noh|8Yg>!4k>Fg=TYt1ufY#ENPtWe7@ z`)G%=&vy%EJUsV0RHml6<};juvI`j7*oyo(fG6tgMID%(wY|tN(drEuLiyq9a%>m& zjH7CE`a_!f+~rzdO2mvoj}80Vg5Qr%?(3K$4X3-D-V*dMM4Rblk;| zp?#3*Tcu9_QJJ8s1c@YNV3&H>fz={tq9=$H-r#mX(kDe!vkaH-QGQ!!NlaL_rOhbn z5upIR`R+<2otS^l;2)oL?M4Q0YQmSk{p%|qELvkQl+}GVgT4V$b%KV#ZJ568)xz^y z3v&a6l!}W1$2`aVTtp$!+~McRIq{;W$OOYceXniwjWqRdgcxYo}+*v%HRm>5{N4-$y~Z_v6*_q za3OmV82rc9F9--y=A=|w+YH}rd)Mh^K%rCNpDn2f*4P!`2etG~C!JmA@}BQM@YCa? zkmq9xl7%lZEc%(8aWQ`U5SLkY$(9&zx2^x?6lBnED7pqOuBhn!oHs}hc`s@}vG;u> z&S6CC`e~a`gVo=bWS2suDboXf1t>kmP0H$(C2*$xb#zdGx=NZ zmfREaOQ)Y)n6v4BUXKWM{n)1bxX1lH`$%j4rsMYUudwgmS^vD-eTUpEyyaF}P}2EA z5=M|OVZvxA(n`QSXDTFEC~&Pw?$M;$tdFCGJ^rnqXUJJ_wu~WG%rjQ5>T{UwRCKn|g-5*4MYrDgi*8>4$n$~}8UbSL9J&q8BdvoKQVfMc!eLmJ1m z;KsbW2P#Xg?`vic(;B4G5+-cbUX2#ON?25`Uq#6@!`(LS(;f5FRc*0_nU!VOYv`CA z=a#s@>P5EN}V#mDqZ1`T2FNQDBiWdr>eO-m7fhxI-AZdMD^wc7f-Q@dgV;^I{vX zzFfK->4XkDhXPC|xEf$+DjoQ_UIY!Nq2?xL0!+0GAZ#V`r#twVoW*VUTvo*O&#`*l zFt}D?eR<(Fx(BC^yp~1BH)e&XA@Z;eUY5SAx)q4Mxy1s~y`)8Doy?#K!$)B-@NBiq z(gGCDu{0?&(l0@;8nGW2kSER|5gcPFBMRIyEZxGE0)}wV)WSDCsEuML1d#ePG9Yh1 zdmW(Fn{_+z*y24b2pf2_ccX_1Cjv zkIZnz;Z3C<)TS_xUZy5k-Q%H+uQxVwJk=x}vs`mj8w|rHq#bc)N8|`&)RHAD-+;(U z9>Z5xFy?Zya*NuYX$%41s*BV1g`QQG2ABQ}41AvhNRX8s$Zuw$U@?NkCG${aFhO9- zB0<6ok3`;@Dbp$L9kQ6$$B_~EVAQUqEv4aT{g2a>b`q05?w|19_5u7?(9F*cEu9vBCR zX5EjRx6-^nNa>?Ou3K#bK*J7MWXCBL!|W5RTHY!uXm;t7pkK4-AZpQ zkcldi>a-@($6phY;~44Qu6h-b9zZehspM{AyKkB7W=ZOHMP|Y3Y;hLk?idph@~j~^ z4yD7{UTkP3NCFdJ2M@J}O(F+Z&7bFj19Gj7HhgY>TsZ1~A+OEOEAprz z)q=x_5Briti+iu2LiyND{h{Y%j|rHXX7e^ia-BCC6lXu|cE`429>W*#Ieea>WK@;lQJPw8-IyI+CCtR5EXH zFCN8DA0oc*Ixns5P;n}>^kUpY9^=JSUTXSPSb8{U$U0%z;7>?O(uHJubmA61O?-3A zRfwsds+Dg}4P&>;KYLSr7W?y%UFBKw;jHhqTlubXA=!5Jwnmax0`uaM0rHHFZf^tZ z2FdZ@H%O;(h5Z~)7B{u0`EafneEBM!30vAb|AGL5wm`nuJMo?vO53&u!>q+|-^M!< zkjrULy&7?4`mk#mFK#J^9I1ay`^`&BcM6Sod98U6&Gs`9JA!=ET(D_8J=)Pzg(aaj zkHMKY$I=4XlfNg2tT7ZByE(HctDF#Cv;_0EnA_+9`K49MP&T@8saLWx#&Tl9hAq2o zI^n6TdYN;N^K;msf<(EJG}vTCajB%-mf1xqdxd!A;)ePJ+zm3!Rz|f;b>^X=@h8r& z$Or6FY=iyAYYp6d<7eA5QNe3d;{(yXpFY=SJrw-fjxJ89g(hNX6ph>#@fBx$zU2AK z69%*2nkNCNcvoKFH_=t15Vosa{tN~m4mycIt7WVC2LuZvO+Rmy zPSL3S5eb9U7!r_7?yQt1%O>{I4;fNV7wlX%=M9=&s03P+(XpJwfkVBoNODShi*dnI zpQ7u>*B)OA7DZ9Y)IJ>w2yqd=4LXa-6Hhf8b)b{#VH@vFm+nbu3UCRl6&rYH%d-NZ z0b_J?uAi7>MPIq~F`fx#m@+jx$+Or=H45OaqE%djeGH@xLeCc?VsX6+w#<@NYb$>4 zu@X8Z4Rkgr(9?Gn$c9n#gsEx-m;^Od+S1YU0!VGn0eeQ-7UAEjOYP3t7}bSL$nYyW zycKLNw!s@g7>b6h+)(O^5*pb|{_}fj=Br~ApHRt+yh+0lJt~(A83N=*L_}w{_+gx# zU7q?(UjS}Ge;8SP{e(Oy>`#%Oj?e!jcl;-Wxu@5^_fBc3HU;*Xkr(O2l?sh;K$j@d zLU`5}4(qo1$TRT63$RrUb~gM|L& zXZ|t%U!~lO;bENlgdW~d8!yln4eV{UVdxl+yTp(49P7XUhOr}q#SI+r`SS(gdPK4q zF)nBDF&Wk+#%M{2{}^L&Y5)P@y*gKb`yi1w-LYn7LVBXBs5FCJ+J0PPGkWzYMyrtl zx}%iu!^70@T_hJf-F&yD0Sm4|_z}dRZE!$dm&TLK?c?rqK)Kga>g0Aif2vLUBfd&) z{{PnAdxkaDwds#+y-(LHC=eo|`=f_#|PsW^c%#tylF`hBzxS#vR^x&75^aw~Y zd0UxL6u20aT#bRA5Y;67qjyBZDm=Yft8)Tj8Fkymt|d^Ys7vL=vyv5D_irW(Cjh{~ z_>Hdnq4SNBXZiW8lnB#Udzu6%chBnk6*M}{D`Ue7oM)!02fq51LICJv(v##1vWx*8 zN6C!>J}{UZdXYMMVz*1-D|EWGtr-`HinY#ivlLIOCJUF95Wp5CY`^w|dg{TD59#_NW75{yR6YL4q|O)_6rvxqAPyUv*NYg+0pV6# z0SR1`^y{`W0@fW9(KY%Kb<3qY>XOy`&|v$!q8wlONa`xwL^(h66r?3PW3ZP zEL_8uXoZHX$0^1ub-8FqbU{J2xoFX%&4^QmC*3Mk^Ch-LJzcv@_s6v|=8#_1cj{z45`#_vw9Tpx-V z%2cQ_zsFvl=Vb!Ww)UbJNIW1@1ClLu*Fg1t;6?)QTss;B@f_=?kpgkaFVOdZTskOl zq?$#D{R%zV12xBn*ZX83(v9#i1A+isWfbo?J1lWwxc`j{`&YwOM(sMhs!ea`WP=P3g?<{ZIu1^AkyU z8$|qW^OQfSlIfwS{lC4!KS5vqZ(0BRrLUJ+zYQ*&9CZNX1~c^4MDQer2}W z*~z-qF}-@dkdG4*nM&+6*`M!MD?nX7V@SPh&vNL>VSoJ*g*}KEg0ph~4+(O@M-9zC z#d*tq&J1dY5ZZM9u%W3^qI8?!BSYAxkaOo(*;O%;10h_N5C)3US#+v|?;dju#q4*F z^oRmRIOXI_7rE?)sbs^U12_UKumDPgvbF+p6~gV_s%a^nSBAMdm+zK-*e?0yeP}B= zub+izDoXeHqv}(d)T#5dik>-lV3?iFdS+xzpNpuM2E@z5ka2eY0u2JObuU2kE5Cea zmy6iYuiYyQ_eAf2u8*(ZE1RUHDLm`FGz#n8a_8G;ms)RowX6#60W$Da%O|FXX<)$- z!(ZX?6r(;MbGKU21?{pUv`M>2;T6e+{?s|-xXp?3V{L2ce6G1x9n;7Pl<3zvG<<6 zozb|1nn0%50>n@P<#$2Ligen(U!5-OKZf~U-CFPBkQ&mF-FTR=?S37hkfqXyyHU&hzcVXwJt0_sPR8dL+Z-m+m^1 zx@65R+9L=H`YIMPZ|&1&@UO?{QS(_aAd z+XINuFY@Dz0qp8pn#aa6h8Z7W+DI#8J9vQEHlCp&z)1xt|9jIzQ4L@nPMxhoskf|O zZ6r17&W437$xg8~&iA3t&MZ_b$3i^BMJl-(BTj@OtR(NKKe0EW@s4*buN0^|+YA}a zxLJzZ$Ss+d$g7zeY?%cBIXJe&uE#jkX+RwTjf)1-4fkmG3O7J~zq;QyH*iFPmEHJs zsisu4<$18=0ctQDp{;o}Z%14k)l({)zW$nBmp^4US5cW^@Hrj^&q+;R;>nIiL*rxG% zGb*31Hq+J4gqk!~lHuo~?iQ{n6A_HGd;dDlQRwXXOZ1h%F<-9|S8YtRxmc|!$bmnK zFyZ8QgUnd3tasUEvt1q6J;@w+Dc^4RXf6^bh9C3Q4f&W!e%@HuBtDJ6PjBN_ z*vsCCLO=~Efu?aw(mQ~wkLeQAV~XG9yE;|jej)koMxjK$+%J=GNPJ#&^gA!zJhLYz zFc$@|hi#ut^>z*>LQp-0shOSYDg%1(BA7Ur`g(^H&C|WjJOv%PUS>9u6Rb5!EBu>} z>Nr%>b~-tQfaaHrsa{`nIuq@c*~3dkbjdt z@@I(M^#0gXpEf!z#sRX=x;{_+MtyLAX)tn2D?bCyH|9tQLVa8WB*4U$KwQpYgUkEf zuWOynIZ@N@;!mR+$4A#s5Ew&K4~UKN3q48Ha3c#pXhmgS@39IV5?>Sk19;#~$aG3x ztzEnqgsMx4?*HR~08uIdi^c_t*A{2iiK=NJQ;u`-X1loiR6t5-&t${Qb!i zg~$6bb`5O^krWyH{<`JEAvXEsL7}EfJcaywi7X~=oEan!xAJ~<`YRM7L8h29VJ7|C zDS2dc`ZJ^y$cIlZDdrSdSNyfez2kw9U8^pnynqqletGstp!KiR~zr^BO6Z6Y?-x7YHh^|JBbH1%zrKtol0 zoN9101C2_bc>0R&wH_+9H|w`E?HszY`F=B_^(Va@r+ra6Sh!n~nD*lv_fg42dhYn- zj@ne^6$_D~sn&x27JgxO%(g*sUrYF{n6#yxJf7wnf^y#3w3+tZ1>?AKDD5H&S-|?F zw#tw;=#jUWNeJQBe!AU$Lryi&f423|Fjpp2h8b@qOW{v zy}9&-66BmqOeQ&XF)A3c9|#=g!mroB*B=u;Kk1YLiwYB8a>I2sJE(H zc3yt!@?lr34{WzHoC<%s@K?JKqZy{vmb|zej4p@vW$zqqw8OT4fL+K2qu%pdhi-)G~O$cBSrIPKzC=o`g8x8-+H!w*2knZ+LQ+jh#X z-}8&v7@ERjC@_E=PIL;D9zeabU45tz5?*|yGyS|I3(7#@1>YOzkQqxM#{g=!{Jlr^9oG6b$AgIS9PU`H_18-hhYD~RJ^^t0&R{(1P0Rce)ETnTrPRhIh!D_YnFf|g8ss0pNYL&LF4<^#bI18DTS&QE_GwaS6qJpx^u z84+P7<$fVz=Xyl0(Fv8_-irq|kQ9-pl^>328b1iJAOJ*E673IUV2QRl+WQo_a6UcJu=shb>gT4Z)QPUu^~2?# zCl4MOe@$H_`})a3qa}uIiIG{wD0rLr$>AwYtE0;k;+&vs<59vcU*GQ^=cnDRO&$hs zHiHH^OC7TvCHae~5(5j|25bLz?fK=tb=JR9rr+HSC`L7)B zw0yt+l?^`j@Av!h_tYTALYq?8(?Cu!`kFedVTGMqPj^VUxHftaY266qJf(|y#5WI7jC8Sy^T)0Y zZCe!A<1qX*UU9!`Hz_h;rRaF}QO5+|I*p-__t?Im z2I?L^+OC6PuK>9Ub8N9Zf&kiSmg$^H^PDq_#-KlY?I&<3A?I+HEkE_kKw#@%AnI0e@dcgYOM2F^WTZ=bKKV|F`sK0&8 zphdLWF(A3B3Zv4xMya`By^%ZDb()6856oK(KTn$+au~1liYqqlhmp6Zx#7+p4UJAD zaSkO>3b?TA;kF(c%`kdgsdO>odz_0tfr773bOyzGE}m+)z%>!N*v54DEyo3kd;xLA zt9O1Jc4#!A|g>XV!YPc<#rgi9HlXdp4~PMuH&$<0o$VJ&O|l) z)h7&K?sbOu3o_AUOyISL!+iKYBVBD6rHNZEyS}gxSm-0+_ub3^FQZ`H*Cz)u1GQ3* zLUFxPCGq3+lH*Fjh{Tj;z53eixuzD2jW?1TFkyo+1Lv1xC$T`R-}$ct&X2ZuZI7!W(;1jBFk;Pd!> zzjt3EmMEw}Fuy11T!t4qQG~{|GX$YG+6#sNmY^NR)~P7o6T?x64tS7D2n*p8GdH{2 ziH!2x!v%1olz43f5S0D<7Fx|^MEC5SV&t7diz-2uYs*teab)xI0EtiCL+>-eS_8}@ zNCJD$m4CcCbV}#$pXK`Fx%9zieutX?pL+vKHI;0$=_h2jB+6H-mJ%HwR|y`w&-iHC zS7-SsGUG*e*5}BbU<+pho@yPAD9?EJ?dd9~5S=hnH5rKt_LbC2`kxAHdDLaNsVTg> ze;F2~yfw+yf6KM96v+v8(tbs&8pYy z&OSokD>De!9-ro(*1fmBsMjY~Mq&Gimb90@h08pTlW8j28{;lW#o~%KK=Gpzjc0Fym;Als?$MBZeK{_z@26HW zF>&k~DtvRj(e3HSD4{#p#k!RHHRiypiXX6A26`;(r^VY8&8AC+LHEoPg>7c(=Wf}S z$!k7)5QqW+cc3S6V;_;(v)b$m3Iyl(ESW||$+Wm$>5DkFPInswE(ZozEmoEady%-4 zAW6r%Vy*&FcwpPs&lJrREE4bxY)vbX)2Tl|^|FB5=Ee=ns!g>cl!n5vHN3+@p0V8^w?M4BE9d*f|dZ z@l2jSVBZQ(l;h*G6fWf2-b$>kW^8Z<5DD=trP41zzE7ulcjSv!x#hY|i_X$HRA>Pf zA-O^Wcto!h0K3t&fk~I9w&^PmWO344tOLUxE^;8O?UwN>-ETDD?R*hawyw*^)B;Wx zT7LY&l7)%k+E8kLswTQze0=O9b5?!R z6|Ar$%hUrmWS`G@gHx)zpO8Qmm;qsIXbPzHi5<(@;PTEf?hB$8d)fU5-|K>JAK$C) zXDV;OO)1eHOSVMtrAB8vTl|4{x_)PCcEuN6Uhv!J_lTJLl zVuj6NlwH`E38>TSrcjC%djYqqU1)Qp&=>eHJh>~!OOuH0Nb{NawRKbIf93|su9!>0 zvC;A9K1TjYFI0u;r>Ge2Zu0)X-=;;O?~IE+tYJSUuBOH2&Ds!#6j>r!K(c+f)k_{a zhkJLgUAz1DCBR5ghQt6f+{NJT$WyA9|L$S``IKn5X6Z@2?$MNai6qcYqL7k458ymL zS~L)WpGH&erFOOZKWsHyPF+uG=d<|$n}iuUxp#3^)w3yB-%kW_u+POGOmEU#af!C? z$|B2nknICKUvn21)2AmX#g%H~K1aty*Hd9?K1}xk(perzpzqFGk*08n?GSXxjBaCU z6!j_One91Czktc}t0ZYv3~be)bV6+-0&ixY0STf59ewRza|eKBjS1A$r<&JpO8sML z{yzki5sT(Flb%=M!y(`83U-`}MEKi3!Uma8EWg2U!ob0OPYdCk(#VseQ*rm2o``}Z z_}=TQvFTm>2)pg_&o)=`?U}%~H3i-D58^Msivtg`c(XkF@h6C^cP}J{&G)uVK!d7| zNqr7@?^}uKVoc{yqmJXKvpKEA9xEE2*Ck8)9(&k+R&DMSq(OV(>AP!E?Em$1E z7ib3_^{s>Y~#FI2RNy7z~i0Kr18bZ&bq3a4Iv0hU)K7?7!o z0rWBFPLBBUD&cm-wQINRAp4HfyBc3#w5I+$B^Bw@|MH>#)rsqr{waDkcNcNEaC0J4 z(xe~n?=O#Z;$#cOwfBA9)+V%i3wD6dOJd9WD=kWtRKTgFuQI6JT2zD%cfLI9wi3pw zmC!vZNQa!{SGiVl*56`A@5=DdGVa*)+gE9v)midrFB(TN@8l-K!CGzBmDEm7{Wczy zG?Cl2S&Dt$9&+q)S6dkv}_FmvYn_39!;-E4vq6wCZB zayO8dJCl8zMr9n4!`u^}jV?^J205oqOAIh{)L3prcKtxvZ5ube-RtRAM{=&SNuuo+ zw<|)@Drg8X)XP;Y(86`rtrA3V6TJ~lS$WPMK@)KtIr0vr9j|VwC8s%~663v{)%X{F z{enofIh6Ug>fptXf3-IiI=Lu;*OChqgXHY`imFR!KN!_Vq?QhZ4Z}w+%N#j+8rmoP z)jVlR5!fCNVWunPFYNKco>lva`iizaoB?sbITuyQYorsehi|hdBKOai`wq{;IWYE_ z8tl{P1EhgQu>dQCa+&hJa+%q^J2KHeXTL{e`ux0pAhJ_#jA2*5t;~_~O!k7;Xp3Zu zQ}2ndia}i2M>6&lfxG#fKP{M&M~S^6;0BnDMn~g$ z$6$Lt;@d}eWg6toM|)UGT*giU^_5!QLJNKZTxy38_IQ?;+bEr_z|e?l4x%=aB5$dm zwmoDPYH;)Y{k4(b~W`*Q0S|U@^`V0FqpBwXuLFLr@wMyMB;2JOsB=gkPctRo(q<>TPCxZ7_ifHwTe=HqU^;`6Zn>{MgHZnOQiw*Em@ zPk#Y)t$i=X$U5wD^k|yXeseOF2wiBNPTY>Z4fnJ7;8w8J!<30}^mwPkS>~?^lvJGdFaj`v%@AaiF5CEOAUu(Fq&hx@D&= z?}oHiNUA+K6kA?-?BgdtRkZif`n}hOyeA{L#l4-bE%QH5Ds{dTeDcIq446dCA=bt` z%_)nLRCpX7+T`!-pEah4aQE{XA9PB~&XNqTkuH3Sl1iTeXtesDdZ*sIMz&R2U>*=( z*ZRXmfyEKX@}9~9WGjkHfFE+H^1(<5CyYrsWHJb|ZkZ#9N)G~Xbj1+!ah zR&VBKvE;c{jlWRq#vMM+3mCOM5G>7CcSkidGAzu;o!^aH>A<6c;;4TLRH4nId{%JV z)(dQa)2)h7gq7Y_golY=8#UWq!*z^9&ZZ8i`s0G-qMBlRB2?@vAw+> zzEeHcO;R{!9rMb&D~f9oVT59osLEmA2F!f5Z2sFdIsPD4wKg+{tvreKyjXv2m*!`R z$8Y3XL-m?(G?Zcvx~{rTDLN+!Q8QFdmsvk|_rB(?-}N(yH04Ng%*DP+WY~VW2z^=> z6{NAgb^XHaM3Akp!3C(OTr>_($7)E~+%n^w|)N8jTd3$8u?Oaz@}z=f~R z)^!-o6eEuSwlwX#zTUYY>nnIHh%oUJc2|rzz8h{kMn*D(B#Vv-_B^ig8mDW1{cw(EQ#J6>3KlL@6WWKuU(@=YY;7`k@ZTI%KGnO zq%S$Y{ZIVR|N4;ANsTmtHBMGR%%$s{JEvG^#n1djjf8=L&l8ktV-^CDCB7cj!VN|G zXg9H!{jT%vf$~Gx^9z7b$@=x=bzAqskp`(L{r#vC^_fQ=sNTF3c1XVbko_$G6{nkr zrF+1%&s@oyvBJrv?cAuK+HZ@);i59HTgTuvk*Vq0Iis`q!vb!O2B$Y8I0=$#SLeR= zoean|y5Q6G`L|E(ls;K9H7T@Tak2$Dd3_xfITb(KG%;$$eJC2v`64FUM6HVh1Fry2 zb$5rGe?RYi`@BJkW>|yL^mRx3A>FK^rYqr@erph7;Ip6nj$ns>3;!kEW>RYE!YmIV zDpdWkN*%;zlzyLC%U&*$ceYshDl*cUr}WtiD@Twn2m7ulmMBu%!qhFPYNTH*Yvi@EF_)vTzx(oi&)9vgD+r9Itg29z8M1{xig2mHTN_RH29vgBUUaAuC8!Ub#Vg~9>sGaI z)ID^cg8C#{O5C~=8QGd$-z8RrRC{AUf3n^RzSm%4seDx+=hw<#Hy}c%RJ6MMD#qvp zjr1JIY6|#vjmqHLU+d*x(&XTBF_9U*8o~)t(Lywfh?0LwQFX-st$IKXVC5rxUbio* ze}As8)MGZxESuEGMCdAINbnR_Zdy>Rh~P6R+Swx!jru9$xbLvtK&7S1AIOqqcA6o(Su zRfu?}Syok}1_ts*bg8#Ow_p=CYyVK1|7%q5KV<*M1J~PKaVgT|tyCuq^OCd|ekX?y z#({vLv-^?$k4dw#YW%JWOfKpDuIl|xcrvudZd(p`)fe?C79U0j1X1L8CTbtwilzFw zJy&z*6XlYmP+~(%kzc+CIHo8u;rxF1lS3z+V>PJ6fX{<5YOQwpo zU_RdDgW=YWoBj+g@a?(O#>~de_hR_99KGpoT)3BKQucWgH%GkQz$12^8egp#Vr1~e zeUgL30v(l9p7du!nUc?1&DFAGQe3{z3mymBo5hl!EG!7&&jB$<7L|PmGbRHSSbH(P z-R|bxDvZ=fcLj)RpO^3|4+Z3(C$=)YI9o;88UTmSIY!8EZ=1Y7Hg37W=R?>@#D!0HfrQ~HtH$_xxCBN9;{ zVM!6Wk!||Ap0o^^`*N>CwfE>u#Oy<{uFX^ksl<}1h%<0tE!h~|;1~H3;rPi}t*OCi zK3XPHUATEXlwb8wk7N{=>}0~B#Vq-E3e>q@n&tG5VWlF%BTdGW$@0v6$J zA*C?>RrJ71ohB(a*i&l5pxUd|eBkTn*4cDNsd}5*pcw1>&r^(}@JJi21buNCsFYyG4FMnW-l_Lqk{DhWN5=Qe8FreaTmAZPy72-Nu5IC#+Mmt?A9(< zWtN=eoASY_4L`&6*dmv9Z!=SN4N^Y9JXh`ko)9VLuIT%~xRSZvNTLELu?t?9TB1SN zxWLAY+Z5b}xjo9#Od(woXdxKeI13cIyAEQb96k^@CW$hx{>okESWv>2?JC_qKVzY|<%8zMB_X;%Xob#57AK5DXVXM< zy^0G*@-m}(^s0=f`3J5OWgn;4SIlWL`qm75ys&~S#20W3Z{!yV0YpQt($fcq&uxNY zA53=sXb|#w1a5b398i|iTl_%Om9tX=IEfpJu`BVO3Z00e{?b+alb^?K=|GZ(=Ox@- zTuXBe4BK_7hn8zJ5x3XHe`g!CpFdw8ds%0z2OGsX)x>RS@V8#Vmckm zh|V^-d|ZOnMmeRL$s(n5;h+VeM%HiVwdl*R=u@~x%Ih@HV5<&kk5M&5u)=r zb=dx_FGv+RAqfX7G}Q^VYo4(HR-nmC8dCrX^ei&1b%zeA5%j1wb^1vWRWmWJhs*FR{O{PZGktj9O&biN>f+*511qXQvqo;Z z-BN@k8^78!NwLMh=ow=A5DZJI3$*z#ud{>_XD3|v--U0~Njf^6j~q~_;oPcqA^tVS z!c6wMKl6rOCb}%6qJk8z6mp3Avo5&$Fp&GnpgUIDhbF1IfY358TZARIJHMzIOVC-K zUfR0pE$Cqe`j}I~oMJsp=UwF0`65xH{TwPZ2J3NAUlo;-KPg>nEF&AeZC+O0QfJ3C z%_=G9jv)=~A{C3y6N3@M)5M7UfrM{wgg1IdY8vg-xo6UrmYg-KBT7(^L{Ds9!=jTi zml_xnzqe!e@>%gJH>Shl-AqQbudr@FuDa?DmG-F?o|ChFAg;W7>?7l&tytS~$Eyl~Fiw@3CfJDMFqzCio+ z70JI{qhE6^q45)4_TC*g)9hWC{N7cPu6wgGo-?svr11WU=d=|m9s9&Pm#Pf6Z`t)P zhbIl)nJXOS4&ujl5NwC+Fb(|74)j5qg7@%cx`+pDNskkc?cD{AcRkakeM)zuYI^HC zk#RrnGkwx2R;wMp@W-3E8}xW48_k7K=4Gq<#InwL?P=n67?7Ht_aLt=<6iR^zQ_~` z6_<5Rr?sk9JFCbZ5(l=cr1LPEmUa$wPa`@>HsAfFOS{%;7Cz*}IIMM0PgzyenP)bE z=4SsnmQED(cCk&`r@iFry=)QwK;o9Kz49hJ7{>pxsHgpL;8F)CWx|zB029M0V(JD- zcs>PX(gM{kyx>t=Tl~*m!?KmMT4o7vuN1l37BUWatD ze;lVWP_zGzr)yC_+Mc9-LwepRGh^7ma($lX8mU|0;EdUk$>)u+48+Z|Y`O#+9zLBq zFY>#tijA~+CA}(qPnT<{hl7Z+H9r_C&p%D9dqqjGUT>zPZc|gJoQVz zJzX%GrydseNdm?I8F?RV38X?*^Y+&6d(ag%-rLW!NsHHJhjiKtm;7NeyfWxDx=eKs zk_L6$CM0$f16skoH^t>J(jWb{zWZ<&Ioceg+%DSDMhMHWk*MKN2}Ubb=y*mjnjzjN zcA3IPJ+VUytqw6icRK@# zyy<$jywYtJ^Eq;_!(G*4-_d0{AJRa~{pB24<$LP;dsHe0^G9`_$w&Ho!~KaYctu6a4Ha$|AkZFys)P$$(FxQTzU>IX_(~g3zf%M?#1TfsKiGr?{VaJK%JY*;N zNU&_UAuMOgQiJ<=LTgH5UWoyBzLXeU%ca0j#)QIO?Ht9_^jJ)dSEn20ZBn>qgqm6?IzB)g;&wP^vpn$>mwm+7DfE;X zy*^o%nFG6TgM6{*wiVZEw;=IF=OT^D+(`^bS!RCngkj+hFhi)Sy|iLsr5ZVEEFW=@ z2}-JCw)&<^LI6nyR*IUM(RzJ6BAKi{gnD-b?T>i2G^QJHY@-5TKGDQ?=WA5WevTudvMRt$^V0HMp8cpCD zsRs9Zb+{ED@@{LM4F$hEiJ1oVP(NrXpOhVB*lyn$^)=`RL)b;0(H-456B}UHlT3Y9 zrlIMwI2S3MW9RgCwbko+mq^|-A2D5p(5RdJHp^)+)l`r-vnGy4Yf5%gcV7d3&*7t9 zWE+)_G9_`6NXJg-3~RLVq_Afj@q~#T)&qHAdgKwvs-$Dit=lb9L*+}HNv(;e9>Ufa z1!){UrUf#>Da%Hh8SisY!U8&}>mW{kMyDUG;RFcn5}sKTD7eYF3L^STjV za3MGHP4xR4TZ@K#HRNSrl{v9_JrtY}|IH3-G|d}v<|8T&EH$$Vr>^m)W4|9&upkGg zqbt!cSZrUQ;dvL)i3%-^9&gU&SZ!*d(>uCWl0OvH04&Xh*^9w!i|{c<8i=ug>$WvZ z3wkIXKNMcld!KdsUiBS~iz|^Hcdh!zT}#l50&o>NMvh8cd&1P0-E@Jcj|BJPi;7`$ z!US@#lf-yfTLm-fOq(l^r(bAcJ6IEKE^5r@>+EEzd$A zKX17u?4nwZyZJi=zoE~r70xLg;(CJ7FF+lle0CWG|2itT_D@cAh}SsTYn9peR%P_b z28T7^IG2nnRN@K*^c8;J!a50;<@0go5*mo$(CEFdGxy^43r&U&qTMaQseOj;ms|2* zX-oI|V##y$rX~Tpk41? z7nNFpoS+172fS!J#E2W0-N6zvjo3?fG~Jh3rW0Qj<1kt!I$xGi3UqSC3jMM01EA1{ zR=cCr?dqduhq$0e4$w+e_a#=b41RDTgbZw0a11ZloD>#9P8+W`<{r2^X%xGiS+0p? zd9u7*Is?`8Mx=w`(e`zJ3OSh(RPYO+7@gW)B$2D~n@zM_ zK{SgFuI=Y56GYiY>Ba=U(oK&6xdWx>au;=@8rUiZ$H>+4)Ne9fP(Gefk6z&ccUS zQ)gnJ88;L$5OI3l(y6dO)TZd+{#2>mPXE1LHMdQGsUrOb>bv})(5-|?dG1Uu7pUum zKSeU%@K`-ZX@Zy$gK!dAp7%kczB3x^0Yx0bh$?(^*Z1j!E&^I4z-z~a9^&+9(7wst zh7Zj)l6(bc_tQbXFNm=YXL`i9>#V<{z7s8L!=3-3!;M$xyY-DtGZ>LAJQgoI`lR-1 zF6pbwPf=m?(N`N&FD8;;GOmmp`Q4SFQlAO%F@r@Rs)&&$3NAV^vHAIg2*Wc*>=a@AUBp<@T<*q}inbMm~xs7W*A zxEC|TS(k@KAL@Z^L=nAJbhk(N>!fA4Zuy6;zerSe2VB{xq1Y!9%}>PG3H9GzJ~|q% zDH%AvH?t)()KYo&us}uMg)<-1(OH*QM#erd<~WT`A?avZxRSH@!q_9EfWx;9$NpHN z)d0(u>vsle`6MUkKVTah2pO_X-~c8si98ErIof3Zn(krq;d{y}sfOabvQ`&e06RO_ zQTNrB8k8Y<8P9fyE|)|^-y_@~p2(ycfBwF zLr-)~(Z~)+#Pt~wA}$w|?TzPF5+!hWQFrt9n1=VgU-H{^1u3SR1S(r&Tl{jY2-B+~So9cBGI) zLyE8$Hs?8(&v;fNr*(1|yH%Q=N5jRMs%u`7jvhmDO;5E8uva-#)&Mv9T9i>#i97bS zE^ob>THN1}#C^>?LdT0dffX5ve%zdL?(6QH(c+MaRDNVgyK*2KEXh0G)UIM*V01HO zQh(lZl~p~dzas3M86E+27n~#k&OeJRG}Q--3q``0G6v!ZFRtgE_#d(a{m7E4HQrCX&w-tAEJrKN)%^h#bKwjUl=thV z?Yoq&+u5qmU>9S;n4-HGVO3Q__iVFoS5o>#ymMHds(ZdzTm2hI(^Gru?aKv@UuX|z zG2GTx{lM1+qS>=hdE}B!;?L(aFD{W6fd!Dqo6D4$StGqF?9C=2lgY}o>tVS{Kec7; zsZ{r;E|{D)AN;{D6azFn-4YYZVkQk`5vLPObPk!jKo;i?W}tWI%7PU+@2q zX7pbYaz^jxq%S)tVv!EMgW3mGk4-hIZl&HGVB~8u3f|;B36FU+C!lHt}?!UAC zcM1F-EP(>YM$Zy0@5-Wh0%}8~_^}=%u&4{S#Zl&{P>)K)00LV9f~*qy0cCf_AlncA z&0yapTxVERf7bmY5XSoB@3u|)lrr%|$^j4%004yN1@N?vFs~>h zW2OOAms3=E_jg4<;F*wL0{|c=4>zE^G>yK2Ar0!Uzt{LnXKv~4^4I>qq33q5X8&p( z0GQ(ZZ*2ajVstBOcgtskljoJe?b-Qru!PStq3z#gmcL|+zssV3$=)6=9?v!!f5~pz zK$&OR>RD#7{U_PtpJYoHx4-OTpKU}zj-G${`iuUG7|Ys8N9%cy_PkO6+yOvRam*f>O;VS|{BEr8KK|p%0gN%ZXf`U$njfqY8AJfxM06rR`1Tp{#fewI( zkAQ@a@YD^Ud_LWXNC?mLzm6900`)mWBt!)CXSD?$;Mp+(;&b3H@X?Vl5HbIf0Vpr< z2~cU#Uh)vqNof-Cnxm7@3%GtH=932M>Uns2C8dC?Yw9MaUor3tYFoJFR)wZcy%EyV z(YJI@POFv4>zH6r7ZsQ1%iZtf`p2O z`U3TDCeIPzBcl+|@@T%4GAE=1x_+#hctOOgm6V*@!S}{3rMhN~o?lwqeSjn}g4#TP1*)0s7-kBk;N+vtzm zPxh|(QXE6YCCo}Y4O`W5MXaMlxEMmReFA(7g+nZjoHV)d3^HCQ^U;YW!m`cHKNza# zB74zQe;|ag{Qq1*!t{D-$J!Z|o_=Y>hM@Ei+FJaVs(93MdP%)dU~_DUSHwmHUtYm$ zixn#(J8@HPY_s?`{<<=xRlWi3)okGV<}vXk`Z33*ivGBRCjb#jic&|b8Rz^K^geag z!ynWp?Ay$yRRV%dowG4Z>X?B}y(b)6$BptlMP7LsruMMhspb#|DKhN*vB&gFE$rtb zCuWc9@8>4*(S=iZpi(sq1hSadCIc9)cCZsA#QZE4bTR}hyIVJ|h3YdzD|M(mn1TCT zA$iHbf-7NG`m->pI>l^^Cel#4SC{bvp)$s3-UvS>A1xkZdDcxWItn&Lx6$~KFF z6eb?0sEZPH+Ngt^hK+J|MQOI3)rJW=E*0h>X>tY zsVC>f@srD*cA#2@5@pSC1*(E?xJ}yhkWBl|ra7kuwa$vuii!7%JPgpd1kRs=d1$Dz zlwmtTM1p_oUwN%^ds(cL%H-a$ZMOtnPOfW|^;;EDM&Sf_*`FB@J;db-Tx@FGBB+e5 zBQ5(uT(GL0f?`MNb$+UgA5+YyoRhI!k6Z&mdrX41&0QL@QG8Ay{&U%o9KvL1d^h!& zVf$J8PQoEH>Itxl73>$f3XwA1kAKl?VAZRp>1=X!F=%D$8$7)_F~h3|}#jPg_d(x2MMO0*Xy z4y!0rIj=V4>=>Oy&q6gq#1xq;37>-v@oC%1ZF;1<9`AW9O~(#onxit~GMaq?h#A7* z-$lBa3MIkA^YI0&_4E3M#JJDhpPy9wfA@Zfl(}a+YQv?Rty=yjkIC>l&mp(rN72bR zq0kks|hS(m^dJ6NTmnjmG@NfvWM`Dnb^? zM8c4p9FJ2iZyF(F%;}Pr?Y%=&hfxo`X=?|yHSh`Gd9*aMVk=A2qD`9W_Puiu2TYKU zM@gL(C*PtUIQ(6&Xc?>3I#4VlIK!_7!e=Ghs4TPN#U^#$0O2f%V4(6}%|n~>!>(WI z*fVqym4K=)SCqOJzACF#{OF#hp0=nM{8j$pgN)L;IP;YK@mu$!6f5tBnrcz`+^<|R zQ}a$NB8g`}vF=Cf@K4j32&+w>+Jt{}aim-Bp)Nr@SH3MQ)JCbld&D=&IVfwJg5FEd zDzfQ50bU%tdBg>vg4#a~p&FcXMOXq~kG5EN4ckSr#9dfcYLK8o#P7Kdsr4xjvQ3WL zc7%U$$F!I+o=gPn<8TZfoq#l!1NSY5uv*~IB75sGQ??_KYh*mR9`_~nCfC%>FIv8G zJmWE!dt*dwg36(0deuENnc)p-V4{MZ%+uL@8Jm-3_Z{4@^0wh%?%J)imbBT%q;y*h zEP-0ccy-oKfbU26WnP6kKn1q_u}l#|T05dMbAZ81Z*Ao=CzSapy)kpa;_YNs%BJt- z0#DbqLL9AXy|V~?bXquf(qX0KY@5iT?sWw9D*^SFX+vFR?<8WaW_9d0Y}4A_<+i#h z)e0bxi2%Y*esCWt@P2wtRUoh^jTdvGBuN#*Xh?*;vkWtQ0QqEnw4xg0+-Ki4=e)X} zCYR_{{KF^Zw7u%_u4g-)L%pE&O^c@?2>R}&6?`zUlkoYE2S%iRxLOIhw>KIOrnm~K zX$|VzvJ|8+EB1;i@7Lv#xEMe|XAV;uXE;E}OZ30f(Yj`JD>EINR0$(Zr+ZT`@Y&mU zD1#u4Jkag{PwhStP2*lr^QdSU8Yp>RxGW?@9;cFF_ksMPfq|6*h|@VOLWz`r`+?Os zraL}ZqK`3OFt{N7iucv*X7%i=HTOmBEM(u$RHi@0+jX0pX-;#Uck87O7gjZS9MTvA z8d@4oUFD^;=f0y#0&=qYsulnkx#zpFMnU^j!WZ>Pq>u$eY2Ce0pgP#!aoYJP>y4*F zIVE+!<#IlpgEMqU&kpT8uP2#u^-cV*9m4E^TNaM_Y>%MpSYuANrqqdmNaSuOCuI4y z7VUL_cr;LZ(chunLe;CTaG&7bF~&9Ij!_ysX=?p@bh4Jk5igk0g&TNBI_p&Gaq1OY z)BS?PrRQh`2&9M(XY5qqJkgNrSgEVX^t_G?$}nz{nP{=ZZ8lzPQLcT5-&z%KOq_J- z+4lIUSRyOXA~oFP&yKD@sco#e0PHf3R$O3I5*tzFG{HPk zpE>X?l@nv8N?@Euu09|ignwTUkBo^C`_selM+vO|pzLvKLcYtzKIcs*;yY9pI~Yjf z)8XjIfZTFQubP^f=u@|oTrPYPKf!k~6Eo~N7zcY>)Dff19{`Ydp_%Z+ws?ZDQ-?j&WW+|>n3oL*l!d+G+k zd3ogQ=E`<#VuL^hnt-?a;|RLhVaI@F>b>nx5A0n12j^BRM~>o+Ty;DOO3_xr9}5!H zJrP4CHsHk#?HVoeK0)rJ{hakz>r2GxGfv5oVef$Srx@#M_4yAvkL-xZ@4;Dx`SdnwFH4!;aH^iC5n<8(rM(RqVe( zMZRe~5u6bJcH=ctZS0Qj1ox}&(3*gtpXnND+?;BKQ;)J6{2J;olfC=8mQ6IoyChZ` zv7*J*$mWo;SxZQ;{;6O2BfVrmY2JhH6X19HwDZc$jLBLw+@tw%UdkZ|JAG5Z_TE+P8;A|;KjX50q#u1#t9wsr(pL$Q{yEZXGBi)tT0_H9m*Fcau0AWew5myf0VD>syE6kbVP?5;AubxwPfiN70@wIwF#W^ z0w);rB^uT#8FyP(f6S3vL^HKJ0}}@caiSEZiFNYfen^KEg)O=Xc&$i#TjgsJD=sbZ zL$(2G5iWM`YLmlq()mRm!;!AE`Hr-#g+>yJe0BL;QplFS%QXu(r|_oX-IuH+`(M*| zQ>QfvmVX^(2Ycw1zAn@A_w3zzF?q&?-Q|htTIXvUW!ffuJMPjK7EIzBtQ_zbemcya zKev)oKEtFP6MM4+kNs&B@g9Y??jStt9LuR-;y|&1efoc#((AO1#p{&*7uQD+sUs zSPm{Pu_J`$9pMVT5E0cASV%{Vp2fz^59#9 zkDYl#u!m(g5(f?cWS*jd^ox(0zQ~zDz~K1MkrSespzo%Bdf8S&D=%Lb>WY|5%q)HW zUV`odDS)b#I2pj|~^FLCsi75r+*2Rry^vh^`!&Lwc{;R%42xu%5~ zW8EMbby!WE_H!5$EX7_6;~IU_Ke*-^uiC? zw35nvTGNhcYu$}BDv~pSTCI-Z?YqX^t1_o*ZI4qpB^zJEd}X1Fw(u;N-t-8m$u z-cRgWlHOYEUCc*MW|St68rE-;pGJmLdCwB^d|6p&vOi5#z{+qkZ}Qcw*CqINDZ8iz ze6N^VZfdZyU#mI?>oZ4f%!lm-I(+-HTPXE@sCqQTQh$HgxMtH-V_%veblE(wU_VJc zXH&4E#xrb#;Fh@yHCjCL(o;dxY0b)O+12pHPy%fp;NDA`MHmVFw8wJzLEF@if2N@F z{p|>!&sAbUzi+FXBYlFiR(^YgA}1Bg3z_x8!-*kK$bi!nKsCevDwDC!j(KRUZ_Z(^ z;^*A?Ag+4RMMC}~`zg(aQ{g#1o7$rnh+5^tyud{dw`cp+zBt?H78l4f@q&fzxX+Fp;q%7hI?aIQ!WmHS8!n-;v% zqhA%@sn>960GXXt;F2=NV6rQiENrI{hZkRoYENp_oGkYK@$|U)bxP@WGBgpEwD7)0 z_)@^Z>n3Xw-8z+(P1E(g!+YdAN599^kri5=c09$Cv6NNbwqgst3(?lN3d4_V3Qgc9``1el7wV%}6(Oe)^S0YiQ0pKKgAf z%r=j}kJJq)>rqW2oQKot*3+&e+1}_X&N%mjiuU(U&-z~jt*{Gk zLlJ4X*4dzTc4a79dFkoPIkm4BY0DpKFIngJV4Lb+>=M`nvnEq2eAO625nuEUT+HYE zHpYdBqc+7wcGWFrSKccRLulQ~M|oT^=VpQ0RS@U3KBF5w*&EQweawx1M@7=XEyX@G zS$Qvedx^o#o{v^6IT05>w^|#p>a_gh$FL~Djqx6_=;ax?z&?ZgA~$B{xBU))LqG@+ zoEaE`v9*frR((Gfc}(EL^v3gR;s$X;g`jn6e`y*xw6aXZaVi3ik>`oINex@&m%8S=(H{=3>NFaU`$qIlGCIA*qa_8rPJLZWlDv)wyPF zeZ1XtKEYl;mx|L4vamZYP3XmIVYdwTKXmVt%2%xoOgVYWyo;#_@7l>JlfPuw$>(lDcr|8zfYmmXBal+P z%daBnRbCDjp0F~Lt%Sp`RBBVC_?ySJx~*&RyO_A(WG=QMZjCR*Uyk1Ji{5GIvoMq< zGS@p@4rbw5*GS}vvu*ada=+dOrolgpU?_q*(Q+D3oSRg^m-g7LKFqS*z6{~5oqNT{ zP!(xK(@)=(<&!p!9b}9yp$|?a=9Qu@w#3cOO5@iWhc}DcKOTnb(rpO)?0$9{%n?G3 zC*vg@_Ya{JOI7hxrg#Txw6(N-wj)~6N|Nlfmm!KbSAZaUXl2C{AO_}q|1nUw^|LHcWuzNxM6JUa#c(P01Fy@mw5=B8n($)c@*b|`C#6S7ws>NgX2;BJnqSuQV zRlv8G;jOTyOj=!@fwidx;XK2OA+-mA)W^*I*iR0`t4*i&Zbl=$pdn^L&KcwgYI8kM zc7Jemd2kc`#8uP6TF~8NBU-vwpZb)G?gR!KHX^ko{Tzry1XqWVpLi}R2nH7w^N7Cx zzO-=FEh!D`=5Oo(ni_^0Ia1v%h>gaFFt3?DBJP3u$}*WMZM@mhdzV`Ht!kXL{nT z?k4t#Bvtco*8iKCl;-7j7Y_M4uPG-zht2?(r5i;G=FTnp2cj~Vx){WWI#hjVL$`V2 zTMDV`3W)plI;I1ttCaT7laVk( zVQL8o@lJA%g;steRUO6#UVGGA&})O4-bxRnUHmn)~$PKOeXL`~pOP`qIU&sNLTJGV&EpBn-T-jtOot@~0m`)Bwjo*3l1H?ckD zqOYtGYMHHeX+B_QT^=f7u$P-+-07^puqY>G4DW3zw$n4R%#w|G0(5&wx3iMp+qZNC z{U(ufIS3V7u_STKyxGIBi|MZ1Kr+oxP*`b`-9IRAfS`EJRNyw?8lC=jTnG{&-XatB z>2+m2TaJ~uc2M_7sF^w_caOVfzT+L~z!BkQOhKtTS3%`MKf68?mM_|z@rq+%XFrZq zp*o!~2xLoe6fhQ6Mu?EBE9S2);NUiwr7SL?w>tQ?>X8DOHI4B&8j0BGRVteWb2j|8 z{MhGFM7Xw@^Wt$UJ3@I^O4Bo;^NMs%bO>*t`kir#TTiCeI5I0bs5-Bf;~a%K-D~QU znylVV39U%G$>GD^6^c(ccdumavH^zz4+VB}_n$ghO6n0)O1fRul)Oywy0wN~aH&`$ zn=;d1t{_BIN&x3Z35>76urE!}YmqzE8POkda>dsi_Q#CxfRt5AR~{|-dk|(>#|O_J zRd=Xs*=Q%rHtEh?d4rB48pb7j3?wGLGUKJ=!FZ*HV2Zg}=f=m$_*!e6%@jKI%2iX* z>~yQdSLX@cP+|Ju4>Qhp&cB)qP1rhkDNMVZz*BlRgP&a+nw|h}odx8Jc=^TDb4q8y zcvO)WT9hY`Q#)UrI7*^xdOhbd_*QV52YQIu?a?V=K{_OzEzIw4h?S?=539Uo1mA){ zCShm50zjMpStLT=MUUU{n%)~0cBe28RFaeJ$o4o3`|qXhlj%HizivE7?Pu@9i|U?x^(p->Kj8`R(t}BJ z4k=j?ow=Ic-F%2^Y7e>5*|3Av*wo3?_ib&;T4Mr? zuS+#j-QJpPwR0(%B&N#EA?Gj4Q^xmGztr7gmD1V_B0YJ?FDEuFRn_UX7DxkjyG)0C zd62stpzu<6KZOdK1(09ByNwIENI4sC>R|)~%Zu~JDfCen7{auYf%(B8Q~sRowul?j z-C)V%-Y0-<1lMt*_nLc)#CFr{qmLJWy1wz=-h3Um%Z75@k}S+>SW|GpT;VE>BILww zx;T#72VxR+&o&J~as6_EggN@y`h;U^E!8uyCvW8;pIxvn&p=j6I3u{Wr=*8y+)O;U>OH7~VOiM6b=T-Q5gc z8M)7^K2D1;es)8G*8ePnaK_U>=!fYW#X^p?#EU#q@b{^Nwh~@f^|ZoMmstYE&+Q-G z2t^7WbEezC6sFfp**iliLs99;Z%~wfzi0ZD%^l2LvTGtTxstFhM%taxdrHnnTnhAu z*2XJ&%6=%Z7n@dxYYA(()oTX*-d1pb$kMq@bDHm7h-OQUp%|krQWVi-VHI9ofFG2G z!#IS=PuJVFzuhNiYn6+(IwXHSZ0|=g;Lq7QDh(WsPx91aOG@J69HMAa z)YML$-~5DkDqc|Z8IWD0V}0S=g{m%0@>#vFDuV^@h?OPlpyzgU2>C-K`Xw22B|rzN zgd-h%Wa44Il)ImA*CrQw-sS-FVm^(oX+5QX9!q9CT$*2ub=aykDM$YH85QS(E>`z~ zhc6^J*?|N=t->*czN(v}iXR%1ezX%R-d#@s_1v|Frb4zcqB#@CQ6J%BW@24vF++gj zjcyuzEgQGa^KN^Vn8@{!#F!4vnGuK(w{-%dOzL~TTYJ5AUTOg*v|*n}>Z*X8`21+xNy(qThH}}?P^J02*yR$5T(L3U>~Iqu%ePdJhDh*7 zoO`nA1H83<`7e@5SbIP|oEei)41bSme*6UYGnUS$SfZnS0iBv|*T_0;e2TMx z5j&TbRe>@D;D>SXAEf(rYgir(n}as}?_R(>%LXNg%U2tz{(Q_?_|HHC|319nZ}4Y9{fT^Ea3$2$4?_LJbVytN*&?yY0ZeHXRo)>{|MW>`iRzD zZ&Zkm_T7d()&pWqA07P>)lx?1$C#+HCr(fN*3H9%0F_vld|!bXXp4keSa_Y54M&Ki z>!NqGwUb}HH|)o*aRg@1kz&NfHbP8_AqZoHS4i5}bExT76H7anq3t|^Y7d;B6M`EY zXV8#s#MSC{@|bL;&mgyMgFl~+9_7?P32HnH1t>idCwl+`ae8;SG4x(u1Pf-{kMevW zIT$KVDRL6MW(tEd$?b?;HC&Mw<3J zO<X`Jc&X#V56<7A_maHHtKI$u!r4YNKSB@MF#v99Ge~>1_%2sFW4N*@J8J#pP%p&|YNqP;)TQ(4K<% zZKe(9W|#(J<%vsUrO)Za0juQg_kvR*+Mq#u&DRS!LV|FGs&O3B~^?$txFQr$l zEFKg0hZ+Gfk_r961s|J6ve&%2BuEB1?38HsFV9gH;M_}q zObBy|_#`mrd%Y7kul)-ehk|Yq3F*?(!2WdW#U<&B8ss=Aln2Qox=bQ#PH-al!*B+3 zt4)1OT`rttQqUHUA`ZPfv2YPh0e$644W??WZb~EH*jq0mAmWYAVien>6(YQ@Rj#Du zUHziB|6;yu5Xa{UKwr#E*1N|3rj6@5?#yN;5crDsRct}{mpw&!l>BAX73AZ#y|Frv zYJRn^?Vm=Qjk!1#&3hS)qq_*cZ?~4FNQtTkVk69JSDMn0S3@lv-QASwq;n%tzmYK% zaO6MF)fH9?J8pHYQ7}O#Iq>U!iigr-QZ+l`?y~|o%`&y53@#xd<75FNo8crOl;|b& zc(#h-B$#UR3vu(RMs-UVP8iSEvW>suN3N}eCDsew#z<>}6EWH-_j4WA<}|vLU3(y~ zaSU)>Ci2HU#TyJXW;H)C&~7cqBtL02gr2ost$l!3=rG;t^mq6NTn|^Wm9gRJ_de!T zMO{k5Tp6w1`V*M*@49`1X8MM?-0oVuYtBmPRx|av>XkuFM2q|b-t2DW;}ett-BISM z1uYS{9{J9KtAV$1x>uobJ)-QAu+3SA-&$X+3@(~!w;k;7?lAF|=J3{yq=CsfV(6$k zCC}%Wor5_?xpqs?-+GXtIW~#o4ds!-x8pDS);I!MIg;y8H)`QlIc0~pq?PDU^8T2^ z!sjl6zIleWU6D?Q;<)pc`_0M3b!peh#Tu6NoWo=4t%B33c{?6(Ku`@hQ^MHk}aGo_sY{>=y=2673W3`s`am@Jqz06g;`xe zsRdN@b{I3xj6V96%H3yh#PVxeYKEqIuf(0R+uC)ZU6yOlz9p(=u{|UI{W*2lGKkL> zoC{{;tN;WyRA-(IDzz1}iSID4)o68da?!F&hbHD$7G#Esd*H=zD;{2{l`QiW7Am@} zxzx$mmh;}+DuN(Hj54rW%Io7F-)Ea?`a$#1EY;aV&IW;+C+|_-dEx(Y#1^IzLQ7zZNIy97XFOodLaKa|ERgXnZ=uS!CmniSi%?>R3hWZ?epo=h-nO2%l zIky6LIXfxKhCVyBv^`u;&eIkS3nmiQoNX2)jI#ZLpi(LB{L!)=4xbm4? zbP}z9Nx5(T%^at=n&@XiNNc9^GBvFsmIL7)=-YjSir+$#$+lnz4t)D|ME&@qbPg_2RW3#GN*t58b@}xsNn(~L)~xj zpYFBD$cOc7rN;A=Fav4Df4OS4dcz(!L;d%4CNqQjoKl)(ORckXmK!-a!_zrc(pOy3 zFL7vuC&6JAKD6d$Y;`44-2zz|n~^NLy(R%273KY($+j^2!(38o-wwQ>d&bca!;JJ5 z=h^hA;1|W40o*^X1a5ay1m!q?8+%cvttQ_zFpvJ)uQmaKNtvDKLumZ;E?$5s zBEMd6U$D*&)Re)#*UD!^vJqmA3fnLk!wk#+6A1Op%^bH3(@?S*8Fn^h&sZ1qMi!9m4&~WkAJu7Rw4RAUYq#kh; zIw;ZashJNLvq@q7`OxkF%EC$Tfv6T_DYA{qH&^}*$;Ai|Dkvy5XKwpAVr7{GF_MR}oUrQpaRu!l3g?j^!P zEiDo~ob&4_-6tT^r|y?4lVzi>I7h2UhVz$trdHDK6@mr;Bc@E|xUv%~I%?#v4!{S! zT1WYE!IoyC^uSEN`eDC03vJTVU#%=Mz8r53`vaI9I3e$jUsnvvk=r7qp3+y0)h>AYA=C#y`0CK;!isNRjEDs-Ev*eeqjy2frq%Y8w?RtpaI( z{BKLybgPbWJyiCLwUq0rD7O*jie?Pp5dRNeVk~c|h|)P>>LUflr^ciA+38%EF>hi@ zV+|R4XibL~h1wFc-J_+YeNjTa#Oq}JtUq3&`__FQsWs`9c*y`;dL;`p1L03?2{W*= z241Hn5_S2Txvxq$e*;rwrpUG$x*t`;iKawG4S8#Oofhv^cxfqL*Ai@$bGV=bKOUeT zF6QERWR&S*wNuurw3ujjv$&ALOhB$wY10;QzPq}Gyx|X@UF!wJ#C4VlbJi%v5ciDN zvKMm=oxL|)&ATq*CW^iqIiX{h)FPuR+enKL2<4)%Que>e1YhJc_P-{l{~9ltY+60U zq)etdvqLajpz^sxW;hBArDaiIp?Cq#`qNgoM3}R_I=-qww`u+RaKSO4t~6t;zqS3X znM{U6S{pVp35GB734-Wb6hy_FY;}x@Gxpk}EK|^HxB20>AWz>nqmOT$IbD|B%*_Qul%F9iHa~A`Q5!xQU|c zWq2pj*SUk7r5 zsSu}EP=WRFQUUrGEOzYb4cm7xvUFj~vsqx;Bu48N%MNmuu*gCQ(u(Mb7w2nJUB0h@ zZ|F`Oxi+LH6CiZTXAj-EvjNNg14lw7I&}jrgwKytj#U4YxA*-?9`&6cf6FJsjy$3@|J~Cyt`~j%+G#8j%W}TX z{Z2*6j!OVa83&OJTM3H>t#F^GyM8x)^lUJEznHiLUZ#ZgeCEDC#5VQ8>68u5woIwgwwMJZAiS?$G)>(qq2=IT zPxjB&V9yVhCZ=+kyI^eo{-$Jp@JjBgb6`_Der1>8AnCNisQis>ThmlMyEb_k6os*F zFg{WV;UEX&$wx}Ec)IEE7jQwpNDcS(*s~dI}LJ|k^fcJvt$l95~ z?9A={%DebwPF~<+8~Xq_1isb_%-NIA)2lj0sF@#kA<=G1`eGxnuiE)X6T_ue#f}RO z=`(Um{OTAkDr<)ei@;N0#bGy;5Yu~idA93n+cm1pV5NDKQ0TyFuUbVV6`vqhwOI_Y zUrbqgS=D-n(foqb27lqbA5rNB$;FsBbj%B6uS zVmBIfP-}Z&qqD8;##KPEkllS2r*o|*&eg0DFRC8>=4g%OXN;Tcx_%8HlH>g=7fA0k zIx-gy<)27g%5cUBk z;g;YF>)T=x-7{h{I#QAO!KTQ_nD%`FP@I|R1(_nKTG_0p+pw-5OPoiSsm_C~kBdtd z6R;lz@z^xA4V{Bt+lUVO@NY)-8)m3o7bJ^dKYyo2V`JjjU$c^WDQtWIr>ttq@LXZ{ z;k9&a>ch&F!aP^*2E&dJpAs;rE)N}jY~4AXVd{lD62aZQ_Oh>O( zzqGWm1u$`u5(F*|hjxZS5k0&~_w~%&M4#cjgOgV}mR+wRd55UPna)KTjp2bBtGsxp zo(v4>+Td_n8I>}2t1Pjf8Q4{5zcM^z^ZTz%X~REmUO!M5O&4iIXl;z))U=6kAq|RI#kxKQ>4sG`eGo@gA8+(R6WL}p_rOk21FqB3rb<&UTsKE45Zr1UV%SL_DxI1CvHC2x7k?g#>Cp@E7eJ-P;QHElllo0M^x0ezv3p$}?>kx|W%@yOd>t=`K%)@2C&_GWSW41^qrD%s!+n(0_sv<*DdOIW8bn@;)@FZ5=xFiQcLa%B5H*<9flX?FsrhdN<{<|H8_nx73i~@HTH+nq^wk~dES?NwSER^TMam(fSNs=FMuOC?Zi7# z&|C#0)!Uy8hws=}4I?do?4!*?l8Nf>J^aO5WBl*Ay6xErrZO%!L->P*Dn49cTFGs| zdYb4td!<0(uudSE;-!)#)s6(MRByWa%Vmw;N~Vf>kv8Ft^zW|C)9(tWVTLdvM%dtF zYg@$|3SFIQpj-mP8GqfMsqZK{f}O3;t(n894%+Ce(?nyz&M~OmW{G1rlglohys$~M zQ+t49LjV1unR@J0@zrG~nPrDBXZ;`<9y~4=2Qm)MVpe36oSz$oji0hJknp1b~h~K2`!J**1ZA2$lYYj zB~J{eAcXfhcxq9HyHfO^ZqY?!kfI@>Kq|ul{`rfx(k(RQ4@w05;5dXcqRW3h;QAk? z7k{h3u9K$Xw;E$=EpOl?gO}x5_pyyl2_3b4kwu{P$8S`P)v611s)2Gew_TSM$4_;% zj4WEmO^f1Up_pqlf%AuhU|Le{7f{kwCEH@? z&jrPes`-~Yo`&0ga{+aW;DfRt(ISvnn$ui8{qQ82ogJIF0#M2@R7w+7<`=qHF_)5# zR(?O~4Cf6gO?Gj+s?YU89~@ z=+;=q(j>j@Zrp_bhlQY7rpIUo!E{CzyW>}<2)eVc=*sb*UDsRSI}Rq3$hAb+VHH2n z4@9}Lm~!;p*q7cW(JOz%AiQ;KcTA&lwV_K+CpT0;U}H$+S3qH436t^$mOSJ`y<0em z8@yxPIQ*AK`mk&?B{Hah_#^L>(dRV|GLbWr`8}(}I0L;+#+WOOmTLlWadxRhK6~w* zTK6>VYH;!kfuRblM{kjinI(@xmmo5JxtRmIkDouELRXfI135YSbPISWnP>X6!Nm~z z-w2n*Z)(cs`p#nR&&aFj2_vi%-0bzUf~s%*DiR2iwV=n9w|5#Q9zSHLPH`oe8sP`y zjywgXR>oAJZ7GQwQWl(FdhQ*j%S^pjYxqg}3CH`Gd$N00` z%ScI`d1fm)+XM;fJZD-eumOyeqSoy|MHTwKn{zjA>HTDIK22l9xrgC6*)!9WwKak zp;x6X?jmiMXlAh0#OkLCxMfE&0LoCp#e$UE4n(*(Y#MC80SjD_q3JtZo^4 z6}WY`{1dB^tiG@9sXnG6d?VO~)DKnT*%}`JY4-vvu<^g8El_?Q*$RWBc-E?fqs>4n1};T`_f__!Lfmc@bbp9fPfm6en@{ zbdEfId)qpOgU;G#P>9Td>5|wXwqT0zzGyU1oc|*p?`tHpg>hU}L)>9VWl8$%WWagt2;~ z!{Tp{Wi@p;*ND4SUn-ST3hmqHp6P=WglXxgZhS_MT?$ib+0R+4dzo{EtnKf!wQFSS z`_n?7O2n5JC1iJ1ZgBY%M$x4I(#uMJQ!<$M@B%2 zCf|zFQSd(52N|-Pc)2cY$j6j<8_yKzOe50piG1Xym3tf_S~vj$qQw04ETuzR0fX8+ zChFR|FeDIm$u36l@JEgo1%~NE*ux2I?Dvy@(l?wisM<|tC4aEXA<9qsIHxv~kNu{( z=Z^(5XW<5p*2-uS5!S*G?>(!!KSEN>z87z+Bi$su3#5Zsk?k7^vO4+l`R(L-`Id+wIF(NWp<- zsHbI&!0)k7dv)-4-C2%m$=&v~Av3iJIb=5pkbpCU7zr`gsiKE0J-vj8WjW<}Ge5#X z>^>`ys!-k-_PMIH&9_9aADI#|Gi0OAJ;;+A5qS&u@y1E1KUQcsKzg2zT}`NYSBO$5 z$TimoKQ^JF0?sqVO+kH9?fVu|B}RrajkRQPyEHb<8BrP7j8r{dK4HaA|GQ}NPv=mH z`#%M#8Ny~z0_K16B0AN7l^O_+x{??+-g6u!^?# zuNhcb%ju*9h$zk%>7eh zd&%!&xNWzEjRXA9pVZyZwgjUyy3Y<);35h9+o;;?YEg%e-oNNhMNg&k%5T@q3H#2n zCKbQqpTDgyIGLt>zxg&F8iWX~8|@qgF4EKm*4di?UlL3h zoA_~`Q>Z_eQl-0Jxp0?shOI~7HT2as5U5|1U-vy_q{f|;*Wc$I;dRa7%q&byh-4v) z5mT{lmLEMt8Q$=oS=^8SqSwehAq0m5SoVK}TDt_ldAo5JI`=O7q)lB^Zt@$edFN{l zp(hjHiLrRp=ePEA)VJ}t3F8_!H<(>=5;jUVf4a29wA*&HI;4GE*G;6UL?Xg1x`Z*e zep=_SvuYeu96RLWn}YkT3-`|r*p9O^%4zxuML!%p)#BTqH-(Q#u0V}=`5Jq!%~;3X z!7zMb$DiPEr6eMaHM;t8dWrUygSwO3L6TDWb2bm|sd0LT_bW79TLl!hdlijf96(PQ zG&_;*e0v{0@5*Q1b%N5fAM-#l|Hii%z>T;##`^KoJy2g2o3OWzkufSp@p{(QP8hX! zw+VZTc^gSh&{!ko{OcDm4o|l=i|mr`ml^lAfHNX8mX6Rpjp)dp@mXtvlz4h}iBG$g z57?0EEOsZn+$AvCqKa@mBEof%${BZ}{{8oeUp1TfnSCdXkB50vo6C8!Iz-(tnlC&X z6pSF1k%zXSCR{~)jS-yC&7{aTMsvHYZVUQTO(ls0&QU|)Llw}n>bHkpDIgu5>e_TA zOw#`Kt-`Ca_k@JMTg(X{(`P#{R*_T|99jAPD?@8UN$Qw_BF1H`o@fKej)#JnGwV|M zbeZBu*~le5FsQ8Bv}HKvYwo6na2t2d5=^M2=P;%2P{hQD&{X@+qQm-^W(-Ao96@^a zguon>a{&VBv0CoK)qJhL2*G9KZ!^r2Ul9cgTR4%&0rLraRg73^1@HC?NpMFp=5|ge zZ9osmJV7EzTfHNC67yy=2O%l79B@Vu`lJ|!X*K$7p$lH*X3tW3}*l*}>W z;ShJ}Ca1ZUIeqT+LhL^KxKl;){D&G|lJZQrV`dXhxrAdzWDpbZ3b8PWdw>5NHoSu0 zlfTZZpt8WqI_cQ>*~L}F39>;ia2Hz$)|wO#DA|Gtb1|}{W-=G_H1_p>L3PxyYpDjX zX|+}JVHZ2|XKSQD&}$+7gGsrws)x z`%Gxzp+dD{lSfwW(YK!6XH7|qBQ-pFQ^Y+hevk}+{h*(ip$)1)ox3tNOb+7+?UyEO zGn+5*uyFcQe_G0DntHFM`h!p+&$$V+Of50gxbcx@1r_XmhT*aIN-9oUAj&l&K{Civ zYKeFXMFo4CSSZOSBvwNiqE;K*E=RWd#PS*MEGK#kl-t0uTn4kdWCHLFzR)rNZsp?MrCOx^qf|2+)pWb0nr1+^C zniA;5Rza;dxI4a;_OtE#MsW3iU4-W9@>OEK<^@zQhaWk8&rblLP<7Ww_b;j|UUU&d zG!&y0>UHguX~q52en!qLXOX_OYn%{Q-#s-qpv|cWdi6nNQ)DvQs2;vvRXXfEN|-a$ z)Eo1GW63$7NZ!ghz67nKsZypdUqwj%{UPg_06c8j%y3S*cDWkifl(eyUf<}Kr976F z>=97x#M`dgj*9T!R{;6bQkcl;eC6oUFaZAtJM{m>%Qnol7L-ktgGDf11RI$5iIzU$y-1N_G8cvISa-pr-`|f##Z*aglyhcnt zG2S-(G!#%~Fso^L>p?l>x0nOyiW#@9eIX^#Mo}6T{Wu zi_=!5s~OR-|Lmt?`{B+CRGhw-Z7EBFMO)jzt#ynU;ZV0zxWwGG^CNFdwVg9;bRaW! z0hbAg;%UV!^VM~sC~I8pMp@v{rEbM+Q748(fdTYXH&{8`sj`iJU|^aM2tmI@Gp>_% z^s8Jw`)m>2jk|h6N}9vbUzq1E}M{91h=K3QP5Spysx-}<9 z&sMy07kKnsYCe@W?xKVpyG$f(|ondiSTLCSA_B|DQ0lNtP|Np^~GuvOC2K z#E$+TtFRc|w|&%&BfR`9rgKBF8=}S%CeeZ@hfPR@^XUi2Wq48?TaBU7y(y3CwaI)Jva0fO@9r#) zyuEW<=`Q05;|v4LNfP=4P_5i}t3@_(>g(*&0*aLNXRFnKZJWzSv-(1kCWOD+oW2`3 zgyI1{oz%p?2R`t>8^KGc{P$-jf_fY!u z5H0I!!^{H2J*O_=fNOLLAo|mQBPw(&CbQw@dR28&iDdgsuVJhL!vq8oT#H}#ox%{kc1&@XLWNP=T)TmoQ@1#?Z3>eA>z-c^a`gqLR;Cka$>>uL_v& ziKLr$G@el#w6XO8NHRS|>6poZvnFdP&nT-+7mILJ)NUWQW^(D(>yfVAplh~TCePR) zP@PbFJoJu{`5j^HN2mdEwJ?j$d?-t#wDC&e(tCcBOgpA9Dy#!1XS4TI&VNSZat1## zx-kGjh*#O^IN?S}OUu+>lL~%%kK4}bZZloa_s2BuJEzGI$lzta*xnSop?p_;YI0xL z>4t*g7Lj#hz7{qO_d1W&?aUx02Pl#o;%cvWIi=Ph>`g6y4So#2b2A{vF2vsVj($uL zFiRJhXA+p55#2*Cv#zy5bwIb8yRFGmGpH;1IY!c@3x}9zy7P|FA)h+cejYqQo?Vr` z|J1%72Dagq*;ytJp0?nSY|!?cg$l5M;97KLqLgD zZD-1uH_?MMGFB+@&H2rKo2j$s1Z;QoeS`|y&;g0jFCWhwlpGPwf6K}uH;N=ye2-hJ z@hQcLa{e|yMq=j}R*l)RlCxps%iDW2)h#-PKrRKP~T56;TcaVNNLQizu_3R6)J>#EN_#- zK@fuM2Wmc&6n8@bMcI0Eq@xM2cE(qq*?#!*mF{Bq*W zYN9=R)p)iXiB7&GO-NN&$4cSw2)3`%sMz)98!K?(R4;tw(S_0w&6_D1Ia$PA>eO~hJ>IJawn!X5JSl}9|wMy2Ha2L<#>GBK6K|lZ5}>h8+V}j!cnfs_Dz#Y?lYy$ zJ>^Vp0zOuSToriG69fptX#qGVJzBDTI!&()go#;bDKCLyMFo!cwY6owl)X*NB(6Ti zUu37@3o-tabn%6!7MotI2f(s$ciPdHUz;7$Fr}ba<_Tv8U?`C46KMF{$RBuk@#$~x zI8&EzH{Td2I2NW=(LC7K{UqN}Y#UT+yoltJ4sU`<0gwFE6oHmv=gz~`+kgb$K)#uBKmEeoLrq)M(y4~dsb#9Kp))KFz{mq zHq+}Vt7Q+*#~T!*N6l!urF2q$q^9bL`7$d2Md7iRJ&TCc!iP#Gz89kVV_Rs)p_W*f zEX;?M6-3ctFuc9LRDE9l)A%mEo?dsO-IQa|l&$CG>60|LhROuZFBV+=zFO}Jo5@VT z1II`@w4}W_2huCM*;UZlalmJ3V01DGYYA(h2Ey?vGQ6SCYE7!E%pp(T_@#_KOd29I zIGP=soVdl)U+}cn$+hH?flcRQ?FmFb-YmQ(KMuxD)200N4evdDVY=nOBVjK+4vm;4 zjc`4#gRCIR+?#z7rqKpctBWlAfpwZYoW^sZL}K1TqTHNZxg+mKchG^dINhZ&Uc&UA zdevNabs}#(r-7|vE9^qm3fO6%OmD*=dUT}G`}}hglCy05*%r6PA##M@a6TQBX$iZt z1cf)i&D*ctm($`wKi-tbY}kz^0uGHKS#ejhAB<1FpI%IfCyArI!-+3>l98!;b;9`} zLO^~BazFvha;cB~J)KU@ys@Z!#fXshlHXRf?X}hLTwTN=dlP>&+I@kPpah92ir$^@ zr?f5p|5%^@Z)vgrUwG`l^O5-90)eoMe_LEy@cc_gts*4TPSCVpSKflEM0&w5`T$UP`(vuAUkcAqyF z*UH>8qxzPlXfRrBDZ-HI1WqP4|B!8=LGH`Z`!mQP^kZD)3o&k=QJGi{^mOD1U2e=qBTRvtf9-eO*aEF-Xw+0Ojaxv&|kn zIVn=OVZCIa#NITK5&D$IyxE-3B)4ASiLvnm{RExzJ2hq;C`KisP~psa#-eIhD`(J* z<%6J{Bl*VSmCRD+b8yY6N6=5&2KHCGc19)|=m4G#fBA`t4H-vxgME;>LbG^7j(j2@ zbU?a^X&3a3cfT#|KFA@$$$Tt~K2_0P-oRKTxha!?t8fWS9*nP zNT7vZK7otK(p8HwPs{!{^DaAJwood*8tlh#*d<8z#j$od2cBn)+pZ#BRH$QmN=gBL z1B`l|$q5QPuzUyrrgQY9R2lmc67@%554^4|=mgR>b^jElT2|lAGk%)Xs`7G-dVzjm zX?Uy-guF;QWyF@&mI^R1T}IMet$$Z8vmEgCTOL`$cburVQG)o1wVO7@fPQih9bNQPLS!D+ZYX7Jo_zGze0rvc;$~2E%F;n{V?>5uOvo@?{={o)c{`}}$eT#XEa5GOKo4XAx!{B{ z)9uaI-4uAV%O3Hv@@S2hlYbC$iUptlq$3DCqm^5dv9E3jK1Sty&B?)+L^x8Zr<6i? zvDtP|wYg1V86BL`(&;;0)7iwR^8C(11oq~txr^kTl%$=7gcTDVBms#4XT002p8GZt zHJ^!&iOvSk4ZUIsm1ShWk`4Of0C`C1-J zi+4ED-sz+0nju? zbGNvrNBa;(>Nn!O?-v>+!Vi}Vg-ZEqJ|qvEX8KFXW-G^;AslHz|pCzr@HA*|$OXUgH z&;fSfPt0Kb?{m*XG13*>y`&atA3V+9WWDb9npdrSpCRl8!zqZkoa`$q^kOdHHz$@< zn9h6~x!Hl$o&eS2=lJ)Y!T9T z#PGUzu#ZQ&WHNKowQZ3;peZ?F1xpC59@rm94b~gQrZC@pLls!Wn4rAI1!yp%GZT>q5X>e2NRCDQeIKxN`kuPm~%V17^gRco9h z=hPS6<@ytChudv)i!+NUJ=62ed}dBz=B~9Ga%GfQ&5h^(;-^WngE-WOyukML-N_H(yC8=@xMMU;}nHp|V=%;T^BC`>nggF^KPDsObK#dMKwkwv1 z8J4bW@*7nfQT@XCd#zycepr*SWP33s>`>(7xxHO0br6$@vL=>Fy}(4KrXXxw^9wYTZ*>+( zTs-a-I#>HK;UZ-($s2#aj#WAZNec%O_#Y$a;PqmR5Z<9_^@OYq_We2I-E#H0$L2)%?3PjS**_%^Zw&vk7FV%0z>UFa!Dt$MPoYE zi*wr}x}sq<>%7Ft+SsK-jLbV5-{}x?vW1ydGCcAQRiBi4X%>~Ye^C4sj<&6)JpMLh z&pL>~5qm>2BMy<1Axii|YhG2c$5B>)#ku_kKSHRi{jtu-2N>U$Te@TB zG0Xxb@0>bhC;QpzYPxu_-E*!niREg^gvRt>rG&nNz-p!gpwYJ~XZ70zA&E`1TynPg#%}1u1mTxY$ zO2keWn|FuzRLf>Mj{SI3c$;ZE`T0mNFPlf}!533*h`3>8?2S8uC2iLB@!G|N1K@dg zT4=hDJ9Ev%aoI_|`6#RbJd;1frcb5A7y3GiC^)+Ry$kO?NF>Dt4Sh^VVTTk8=To{j zYR}guQ)a_2wLc^0<8MumN}ItlUu9nCGZhmr-X4pKr(oW!c71b5Ciw0YT;|=!i*#e$ zsAFJ093os-dmHWZBB@x)Sf1d*#l$=~1`2iW9jVqh9*-L5O6$hv?dA7(n&RHNN652M z?S?L_^{el@4cqhS8_yV>$gtbBFk?9wC6RA*>B>==(<2dllMatDJS**f-Ssi!b04gp z9J{TEq{UfZ!A)5t*H8FKj=>}=jJwWjjYK@D(3s;ODHD^$h9Lhj`bi3nPW&L+nB_yi zofDkC2Sa19AwlK>|7~D9f=A9fRC}+53yD<4ywcp;yC=CX(?}IgYbK;!uGUIitON8#llW;l=ORomaRfp} z@#H4LMKSGpP@$Uo)7tU_W!+-ZmL6@xN19rp!-}Rp<}f*t4P;9pmkMuJTbJ_F*foPe z^VMDF=-w$c#!3~6cRcVf#WcEYYO+)vUyBbX@&*CPT9#9jThSJ1-|wRCHFL@3y*FYr zGBU7j9{2da;OaVim!1=nh)clNbgx z^j8gMiUMMD4d^6?NC~fT8t+lpcn|4?Z z6Jvu5T@cM6xs5Dx8YO8-!sHtS6f20m=Rfin;cfQiexAYGb z=+EK%_2xpq1eMwUDFz`T8hozJN-c?U9dfyX&u9*%rau^`yY@F-;zI7_Tfj=NMFEfJ zsbj8D&i=Q#Whb+r*TeG?#8|r#*3Gsz^544b1wy%Bt4H7Nc z{P<5}tH!O(^F3)nGf_curhYHRemV5jQo=4OIirIQ?r!58HklNwxw?49+!o4!H?NWJ zb&%SWwjQj6k4-bw-HvoJ`NKy3^f3rOT?{(b_kulL`L0OwQ_IY|@vHKgov0Yhwt?r4v&CZQw#XY$ zCOO#$MU=EeHRBamPmvmQHR(BhLE-$x07!eC)~}0BdyZE zJ%Z6Wg0}Dqq~Zu9i?BJ?nNzp&6X^v|3MEk>*+_DyC0GK9dPcq;JshghT~4PY*!Nc$ zVNp8i&X5T`>rr`Aoy~zl6S`Pga%6R*iGPJXu!yDXro7;QU;|lkdd5pMFh3oA2n?wr z7WU*E61iui`|k3DD=GHT-H8Wp)0!Ok{R%}*PkpIOyJTBv-LvwNr#^F{F29YFXf}Ps zxkqyR^`8?Z|Dg!z=b8x3^N_y?vr6YV`G@MkWNv(+CVJ1Gu=JFu<`IgEgySAYI zIytCYJNF=I`2iv^Zm%XH@DA7qX;zi01^$w+zdiYY^dlFQ2RGIgmbblBm;9V&l&W^Y za~1(8yI{N&i_c*NU- zD;5x=r^;6VQ$%3A+cbNKfiOsrW`xl4vc)-YW=Lu*)ZJXPK} zeg$klLPWT0iqOf9L$+4oL2EGh^_M?!t-@M7^ZJxB^93A>fXS zmGw5~h3{0ylZ|>BnFR6f%gT>0h9vNK?5OzaHl2=?d2P=Z4y2I^EDgh}a&gK`lCU$|6Q z!BuvojB^FbM}An_w?k5cj>Mp6z?(lRG-zQSI06f+PpH~TMmPNo z%UROK4whknV)l(tN1zd7taz-vZh~~^)KG}u!?746RJl#P*DLnJ;>X)(IoH(75%cxo zwph6Sugx#KG|516AP6ZoWRVUVvcbEr{Lm$k;az7u4~}RkeeGT9j^XrPO8%tn6>wEF zbP6&&@Qyz4o5o`kYb|>q8jCR%{UM|eVmZc`-s1W@J)4*+IDh>k!b4z}Bjfm1-a_@eg8(zKwtXn70YU~>@dC|;bXBgfB6AD)CKoH=d#zgKs zVYPQm827Q`e&l=&INiX!<03H;T+R|p*S3xRizN+i0H6z4IF!V#4k)tJZGbE_N4>@V zMar`5O{5*4*PpNu_d)I^qy}c%pj=ldxmQ$V<30hqxb3sEBgKlJXE_S1Dg=Y0-^Ok# z2n?Ds#rnkNW`X2ZFJ)k;rR9N;L)<6sR^44OvZVdss;RxDD!#1W@Ei!BsinN z*j9hBvF5|M{&l}b{ct!?!Iegyo{s8K_PE0!%J$^d`u*z&f_Yp{rP>j3Rzka}0={|@ zGQr=$t2j*bdTmn#xPH&Ilz8+UCMm1VeiL{U04Itrxpyr3%?q^th7~$nha~J7T+1wn z`|Yon+%RWkPn!?JEp*A*66t5oHxeynX_Jle+D=_9U8vyx*W_!{r1A+x_MR4Y!kuB* z0^P0XE>L@cjR8TW91Ii!-e#;%CiQiCG~hV?v0+Q3Smk@t|?E`*u0Go4r zcf7OjQm5!OZ3+fjv5eKzoynyoPr(O8RSa15jbwkz{0nFLCcU?GJv3 z?e0O!kk4F9KZ~g8%u<=og45dA~ASsd%eG~c@w6f5SsJM%nY{zz@Ehf z7L*qC7xj`q*L85Lk*#}6vynW}00ws7D`e3qFR04I+vUviP^os$>>V0QRJv?`WxYeh ztsJl8X`psh;2{^fuH*s9u7*+zbMVgFAY!3TK70PZNyh-Ja=d{C}J|>)thx>H#h;$*A zDt2H#eK73-@KL5P(J@k+DHWA8wiUNHHP#Vrr1n!>hCQy#dJ7uu)iztFEuCB^TdZ9e zMapET^KJ6~MZmO#2%8M1G#P>HqUu;(+q$;g=ul+ku~1Qa_KtA=B7k?*KPy6AsPlQ& zk$QdmwLBS=fL>rbk%M-6c?Of&9!D&ufv8)!vF|6N2i*EQ-C+fDwhfb2O7z(R$5af{baSG$p3^2W40{t<;f zBR=D$RAc^UmW&b!5|-`e%K{Jz~fBE2#h;A;*}lk%u#Tp7m{k zrT#*Ib?n-&S?7sJ>G1%MmIL3+ZhND78?#BG1`+{}9J)J`JwK3j@yrH+i(0JTw3bd^ z{YdYynxgaU9<9;oyzlmJK*xvjnpjecjS!*?KN7)0DTbrf zj>vgMt_^)l9J*s>MA4_&8!r|ZD_S!XGt65h2d*MtSgl!5=1>2nA zRZ|S?YBFw)%MVZ)-Og-5$cJl_Wy#tiMlOctH^+Ywnz#F()$jXokwk#oXB?~_z=4pd zLIEvuvZF~&{6}@8Ednw%ulayqwFcD9HSd2nu{mwQFESi+I`nv;L@^^Q8^hnhWt{Yb zF(0-LUwSQ;$z>iE%ZjvupmsOvuKM26!c{ zq(-oP05_JPZ2bK>A<;ezIBzj*<@L2NJqh}tRqjF$_H0d$x) z3JoeBUk$18Q@gke-ahWp)fOG!ylU9uMX#Y7Xsq2b3CzJS6d1yZe%gYHvRup`kE4Hg z>*#M4e*Zh{qvHmgGs8B%=g79QoMFDb-wOlVh~MqU$o20g4nb#KB`#^zVw{gYUGCK^ z%FozH(y|dt`7e`yl{Dr{cwR#rD$|7uq#dO+oZn;2G7(Tz6>u7Q)gYE z&IV!jSAP-KA8w}nuWZ%bIxeY}{KF|j20zbbU7q@XcnJ^z)gxm5bw~QoH~$5ip#hdF zc0v%0iCI8aWo~>xa|J1(WL?iDJhFGhWTT0(v3>vo^=8pm{#uqTI6Qja3h^U<$sHO` z$JL%s%w2omW5&|V@Bu;DYFT9&H`GZ1`*K)v zz;8^nlGITaj*)2OUC979ED9`v?N}3LBA0qS+UcYa-+zg$+t?E_XF3|G$7F)^9R zVrXx;wjp`@_`i@qD0W68`5p*Ozq1$NF1IBj+W~^Vu`cGl?At^wHX z@WWoqTC&lxiOFFid~@-pBpbOP?9QU;25+}7W#2=6=yzsBvE&%OTHGJb-A>k0LJAE@fCjD=l7mg+p#)o?EpN2rR>k_Xmq) zvbSlsl_p6e1q-uXv+PZkE6nJwH4A4dz!s{faM{Nha%RmzM;eMIoX_QJR(EW*nSzeU zsZ(w_%a1{5P!?_5mm*ylFZ!RI+K0Itp;TG9Cnm4e=;xtDCkjs+$GID^2jB&P4x@N? z5u^5$`hca(5zO|KJ}ert_2=-@B=McS;%2>;Z8oIb+#4mx1_9@`bbFa+rp!X~&HNjM zk;rq($a8VH3)K+X#C$cSMA}Ae=~tGjHRC-Ck$nRyXK&gxH$nUkrN0PmF`H%D?vj$J zIQ)jYzByJeUZEesE6>Pv1?k_9tm4E?Fxj6K4IGdiEX(xJ`xB^U>ceS6F_xx88( zhd^#E^IXSq>YGdtO)~M?RfjMpHk3tSP95W@OP?p9nH4{~#z7}sCV880GpbO=dv!^p zuZ?@Pe$`5-a+m85ts(fhVp!}3_)7Bfp*7NO zzRaS35b^$*0QxT>>2T~0c%QvjSe~|i{zZ_n&bN-aKlR^_QN9nd82EZs*UI`A!Pszt zW`L1Jc`1g?yFSJhIoDlMt-(EfQertw^abF_F??L$6iFL&x zfWAfNC#T{<*hPV3O1hajMOyuJjb2rKR7*4SVMBz*-s(zu)bS8gconha7fwowrSWh2 zDcm&4>X*Z-s$mW^@Q84^cSkmwGSyQr9SO57(Pe=$1DFp`t;K2W>D}rAT44&e2{^fg zwcVZXeYxE(`Ev2B51yh%Gvm;zIYOiT|- zg^vf%C0I=R{rcr^vAd*_OL^d?#r?oMNxiVd;(Q+hHIUr=Shls1bsf)*ne#M-$YO}< zxm~l3_5Q57*9v78^)c_6hKGV@!mQAXlWh7gLPCf7=WD;#zX+?NFHfuK#{z*Dj%Ll( z?(?<%Uj&@?=c`QG4m)pyms)An(~|F@kc{8*b;om-@k1TBt>n=8sQyHd^84Pt$aBr_ z+rJ1*(Ix&zPcIIgwwD>yU3jP9s`}C|;BfruXziKO_vs%$f4L8OL_akh`#*{F-&p>y zzj6~qo}BW1GVeTlR;5(?i?Dcq@fX3H_b)<-3ON*BOlf^+4?NLiOjJCt{5xQ`#Cs|A=0iL@7J{VZ-ar#%&(v69h^&@oe+|qqunW`>X>&= zJ-Z|#Xbo7IDG!|0w@+bnVD7H1A;5e@Itysnoh754AR`JxqWQ;+U}@+{Iijw=dI@F@ z6@}e(ncFDUq;>tTj7#s=dZd(cHUq9rqe7SiRCgG_OQoKJ0$;#BbROqGc*o23byw{W zPVpkgQ{q03Kp{WFpl89T06oh&PR6n^PaJv_CR0{dnjT=O!V0lwiGflPP5cj4)AQKa@>gYJ9U`lSg7vY;m z`A!)Hy!g{6dDAhUJ;TVJUgp9Ss89_X*Ko<6(!!h9j_syD7j? zGg*t2Ve&S7F28u8a|ROo(uZEj_)1O2-T;W{wZ6XI$WMH-Z&t7{|kuTF`Sj%R`A2?fLD8$tm~iyH^j6JxeQW^uSIK5h-*#Y&qH(C_ty4zICOYWbQr~iA^!Xb2|Z(^q@hxHwR<|_f>RB+#+f>%W}>$*)Cm3dw24I1qGoNH&{lBHeQ zM+g_!QgGkH!Hz9o5KlZI%nrM%5tu+0*`J~>9NXJN^gy2%X((kp$4K$@e{gRFP1X2?| z9lR*xb1yJu1yR@z1ceMEFYy(QPM|@&GxE>p$E%LTM0TdZo{69^23`ebh7mH zcAes6sF)h{(o+@A)3*nR?)zIOKaJkN!Ut{ZwK&o2Ahrs1lzmgI>~t_c>tDR5e%N(3 zPRL=%6IK<=-DMg!JT3!leOUrNrqcF{IUGa{MEBufpj$Qu9pzD!#_P;hEko@Sr+l#% znMSb2wA06N$@3kg^=VTk6%3HvB}h&6dO}8EIBJPqR|#Lelf*Iw7WQl8r~0$CiZKxG z1J(w21hB}Vin55lkVQ+dJ_D;+voE`;?mu+eOI7we!o4;N7#PPMMjsb7AZ zh}Z>6?yAK=lXRc&QFfkv=)T&Xj^jq^#Sa{A?rCm)lpy}FH>AVl#L*3d(Tka|eR-E1 zD;Th>^wnkO%xF)c)oJ>*prfahMZ;<$@n!weDkhi--SQ(2;2d2EDKIwH4YBlvlv+PD zR|`e3Dat_j4DbxA8WhE?1wIt6QQcG&o>6aPZ*zNi4r)~i1hZeZ(X<3MkrY8@ZJoFr z&dH8H@EZ^Q*7%`j90(J>jUSk8nEjHZD-ghgWfgB_bk3f!p&*%7LP~tZdQw z>{|ulnAb%_#HZRxcwzH2M*Ke4kVCNqBa5CIuwGKEM#rrz3;-4$KMG7ntu6KFv$%29 z7Pr)uXOwPV+j`XSS$$8z;*rEGkO6S&`&(ZHJ1> zaAdi#j8XogLtHa!ei8e{V-p1ei^q#FgJ zNWuD!*rV+i8G$JPU=@#UQ3&L#YoWpP6ATyA*1>RukPhZldJ_#dTTg?_X=mkEMoEH= zbWFoo+;lrkydB$sseik(fB$7ecx&1?jZd?l9%AbUX{b<~62%PBE=K znM-pY3?{v|*95(s99vAgRGTQ|Yki6r(XUe+$$t}5GJ!mQKoap@fJLd7B5i2LRgaLH z%(VnT=N-P6O|xGUQxXtqMI7_Ff*x`BT-jNKZN2(niA_;Ao>h3F-{<|oa*APU1*v=# zKwlNzQ<93ot|;LKIkrOZFM_BO{HbKpxq~C3Z56a7+XntBNR-1tAqylKOc)Mj5=Fon z1i1m-yfsC|$80q%vCNh>%B50EzSF3DdoNgwP?|Y##3LAWKX;_?6>5dc0}6mrNvwRf zMH5!sb3Nyt-B$v?9sW4rgqdvGEN?4qr4zLcsaD2=@>GcC2WD00IyI|1NrT2d^WxN* zKoTRHX(jzYK0n2rR5P_~t)&O-8(O~+aq$J{nN1(JuBE`kbUy1s(4Pe^6uX$ySi=A$ zof{e?+#N~(gR-{_YvXO(eW5rMm$tZjup-5aLn!V9DONPlP^5T^yE}y<3GNacT8cw) z2*KTgyS4w#+G`*0^L%*TckOi?`7j?cGxy9rnanlUd7i&L@deqp?o{_N_}iX=c8KWp zmfst!kq%}8BXT@088&67$#T?SUu~W3-V{R|O$o(*vB`#*-%X1PEgIpQdY21^5zE>v z$zz@Wpy{B_%dQ@nA`*;?mtk+78`mu*S7TBFIMsCUkZ8F?v*q5U>2PIXGh$Kx2SH%) zTjiwjRmr2p?Kb?E9_;6W>MoS;y`n74Ly?<50`&mRPT8~{vyM`s2|eyEX)KxvT++=~ zVBQB9#_NdbIX;2>)o-=S)G1v1M@+VG7KyObbDUPw9EsaKhA@-Bi3!ZR$O$bWAEKeV zznZhO%0ak00%y(4>K|O-#WEw+OiVCliA(~+J2%B4<#vC{P6rn*6o15!tL6|x*Mhay zyBvcaT6!UQIsJD>VfvN<^uxm zm4RO0N3n_kcs5^EFDzfwu&AIWZ;xxG!lV8!!xqtcTGW^IOsQ?ECx0ktMDeme?=Xt= z+o4AUwXw=aT_2Ksy*lmZ-*N4+np8TssbXERnJW&Xz*ZTcR4=AJ^&}cF7S@lKd=b<> ztGpyhyOqI)qSfEsUA%&Ut6+3X{j&FNEp*-D3^G9(toQ!LfzLnuyNGHOfHPYDZ7|G1G7_wioi^bB1u;c#LCgOh%($=-0+O1;;~C z+(R)?e9EP@mR8Mt-_Ldz8sCMwp)hYR4vKPx10NXNsNhxuvj!-W!?LM6nCm|9 zXxGY*8wxyCiy7%Yikd&m7;=^?z6>H2*QSS3^0N|@J-2<^{liGd7oyK-h?GCU@30ta z)M+VHk;hKsdwh@e{7;lX;n^dzt4`ZCT;_brVFj->iM!@c|WM=lY}XIe&?e zv39oaGP+RC(;cvf?n2(9lZ-}!@lkR%#~oFo1$cp^GHGoJ7uk?vGt2w3{WJdol8p4! zOm&}Eb)h`5)OVX|ycX?z`M_jFqXMF3zOwWU(Q!Djd7mB=lVC7f#69V&YVq%rS~75PY$)GI|sAA7PXZscEdF zr>ur6_9h?Syg(@>Yuea+IPta<0-PFsbdUx#5yHSF=`)n4EQocJX(i!+mpvj!N=J&l zE7eW|CRsPd`lWE1Sz49SgfaV(pDF@`9jeOgJUqW8?@Q-KuYy+^pii3mR@q&_kLhP+w-;i zyw7&y;_s>%H2u&@N>&DqgfJfv--MfvVvhgd#U>?1M3F=WX|0Fyl zTu1!+6w3VIS8Sa}{o4a`1c?`OPNS4rWc7C>GN+j+#~ z^-weR`^7phlBHJbwfse(T-<%J=4Bk0!pdrI#HEh2y5klMGa*Z&k|tcvxv`8Z*FAy| zs7rzwF%_)e(1+1_rE`eWdvC;cTh!|+u4y?9n)?)LOQ*13M3tfEIlwDumWVPE2~Cwc z{4l6Hg~OU)!LcUoRMWcVqtPG}^RdSloQ#Cryy~;5!@hEQ{VleK{f#9s=Icb7XfU2s zt0=Cd2F|5!+T)#t&qmpxxwcuTh4oUAPysdum68;V_hOE^Q*hmL(GBVE-&r#(SRE}+ zAQsZP{r8i8jeTN^^zO`8lG>Ig;{-%dus_Z`c#4s;jMcQju&>&?{Sz zS`)5`bU9L4rkfKkTHL&Tit+xBQ~5vm9R9UJS@%RgB(iU)l1fsk>&UHRIEfuFrZ~>2 zc=5L-pgtLP#)^u+^Q9MY*9}PpFVmzS+oa9s+njQ!g|Lr>yGJ<#8N#-e?bHEzYym&C z(K#&6@Q*qVp|_v9I=x5C{T5iqzx|%*?_htY&;B-XVPcVQ|H~al;sWrN2ygPJe1BM5 zd#a6gyU^~(q96}j&ouWW*zUo^Z~j+35^}ydq)U!m3wzxc5VcvYU=U-sm$q}{*C-kE zg}$G886lWMc)nIqzEN+L-nmQ1v!%b!jk}ef=*RG7PeS_>j|ENjc^#6fJ*^8V68fXI zH?}u8<0%hunYC*T{p}v^2^-69eRhK&Uv^XERR&vU?*L%33IXy^Mz-{Vi0G zldh){W6!Q1k%DyNp9eRK17eeTl8G~R`rn~Sn)*L;DcG{1zy0Esygza5T;M3D<>6Np z@3F9H2g&g#@bK7}1Y}Ui1GO145exxu(Z^@UKi}LWj1{G85&!%aEbRChmL+P86s~9( zInUDjWbE~N?=1jjTi}WgBaR>FZzD^kZIb;3zPqk^K5Aw3l5`=1G&6}+No8`+3_)iH z&Om;OB&6T~rMd9Gb*;Rm6?6rqE=2A0v#kc5rNBMc0CpUm%hnI zH#qUY{@uBHvE|2ZQ>F$fKc%W`8*X`1y}YrF)U}0nqF!guubsAo#ai(#I0vv)yx-+(DK-_-*=@@p7%ixV>cC(l-cxmk00HkY2qA;2M zK^hcw#iDRnt~@yr6BgZP8`?oV@Hx!U-Ntpn0>qgh&=!QlO2j6qdgv4XHo$b>R9o*R zFWn+{9~+P$!D6|+z+iO;ztc?Y8*0P~?n%v;qtR8}5>|Iy=~N6{pKuhX$QkVyL@6_) zZim6?{RQdcwW|Dsrnjj@s#_7d6i|$VDmU}{XQ{NmF$!L^yYg)r&%F(|U3zKuU>4%y z0e~ZQrRMjpKlV`R>*ARLqtP#=EBXi4$NoWcX*4a$Br)#bC(lUuBRZUOapXp_W;at4 zse{ggcOx0+i*7Cp#E;1T!{dkkA_E4>Ybtlv*{mk@WiG00q#`JFl_X@P{IsE;ZwM_w z``W7yB0N2v;rU(8h9Ipcpld?~CT|e%wv58JyUgNi_$|uhgK@Qw2H^d19#3cr3$gIg zq@V&}3rgnoLpXs!$pX_zYzA0MnVU}{g5J)VW`TlgDmXgvS$ME&<9)*y^&$6m!nIAq zr8xu;T z+SLCQvlyCmBoVs52Z6+40`K|#%8`@NdZCAzJ$>`rCey;CSP_>Fro>csGpIDBmHNaZ zd-CGF&MJas)BH7%$ZgbLYUz6tK(v%}I==9}BAJ(@GYh@~)=^LlUvYE^!}`+s)FONz zQUs!LQ(~WY^-3G=Tl+AfgPvLWL_FXsF>3_yaXD?Mn|)5q7hRB}H+@TeV)Tboe47rh zO}TA(0d5MuHeBuQ{^Yt#0!M;0Ye{lLzvM$p$%`)LMJlx-7H3ct^IQX!*R0%&^s8){ z<7{f}5)l(kgC3Y)yizOL{%q*W9X$;czh_`N>o&JWI?N`_C4_$GU>PodKE(_z>*Ps! z@9-=@xiL`0$eNz^c&}Zwy(t^tWz2$UiJ@8ToqQPfW?3_F>MuvNKOj zmDacbOgU*x9a<9>yA_9N}N&-=}q-@#W72jJ-Jz0 zaBMm~z?|ZN-fv6HKA4xVV1`&l?lp5uR}Vi5_|K6F5_oE~O=Vj2hQ{;I=bbV+q8a-T zYo26SWUYDK!ACR}4C3~t_%a{V)CD+Tq~9FsLz8~65xt4fPD9Ik$LQA?+3YAcrrq&o zu%!UHatQZci&46k0-5$$d{}5rMtVsC+%><*lN2<5AyWeeEbHGe_6UhV#=e^|WnR?# z8-Uf^c?X6;nMI_7<0Ye$8&y(GY^L`=F!I$Zo=8R?RHhCq$}(QHvh_ifx60RW9>EIa)f zwMtswI~eT~dmfGG3(IXqewp0Z@vXbZ#spYn0nGKRkB_x@CX_;ws|ToGy>b)aH(+a+ z`&l@m?Q8TV)gCDTVhds!9IC9FiwF4e#bBP^1mRXQGNc57J+U|i!8K}u7)0u^OiynP zHt|04GCsBvn-2fKO^N^e2mkHAuv>vvPnyzBh4z4RR%9LTC1{~*^9;brFEeAPJpm?5 zU0STddD+7QHaj;j_&EoyEsmO-vyAnXt%Zu2dUQcP+#n6%%_uBBCXgjl z4RHR6)RUg&l`8MdDSFuyr(7?b2|q8MWXnlv!BgVvsE3wKp9mEyM@>b5J*z$oib`cb zg{KE0^C(;&Ji`$uE_}4d_(<8rXd>1@f6@=6pAJR5%+ewZ*dG22Fb&Z+3X~oWX8$4# zZNk#@0wyC!VLU2MI+C-%^*`g1SVg@3$J#=Unx;p!`HC6f7WhS>ZlO!#f&NnXRG3Q& zjOo3NXE#(4BO3QCj~u5spvM zjxW7^b`Xv^!YreLnV==HYBOEU=F`h+(0)YRL)_h?p@nCtV8o`0t{siifymN3`P2dC z#*W|x3gkwx+)c!yxz5MXhey5~;%pWp_?CKqhvI@e&{ZK-Ifh=* zSfJ(xvy&d5el=$c9?JNH-WI1zIu(AUal5h4b9-6QZWMLg*c)DiJJdLQoe|RxD1&!& zsmp(156ZK;eT>8;#cTS*O9J!xGJ%YF+T_?HL2(vjzWX6=8v{x~?X ze*To1ogfG=D*EDP*D%DRWfxK#Nb4Z@@jLBp*gt60?dB(5IV-Y<%_WtS);YErE8jNo zyMgki3$xHk809Xh7_?i_rAi)tXo7OZ4JBkukQxo+&V$Y z#sKwPg)LVf8An-jIx1KPkO<{1CGUl%bj!&Jpmt*@JuKPNoa5q~{p~7cEDWQt9qqp- zUx33sw=3#Ha(25GE=4c~J3dc`0m>CLCDz6I-kLk~bB<;l1i^cSHH@(S z`n+>n(d(D5m}oAV-(#w&TNDnw(P^QSL9GqAs#ev&B$B#i^4gs$4Q7Iwo~I{4y|J1; z{T}0NUPS6Y1F8Qf_OyYqZ>vwp439?ge(X!C_THPLQ&w(OA;aPvy|C@q)xYWS(Lo#& ziRcxYWwG`!@v4gQ=2-ZyZ=-qd0IjQ_PlVK-PLTG9WtrAT{K(^?rxfn|Ruuk?JC(EW z-8R-wD#?UYjHe%rLCu}fER5W=sZXhIE-x=VnE?`-3WuRqxNur%YGRHWS6sYbq6eC! z(e*zYT%}jzT7BH(j@9bLFD~!dQE>}%2=NQGS5BkywN&RKj1WGZ;FMr=n+_-s@H=n? z#7m%zc(&fSHWp7czFEVhgqC4gI$$OY8B_$m1gE3S-oRnMnp!HJxisUj^Z^43)Nncd z;GM?~Cl{D=JtLET<9kD|*bYtz@2uacjOdu?>J-1y^QF1X1LbFn&Bn^@g5#PKcD8Z!l2u~ttneM-EZm)uXywZY@-wy7voC=Z z^v55linGHvj_Y)nddTUNX7;E5o{V*}ZHw-SVdoFFs#x2!t^JBeYjT@bI4U2ToTQ11 z-ly8?jd3X>`b1VV5-}7mzsk>GuXW&||CX*D{k%r19cC2UUM42?ONV8FAr6Z>^MZB- z-)>`_km2hg&M(laFM+_0Xlg3}`-p;%3H{VUYYpxp?%$rsL-uP%onGs6i|LWKadSWs%;+mSC5P>V}iEk^YxN>=rI|8pYh3(A& z*`nEuZYqfu;Ckf;03)ADR2)) zrRcsv=iuS6OIz5>j>`-+q_(zY84`=j$eqX1;q-FpvpD%{Xg#XALj8Ih5dKU$O-n8P ze#Ej$Rp<*w^Zxq@lbUUK%WsLzX4OG(&Fk!#qUG}$lAR~%hHf(PV3LK}@5gUL^UXm=!DU!>>)hyT2^h{}3^0he^ZhSMEX>{{|!qFsRtF)SO zn!{yoG+TNRxpeQfVc+(9!I(Jp@ zA!Zi!_jt0{fnXOu^`z|Av_JwFKBo?bA4Z}~n=C_|vTwxXVz^tLO*6cj7gV%JM+9(D z=1PByf1^;K;Al9|rW`Rim-1>4mGRj%fP=0-gKi{--r(nVfr>F!rXTgv#|x4S(*B$P z`_}||uj!+==Y&}6SF<;P9$0J))v|W}a=#CoIVMIyqiRHzoG`+UWL2%Xzud(ur?a1P zWrTKCsgp+?M&iS`8U|rHQwTDVs%Obnhbbf7Ft?J=pX0dPM~E6Q2h(;lSZc`{q4lQP znEro)M6jYi<#0yZ{&`J4NaQ19|1I~^AGhrqL-z;$toU9h)T<$lXNJS(SZWvuo5ONf zvG5`7x@jtjwPsIXVtUE5ovGC3AL50$+ch(Za+k~go4F)|P)V=rRzWbV9f~Bi7bEf0 zD~CvtH8E)7pdh!`)JJk3pC1f@cf6+srDMgnUvzAp?cOgxq7DqL@QS!H&yiW;;p#gy zZj5iBs30=5$KrAx{)n8uZ08I)iB-NeOTNhImwf$9??k7fFp2$6{l6j(B5&uMrST1- z@BBI?K$(ks8BFyf>ih)*(=aAL{P5db!h1B4gPTIE|tky$t1UsvrY2HvuRd}?xY=ZsG`%m(T|^NrarxOo{Jecj32U@qGiTKzO~3qu>;K)V&?1lVm~!*@g| zjZ}NotGS97E>ze6GG!P$Vixc0sZ6xDfFI6C=O?Gq+79T|fMm))WvwVp#3r1-VcA4Q zNC;_U^@{RgH5~CPE36+AJ;OI_x&fwp_>k3lc?cUD$(`#=ANr8l%GWO!8oyuc%oYx$ z?VX~Lwl z)Z?p(U?tC@i3bnLr4;gwc;70W7lU>wVLLECTkCZ@g3QN6W7#t8i(#$eTgJb|4ZcC^ z*L(naGf;f7u!*7}R5@VB!n3g^laHcI=K5++bDZUjIe1vypwV0=PVCJ_(1W1Ir_0rn zt(x!)0`1Nb6O*7jLHK6G1$gwIffFKRs$2 zs*K6rC{){}+pn#E&_2B9bHQwPvx_{Pn9&en14v{%gC@+DrhW+s&<_+Q2jLh2?&yvp zNfrXYU5}B!)v>t2&W=n@my(k>b#v`b9Y_>inqU%A0O>b2Co{r02s@;cqKe!mX>`i=}q@VmEPiI|x35%ycP zI6Lw#A_ccPv_3}A;!i~^$g!-K)@xq}G)*G!Z>h$-$a^TNgR?CUj}Yp2MGHea5B;yRl4f1auBqOM)x z*D{f2kVUA^84Lt%LD{ux3`Yz-KbqRvOqlti`Q_*$KWFT~pqO{W|HB9NPgBQu@Uz(3 z!s%%FIq{oAVPsX)BKueC^RA?pI)dM;#L_?Vo;i%{K_p)=^_giSfQj;qFy`>|Q(qSI%A)IXk_hoKb; z({*&sat^JjpzLpv>a8m8nuYuHPlyglY)TM3F%UOb(`)(v_@8@!dSza$qC5kNl0G2=j&aWqdRntW^Ji!9^d0 z(^a6&p+*-!#7M-kZK~DnAx>h|`fS#tRl`Rp{Sx;#mh>yqJl-ySv7!=mRXn6c0^M8* z%}=4h0OqCza7~kaJNGn^#7b7vpxyh@uHahYLW4fK1oDQWvmyIp6Utnwp>&QTub3Q? z7tR4)L=pq3W;=^f96&zSH1ZJ?W$yPl>@zy!@S~~GX&rV+r}p$o{Nq$&`v6xVi?><< ziQ2>+ICR%5sMnMSB!ee<0N2G5(+>l7t!hqA&(ve9fIvfG&G!7@6c}-2X_w?w``qhf z69xXBtho7nL2#2GYt2~n!Yg&>Pj4Z5<&(6{Na=8R)zQ6Ugst&bMQUZ(Joz#C9BrV7 zZkH>G7KP(mO!${4oon&Xu^KB|2JOyIhwq1au4mRG7xQsjZEL^|MAY*VOOez>8DA07 z1VqRCT_bhOif^2iDj%3P)j@xtZ&3Brx;S7Wo`eBJs}a()tK0+AlZD4%zrC(G#l_4c zUu>}0RDPQ3t?WboV@OSQavng3ZKy3%M zk%Jv^r%3wF6~NraR&$Y15+i>W*i`nLze-P?z)Iu1`?v6Y8+RAu3}Xb2l}fb5!WJ80 zf&ULgFYomo{^Z-eS=7s}!W-)j1&yD;%*C=C+N%1|h~$&@qFQJwaqV0`gUWIoR|x_F zc4c7}o26(P@B`y=GVnj?WhDrYrpu|M+;<^yzO6r+n_q!NOgW@@v8lSSx;Oa6W&6jts2b+@ z>DsGr?XeAD|8Sw$g9day+#7a{thZ7vDev2vWjgrmxh!UW8>Z#+A*|x2z2l6_!>ISu zkC52Q5$}l2LX@P(Aq5E>H&h*6+@d5{-k(f`JfV4v1@^9$t4F%}Nz6UpG1(;Pe$$%M zqI0v^Txk4s{}9i92We19T_{nEqc_UoNO9pJYDp#bv*oupx%cl%aW{oiT>U z34SX3{7b-Q#OsO@&T#P=pnm9IGc9}PC3dHz!XMx{kl)`h_GDySG|NIoh^pym)w2mM z`KW8N1ccaJ(9K2PtiuVfkJ4qCch^Mw0Otfer^ztIG8mGDXCccn*~j)=lOMMPrAU_shBmkwe) zx;wi3!7ZBBdhhr(fX@$*e1g5sHKa?H4}bAWCxVF4dp=Q3mPq9L`Y*Kap=$@e7f0A< z5bxg~)iK5#L%%F}W&Axi1)@KA1&Z8+c{o{!e5@7c+Zt`Teq+Srq0dxE0k^FdN;c}A zC5n=olj&ipfmPth-T=y}<;6or2&>OO{Rv#@F}WdR=VSwBsD>ASRB@G~Q3_}c2#oda+y zC`%=k0$uVdDB@3c{IW03ry!!{jo1~y4S|%Oo@nD>J80^%Wr^l|=0el=+-Po+MP*`c zN;aaMxNaxzcMLv?{qMQq%+Mss#pF~bX(9BJHf!zx&tru!S(7LGZ*b2cl5$7StL29M zm1pA{U=%2l09O^mNqW3wxMTU?n0~Z{G$SxJqJ)V^Ak78LBOxcO$ct{7iUFR} z%N1a90|ZF#B;EQPi}Cb=nM~K9H>9Zh>wJDT@Zc{4{w)5+(X~K~AW#8ci;c@p))<0e zV@Vf1C64gJ%sIOLjja&4=qTtvJG`~1uD)p#-Gr*S%bqj6=la-h(QGbw2Y)c%Z|HU*-KCJFTK|NI#*e98kj|v1*mR$6C2xUYPk@u+{mNT{FIW4 z&%wJ|;rx^DA^|O^{Vk7FxM)35w!DJhW0j$6mZDd`r6atVzcoy%{;C-`Q(=qR)a(hf&#{O5=(GTElM~Ybs|XUZ zT=a8}O}%%BXGzKLbzC6q%)*1tkr%{m@v1yl4%%BdKTZ+5`EwiQ;JPhwx}E`cJ(6mG znMDX?7^Ummx7)R=m2aEmAFa?Ah$ZAOc9X{VDR(!P|+6w?i@yxGBl57?5;XLLB%inQA z-J&H~otrSf(izv9@g;76Toa*TX5ZjLx=UQ=%L@Ees%mqo`8G8+Yh*>IrGM2oUXwQ? zV@H&x0JC$&izYt#`gx?=W-lX4B3##HAlisB%`TBMvHZH@&%djO*wZ(k z-xT{xM4=d`j2aw4#|8m$9w@E@Y(r$H7&{jHH`&BA8Emmv_rB2lRaP&Hz2Cd{$tv#+ zMdX~!BpxDqGGc^ge=U+HnyoYmY;x)lN=*~B)z($T`IvcL7|Z0M+-6<21H^mSEluNR zN=!V%wHJ?lz(T zyiLGcbSYvZ=8sN=9ujMlcM`)JpjQcmG5AQp$fFzcRvI67(vVUw%uL?{`v?A^_nx0& zGGh9vk`%l$%Wdej9&x{ymvWZUVrv3tfV5d^k$f0+=T)8wh0XpFz8uEbbSsa#)%g#7%h#rv{obG{7()fFAGSiTUu|DhwX5b9>S^~_-Lu=0iP zdWaw!uXs|gdi3RSAx>BEwPqs@YpKg)Il)X`hFSh)k9m537P$~!M#IOiLa!Bc=s_Jl zOOl6QMDcE;2aJ?XB8S|lK>F!bEhBSpL!5s{saZ&J{q*){jRCLP3zZG zojQKT?4ld0#wV3b2BULMaDCu-B0r@Bpfp+R`cpoOm5vELxRwiGDYikM0X?w$BPZvN zbO=r&2KZvlnVQ`lK+E5$_X&w${)+*rFeyVh8aX`r2+!+PdG-aH*UWtqSmk?%CL}vw zdX}tuJG@{VN!<9DME{^M#9^LL9bnSz$i|JOqqL!`uP#^w46erMmAGW-XM@L$AXnw< ztVPIL(WaMCu4B!_a-(4 zOR!^gF1ioH9sMCwJvk{hb8zzFo*fn&4=0nUj#N8?j@*0clrKsky7xrvz%{YY?Id4Bo81Mwc4O?a=d^#T`5=$ksHQkUeA zN6SNnI=C?{!QfyNwV`iVyj_Wivn ze#Bs61nprerSsdAc{k9}MdhKNc%7Y7D}hE|Sy{6boDD#;F&(9w+?ed=!iA5sT^?-V zt0BhOy0rZoaxXKIG!o*``ZI~aUHj0Z_8CdlLxEcSglrX)qPGZpzpoqp$XBy7AL6oG z1kSOh#Ae_|u-%}~Ec!X4l=9!-6OL|wnQQ&{Udvq1F28trI@4pOXj@ttVe}IpEq{h7}^v%9>9GCe5fSTdYse|`{)-W z@>>hut3W}bgkQPX4<~)$*vH4T;f$_Rzv@5mmWDP?+iUXs9+UZziRn5^Z?OGc7`^4Y zJEO;huVK2QC3P}18ZZ#|EPZ|gSN0tV5|Dv)tF%9)zA1Z3UO5?mg7l!_D6Gtl`*x#E zOdN8`Is&!--B!>2?{!g$=f8&}|9fN#&lwO`O*azw7Rj+LcgC#n6eYI1q|OV+PjAY7 zQWJ_`8~S(+Gw8huD-$XV=`)Ht0QCQZRx8u+J=3Xn1HNnI%BPyNBNoy3u^{td^D-n# zdanD=Jj+{ur+&Ni&BpL-c@P*QT01EpXAmFv?N4`_MDDhol!>DI7Uk>+S?)3xfU{GGdvEGUC_+bjuUtif)Jzs5knXA~pLqZI@ zMGvX}`i}leDoH}8jcon9h`YVN+YXg{|lpJo+7=34_SDk2BP(|JvBUc9mv`FY5B<>JLD}Vx;s;C?#PwLGyoHTamYm@7^Y0 z_=VMn>&@3>7j+#-()2}q&&CT=j8F*o=BhzW)TRxz;ZX1Rte&!)inf@USBws5ntC2> zs;hSZEe%GeAb&EU{@3yU+mIDK83tf^!{0m)mFJdsuqK7@k6+l2zC3Ihz6AusSofD7 zi7JBLpA{t3lwH)(MoYN-eWYSiw^LTo>}0F9!iTU#p3FaWr|NDh?XT4m|3Q;xSSnem z`;KH+c)Sd9Z=DT%!=W}L;b+^t$g)J+)hRM`VWS)X7X3xQwJ|X_(@@lT&5&1;sS?}q ziF=P;Z&uteCaxZlP%Y08ePIg9GreVHyeBKvmFe-j*EU)8?GLieBEFdXGaP_=^(Wa@ zc3kC#JG=FB-T@U)jAG}uw2*5b)w+9AZ|Hg}+g4R;LinGxynP7TIm>iQSZu#Abaob#lPLL`?=OAEm@pGG zq6skmvFZezP7IMi#8F)e z0Ak=O;lbu?EpGt5mH--EP^LJYv7|4uKj~)NEY6*ig2R%_B(@7^adW*=5Xq#1*EBSD50`b)^#N8Rn0xAMJ8iNcq9- z795^(d?U%+9Q3bK1pR)NBvM)U;2ebKQ9MKqgi6rUW+#qX-_1+^vR9Jo42p;U(L#N& za@nI7$?j_^UiNCW0F-I8u@r1e((NV__$Rmm)e!Rb&K#qMU^CMwAGT-zIy2n9Xu)e3OGyHO8lf3+e4fDh2=7nuaC1)iv{25q6 zBO&@7Cx6X-=ApyQ#VnlP*s;3Fmpt|WH1!~gb@{~!G0 zl-t~T+C`E~SK*nCSk9)MWQn3`HKu)SizXc)wjas(MMIA7TF@sq3W}cM`ElclDvkea zDe74}6TVAa`W!zP3uRXObijpq4}C=a4sh!Ky_{p~sh%U!_a=GfgE$2yFB8xxsABMI zUE9o`9gZ4?ug>0r*#}^)fxwxw$U%KAjsuKEO|J>53J}Y$6y1VfvBzp`R$1!ln|}u< ze4O>Z+DHFbctXa>P|)xYTUHsVdaJ+|ToY+yJiTUm?Nqi7n~G-OiGJ6|fd(lb&#)pE zGwrsAQc=KY_7=Dem&Mq4CW!l^Z;3i4DawLiFB_T5ZA+VM_R10{?Ge5fzwjr`^$VSH zg6@a#eiHYevKpQ45mKu>IQ{pgE1eqsAt2pe~S-53==^w$Ve{ z?^?RknR?@YXGe}MJb%F5-rEq*~J89 z&d5#-3r0!W)tAi6dBAr|+VU+-j^?kcdNrdO$j;~-J4Zy>U7b4EM?osY&PL3z43QBf zXWiQqkgOffa!uvLCqL6M=BaIS%Vu^I*m`3s)76hgbHwQ%v_|soue#m~P?EL~w-%54 z)r0{^WaV70(8G=qd8EN zDKYvw&qe&Yr}jy=-fAc^`I`b?-U5XXa!Rwmir22Tbyqcxp3tW2RK3biu?sLCI9(i> z8~K`Us}dVkPf0KFP0@=2#`>mSn!J$#Ubl7C@G6e`c+g#MRas_9<=Y2wU8;vDM~%i{ znfe)eIW+%qn1lORy2bez%<_YnukVO;iwO%TPjCV3oYaRHo^E=k1Fdva==MeO6jJQ~ zj5rhdgx*UsANTF>hO%)B(mSS8zg1-cY8I7PS#-o(u}r|8 z=CYYD@;kHtzPera#nc6Q1c_c1qmQF1ZNQtkw#oJSS*He#rT}(SlKu~Cvguci(u7>v zy25KT7L7QasB%Ksbamp|l3>g4c{jd4f`Gx2Zx;CRx_KaNj){bfo(8L{MWW2Ztv}4g zrqR<0J}08Oi-YnlmGB?~m{-k6#5+Et%sI~`_p;Y~UPIBTOVfm@?(!*p8mZ_5N|Gn2 zm7?o$LPI})0$8gN4g-7`3?6Xl2lMr9?Pk^P(Wb%t|5DJH=~cvx@SDU4u1iykWb^RX zN5!V+>VT92Pi2XgD~L(X2&~&|J{DU@7cK(kXnf=b4}i4zAH|011MB4@F(1W1qo86k z-sJOF%~Ih8=broFn{5}BqKNcV#(W4anwv_%0TX2J)11GujjI{E-N~;eS=S~xPk82e z$4c+H>aln4xy&~Qb{{vrBGc2#;WG!o@1z9bJ!3zs49F@{kTo5 zOwGjh_71Z`BAe;Vd6AOdxvA|_V6hnjpJJ$5<6Tw3RV6(y#CwH+8xQTC-sUNgh{NnetkP+ku-z6t_g!13y%$4v z5!vfFHchTshDvP^o;F771-Z$Tl(b6M5?xl(>elKDrolpW<<~k30htF7a3fR{4UxGk zRHOHkWx}X&oakEg#Jw6W?kzUtoq2E}nEtw5Ya!kBP!alHXQ-yAEakH#)j^odnzrW-jU6E?AtnqLjW|oD z#FObi%Lt#s*#6_^|6hwtyQ^d@ak~pEoNSogVhgup%0qkN_#k3S>yMM8w#m@-as9%A z&Q_*qge{a|rpy(S{868r#~&M@Z5%;aJL*M31ID{VBbrV6x5ATKB-Z@B*5LWzB;x~b z=_gYs&;!FRk6^a3<&J#v_Xm-uF57E=*ZrjJ#7RQZ4A<;w0w0-r{G!HPmb+&?L5CA+ zPgN|>_j7)%TQ8-wC0Cv{iw)!Blu3_P8$np=;to&^^G&1g4I>%)yss@k+LfF@X@*GNmZWl9rJd3Y>0W9funoz{cwhme(Bk&8u%YHM@tSeU1*|=oI(p_ z=6LWSJvGIQN{LFsQ982JLA*9hUW+Px{KwWp+m44b(Z zGz8L~6r5>01+)4a=H9H9RX>IJ%Nlo3+q|5uz)x`Lvar_db0gy?@w+v+tm-!*yvi-g6Y6F^f&SzU!TR4x z<^B>=3}A3eXv@M+U~)mN+!dzLGAC25uXqT^X53no%h4`tf0M1TPwRzV=U$wzTUVGJ zvdu2q=C^|DV#1HmNViwKKJUJOJmM%rP@P`OYWs4fFXK`bS{3j4xMJKPnxnRj__hE` z6LkhM4<3twcaS%5(t;i7f@xoeE_R-tFDW0KM?QxbZe_H)lWy|p;Y<<{o1z<{)DQ+8wL8+&sx(dktNEkrhq#_~YL<5zEnEbt_vEk+ z?JuEe4Drcm4DYL?+l!5Ji`cxG!V*9mE4kd<-=^`@Ln}~pk9fC9ODCObWpDqh#at=b zSuJvmWXFX6s)LGxOtl81M`4)!u;y1h4ZsF1h~{|xpgsq!_VEUe7vgTZW}^%ySGhUg z)|x1gHJVn!IxG>Hpi}zj2>jP$sE?0P}8l=|3294gYJ0+{HuuSyX_T5e%L# zU?9RHo7&8gBukMhN1KUL{ioF9R22ksUlFxgPVgbM%1g=f9{;C%EA~h~t2e{>z(ET% zH)uG-23S_RZ8udn;_3z&ndN59l;8T3)bb8@-%;c4D}0W8$K@Fgb5TRrD*o)qnHV*u zvvacGZkDf$EVS2CT}8NeUe&N^**0&GfjFTj_z0$dBh%FzktWxm65Lw@MRVnzmPTI{ z*SG-EI>%4o-X12MQNC=r%}XLDsT=7w25B67i188iMhQ?9OVH|yv!)(B#ZhDWdmeD| z-y66kTov*u<-}kJJlE%HtZM$C`n=pPG)N&Ol`9@Sc=||(pH@ERRYF^63ktoPa>6d!%ULuab?hkvs z)CS_@y}8vI*Jc<_e#2JCI;rbCA_m9k2#WJ3jTPr2AB>ND3F<`~eijk+!o*rX>vQPf znbl(c%61!DfrUYr?vjG**BYPRbaQUEHPYImT28)UgZEucT#Ly$G4BHwZEjw98H;hS zvf&2$2Z`{2)h0RRAQ2#flM{}Ey%25Gi)g&(YxAk7qp#g1)z=;?7Ewg+*D~)_I)b5b zRZ8|TLC3kJ{tEeA+8sG%c#VAnedY642b+PiJEY>_qKT89>JT8#7d!@LueY-fJDx2+ z(ipqr-?^^3Lc;l`2?qZ`6Y`lz`4xG;&ZNf87>7mp%nz-ROCQz6jErJfwA)P1f_JnY zDkNr1z+?{PcQcc7{qkXHiCzIB1(pp`N_`E#GWq7js%}IP&s8GJxO6SZRd*pc{ci61WeSx%t&fGT8W#j#4$bSNXwf_l3kfV;I>V!u()$?rQ?NoY+_uv- zV<$f=Pz8a-P|BeaKeUR}<>e{I8~WDP0BLHqO>v8#=__!<(JiJ`5w7G!Up0?JSjz~L zmm8Rkn?hS{oDO$*1WP|Hfpgvp(ePw9C6lEQKZ)1sft?2P{tt#i_eV_LFXtRuDl^Xi zAJ*O~D9&$N_a+Gh3vLPS4hin=jWsEzV~s$T z>w|SevrkKWlfxOuJhbfEYPS2yi5#UF2$;)-BDppiMSCm)&D4pK2q-c>SGAJyDras| z3V1`zB>ZDSIF;p+BsU|K$uwkVr!tEE%=k)?=Q*EVz9_ zrswqtK!zqMp+5C!eVzUs3F~YFq$|8X-%ApO$Np^G2B(P7ffr)=sr2~iNaV@ubw3|dgVto$=@!Q-UaTF)$ISzB871JElf>B0 zy@VN9Xzjzt%38G^7s;|$wf465N08C0Ta^Mnr?x)dOQAPY$=okNR@;6&l{BnN)2z~p zuyzd0%~V;-PQzgwP_?meGr(?T8(LV+Fsj9JQOGp5eW+PiN~wF3!UheFcTQsCm?H2T zzTgD&*k-!LqwwIu&ta(|G%nIHJy6sZG_VzFz)e$85@KW0#>y$FOI+q5IgVnp8zEcC ztu4v+9>h72_>cg-@#k0kLt1ffI`5FCLdJI}cd?aZuwdr7;q7{9@3=ro)(eF#f;VUV zwhQX+i-H8|@@o$;bZ|cd`jX!*jeULe;=nH*%Ell#=^8sUUuKLI2BM=h<3W%U@}!Wl zFXHk%-qQ{VOVP1^iLcMfz=Gg1j2)-JYG0-l6+JA|*-}PqdPsemDOc9hf3N#}iaPdl zBgC_4o#Q6&5{RFVzQjI+CP^;kB5vO~50{eMzkfqM)ZYMG%ydxZt006=R4*~^yeZgD zH}H8Q9RB@Mu+&C_O=)Z07a@Pc+)YIt7Eb%lLs`FafkrvV#5Ni@>dnz|O30i?gex#2 zN$*K9$J&*-$ttdH8B04e-v_%`2Ed5ZMr0~wmTfaY@x!tF4H3H7HH?v`V@yRqq zc$yBdF#gjxrNZ%YQywLkyxDjK%Aatck&g8c0v@)AL1g8rLw0Q*{eK4doftTHe+jpS zEb-goIiR;FLNP0H0MJWCHi?i|2$r^<|9w0F(>hgE^_+%sh{E;Edwa}?i8>}!LHnNA z1gutEg!1B=P`mL_5$Oz3=PU)jJ!F4(N{jz%BvEA_|(7~P{D~UL^JVu>SYj4spI|Kx85F_*3a110{l}m z>FyElWDs>Q7LxfApg84ebkkvhs>(i{IDF3Oro;3uCHK+J&q=d;`8;qJ7dulKkyeRQ|T|#VCg@n7R}{-efamI%K!JN<$U(s zB2z#4y>=ODg!&IS8nIiMvtKCG5%5IXi#3<6pLlm)vPk@yyICs?Y})*bG+y)h&?J`%-fc4o*=}+rKkhK+I5wfpS z2+LNEfF}n3zeo#l2(9?1hsG0IzipB>f#-j$z&}>1QtB%8_Aio&$m3G(>B7qAoD-Xr zAIXfEp#l^#Y7wLW!k}X0F!~}KY5Ci(`D5b}?+=x7z8T+2$fm%D0uLtdA4k=n`NY!? z$<_V;3m7SK$4;|*L={I9L{ee;qk(7W;DRIaH)|76%mJQ zDpljRTbIU;wf%v(dG0Odl)>oHHRmr41Jubin|WbU`;J_Q z%M;I4UZl{S=d62z8Q(Z=^&6t!>gBKq@J&@^yaHEcP@`>m?f0f3Ji5-;Zqq9lQmA8w z33j*j&2$#^_jGVCdgca9^Vv!TvwTRUeV;0RnH5GiCd4Vwd z9lnVZ{XE5%PLX&7?=+{&Lp0o*)%LW>-Ei0bm_AJ+^JwHSO8Kd8^4OUqEs)<0i2e0s z)KN&&BGYBKW(-wRXPhQ>NeC_>M$&Rf>$xvnBX9M6L^}h09_mNQeLbpU)!LV^oRtK3 zRKlPHIk40=DT+Kv1Co+&DuPvSYwIhSV*)(1wkM)}YT=h_F5#&|@$`*4{cNDKny9;l z?8~A*H!?n1uHVK}#dsBN!r^I~q#i1Q5v0Hbgim-Xf870Acqc3UP~>}CGcC{{Lza<* zmFgs*h=wh?ocwcbiv8~(l)SPNVN~d~W=BK6GVGQM{-f5UNuRu1%=X25hJ|W2fRVID ziVgCGGgA~RW#^wSQr4!vB44)&iYEg^(>U?1(4(`kEvm%RMT@w-IWSPD^BZ3#=6MJE z!FF{??6aL3`jX%?q)p;085JJa%zpBxG}`5G{eC&qoRM;5 zqfZI+8fi?&4-+I>0tvIAXya$B(hY7}ZvI*E5ueFan{?zZZZqd&=r89K569?eY{_dn zkrH9B6R+T_bjlO_?#;hnH!wrpOfzWSPI^SsY-nahUrU@C<|7mAzV;^;>xGHX0?4q} zl{}a*hGzzmz~!|cKREDbxivK^S+;7#JL6D}sor6E;l~91>c5VE!e^^>SR20GEcRi| zXQ=}`UaV3+*M@MYL|6Kyc}8vPiyR~!Y2f3CYCA#OXZMSNcbi0Xzdj$}${SA7pO+%M zgh2*KNiJN$$FeorFX4RZjdE-fn{VDuH+21V|1G7YTN$*^x_J?Cl!kuH%0&htDY7zf zwFWsHTX?q1b!`z_q`Mb>EAm;!G<_MzG=_w$)>Jo^Zq2tz1PjRi}uwF}U`)P`#{tHD3+yX9{bx!8zHP6JS$+ z@f!0_YofT%zAX!Gqj-M?vFTFq$?}h0a`*k6F+Q4*cX_JdQylO@`65Bd-yS%_UwE;! zIt4)E+ba|txMg&GV2KJ?JZ_m?mC-T=r<=zA7;#Z8mSP2dE*Ts?#eZ>CQ|ikTIo#L> z6|=w0`|*W!ag0;6=>?^fLq~GMx>XUbx=ZzPc_%Zicdp9uqWf4;i(QLyrkhn7gJSEH zFDQIEMEL*QV4ILKbW=}D-Yz(9@K%=MaWF19lARgPW#xMJS@?jrM)`)IO2E6Q!$ z#03xJI4*8U#1Tu731gvz8p%!<9fdhbKR8_{#Qptv{P|A$S3JIW& zGl=eau&BDSpa4!0`?0pa(tpVY#8?h^v9qp9u-|$s+9`^Sam$p*l7uz`1 z{`{kR;rYs{_2YUz$rg+5drMjB9uFxcFcR{?#|sQ{DU7unfwn|oaH~_^Tbp>HQgfBG zPr?ru#;Ia^MlMdWDgx-h6m?JBl&5ctXKTfo*i;UqE(U@py{WDC2}2DKthOR{CPQ*v zG@#;7>X2IbBM$(Yudh~IMolfE2~;*}np0iyv6OhLp3cVvFjoLc5>30kKcP6YZo=)5 zFss%3P(nVW$x0aGn;GCY$-m+SOKn-k*Oq<(1p2Sq8&?b!Bdgucl3}Zc4-K5X8E!Mo z&{9@RXBnbk1tY>P=N#E!)bc;Nxck-A$$}+|nkNklu6{9NUSD+-S^ z)7AfGtDHJCe5NTOCCH=S^MTa`fU&u^@$jatES*;6QxY*en6Gy?qzsf}2KZCvO5VN? zTdhvs!jiV5ZcdeV+-$D^fhsk)36oNpk8hEgkmoVYrG_w*4>?wkk+1PTqMmU|GnF!UoMpyrT zILCj6<&Y-(i)3{Ae2#N8pn8bAe!u5+8yRTA%y{)bL;=WWc%Q!C(xNR85oPR8?@O@! z(TmUWF?H_O#(|Xsn4Y!Cv<~2~vdM7kPT7(B?A&G7QaUoeVDeTH6{Rndti4y;vurCYLH_|d@8czu$tlUSu z(>Jz0zb!*sq$>9?o$F~o^$A$U4L)Ir#LJSF)(D?TK<%-sXB%X?r<3^2^qhKv$ikpP4oE$pzs~TU%k?Z(g*Q`yS8@vDCrmM4+{;WJ5xx`wpX?j=|XR9tWmP7zdz|RLx&rRAA^J}G>J^>8i zvAVCm99>T?s-5paC2dEMn#gAW1pKt={>xva1o+d;`jb7l{c0_`+3oGoTbX%>u?OLH zEl=Bwny;#Sa+-qQ)!&|n`j(v$E_^MSvaE0YI@+ON`t2}Bu7qteJyMcc3dCZVC33}p zAteElt|8r3o{DadBbj5*p}5ZAc5uc#2p?#wJ(^5|g(4Lf&$FrO9=!=*#ako3t*c5H&oMFlbp+&8PR5Kp>q0Q6cL&x-rdQolDX6~KlQO01T5rQ zsH1$hUg1Jnr$sm~f-A$dh%UWWLF=Q7LlS23#IeiHFu`(_zBqe}D3gbg+qg&ePZHe*E73XZnfDaO;46d_Z^C z+^yGE;F{nL#b0kMP*Wm7Z(hIkWuq9wt=h-Mr|rY@ z-d`jq%3%M0qlo6S(Ww$}*Ty|k=bky^Av&&bIO(QN_ne{%a*AAV0^B_en4?8egj*PyLn4fP zxHs1r8kp1h##ye~$bmZfjj4DyI$un4Qt%=>d4_19qolzQKG%CQ6d;A>O{e`hedVX- zo|^|Y5{~_6nrtIR)h>gqTt=f;_CEG5wmalCnes__sGaKxXNz9tsmwG2*s+^lB3T5v z&7Hfv?=Y;&YoI@#)jA3;S@dNF$h8VKwH5F#h)#Hl^iu~HC)$j&TQv+4u8M|y1|XCc z`BHsSdj6Ms7K=``L?4P#T>SSsUv4QU@S=tmnp zX-RWT0*;5}=gsI|N=;(d1Os#I0a2UT$^CYw0JLj3Qmv`wI#sTtZ1a218?f7t3Qgtn zZ(5^>59H@`()l7%j`5TkMDu_sR#(%W$CXI_a_n(FQT;s;q4TEkWowH4HQ83z{km@! zF!+@~eul$FKfFl6K{5u6C7VkV`r3lIfGQOXgdcmQ7; zBt0kdF-Zv>3E!(??O*v3gh$x_`P178_%{;$pGx_Ej{h^&e3?w(QTF;8&69U|=Rb}v zuROD#SlJELY;|*Gy)A4-B^}y^`Q1>HGr>ST8Q=sc#_fRd~AJtkMOO)_P!~sLbCc zx+e!l9*?kzSGdf-Poz)sSoAt(_1udNh;9p`@z?7k4#Og>cp?mzxH}X-t*GYF;r(O& z>3rNTAFQ()V(lmFu-ZIGSPLV=;C49=jZ@DB@eU=$<3_3m|K2;&$<@Q#2e#?x_PSAjQyf3}hEMtzft*X%6S@u2 z1yyqHDi>=*=uxK?T;I*jGy7GN+{~9;%|QW&lzM503i<2#kO{l`_g930=b8x3=~LmT1Vk7=e?^o?mk*1504d|llfwyz z@63u)Ap_YA)V@h}n6m*jy@wIA^(VS_wzcr=3;p&sBx@I>OAZ`hN+RT*Vy>cEQ&m%T z>fLs6Q(?F7gl{y<1fd1b>bvizA7hztv8RY1kign z`z?j~ho@gz)G^TQ9O(w|CBbcy;EJ(L?TMd@HvF_@X@Fm zf(~g|DJmNveC78?CH`<;&)>FT@cPUs@C^t5*9j4kPvgFPh$Scn>}uM6t;Vm$`_*}F z4f>K!a0e_vgGjIUQKYj1S8FR&ihNut*(@P*&rJ$*@wM|?fYrqz9j8e?uUQPa@)k?) zvADS`G8foSD^-R;Y3;=;B{D=C)P+hFt9f5;K>tPX%s!{!Xr!ThQuW>(5|YFVJ%5(P zA-!C|`Tf515bu6S+^{G45w9M7&A5+4#|&ollWU~d)Uv%$E7*$7nkXN3iUFFqtFEmE z^DgSMvFb0PN(PB3(LtedO@JE?SvA)|NWQW51ec1uX^;yNq9r#4a#-Ie#`Gk@`}7yd z?JrVuJm~P}RqoaC?CsOk^O_j){lBUU$TacSQ*5_UR14p-=KeAM?_KNv#>sT+UPbCg zG!kG475|`WAFnfp-&od(uhTe)PHTtH0Ca=VjunN)Q^$0v2%c1W?k64Y+DVj_UcU%8 zTFpg?!lotqL&I!j@P+!E6)3etogm%yQ=+i>*mpkFa1Ck#NjSaPYuc5`rNF80#8seY z#&wqz8!D?oJ;x->n)QmPg6=t<3~_0SxsbgtPCi_8j*~*&%svdcyZ+|gbvS4-gIxgH zTy^1hB0jd`1gyzKRZC|^{}_FtNeCE`O$q4&!M@zudnI`g zm`rZ`rK%L#IA#xsA5=(c9b6W5lXRToc8bFU-Xzx>D#<6Vcehtxk&#)r)^`L*zem5Z z-p!|12!Hr0L4hHyB)uXX91@hopTw3;K15>PdpQ;cJ@E6~!C^ISqpjAvyZ8o9fO;2( z#53Etid}a-H_d5)H6YOmWzLI{<=8$o#H0Ysn4jdS`78S`8ZL;ot9)TlDl^?hJ8PM> zpO;plE-q{9p&APwo|NZjiyAC_Yj~!6`%# z*AvL?RXHV?je4RD6yHVu>2148(&&!iuF*=!3CjzkLUgQ`(9w2g!q|s4@scXCE`qrA z&2G{?GpLcpu)F9bFdw>U2M zB?*gJc5Kk36oavk>8#xwqkELXmGp9oh4zrMLz9%I|9BDkRF-7;rO_{8b6DzRH;c}} zCpwe*^zMJHQ_`3^HiOu#Ijrg#X~8#I8mN$2s!Z=n(`O zYc&p@I@J!!aNS8^+1EI`um~;B)J&QYg&8@NGXr4w0{IR3y6S2deq(PLT`N^0xOb0} zt%mB9vZ-Car|_fMs#8IFXP`dG4Liq4o_2n|2PHH~`HjZsAclFYw(U?xaBF(7k#q3o z5Pdd_>`V3X1>fbT7Y_Gtm&N0vu9WPTo6`Q4Qu_v=q0ZL|-x*kTxLP@@2%jU%?l z)wG8*n)y~D3}gmYN=<-HVH30Gt4;&VgE;b&M1HHyG`VH`+eS2u=6ZeX{opQgk1^TU zBmW7hF~XTf8rNOCLQ0r2+0;DJ2FciTKISsL$2f5724E?q+FIQ%eU1O* zv4bKo=1A3hz>1+xtqoy{i)}?Eu7}h0no|@Y%TpICM$Y0Tgbo3^Y8_q_y*h_A8>Y2s zg>l&>+uP&THi45iO#(b1RCendD@Csk=zwkMOjoul;!2wQ_vays~4$QMVRy z!U-h&6XOnCipNz@NEvYmUXtc3I;Y0VU;I)rm>y;^yFp8z zg;g&r%UyP0wg@gqx0NNLrjJEVN>IF_$4NyYN7MtYWWPt*TE&D-RH~#1ZUt+PvAX+|iX&H>1%c$x&R{W*l()T%`S~5p? zljx|hJIozfg@o)2=HiycP0P7xD&jv8x~#dw9>&p$d=wpt94{@&SVwvgKsu29pzNI; z+YUMetcJ$8(WDdRR%i!h@N8}imDE8yslG8#bU5m30L0e6vb5}Pp(!v z71+m)RXAzw(uuP6LC(adHs+AO#Yav{ijavIcWPT&9+pJ*ml^ZBCh~a>KVckn=$7ld zCvV<pTo0LASB`ShnTNdNDqI2|D*%4}vTZ2?NGD(GX`9RacWI3ve80g*-dp&!H5jBg%xmNn8K+E4i8_G*gJ8PSKVl9nUFPcYm`KrrzN$7*ceh3Gj2W z$j2s%tGKz|UId5#zO*WJd`%Ycvk|W#Rj67P4=%e<^;K?31l=lf93ycHXmh48fpeFUR&x?iU-uB;6Y@IrP45DcaD#8V}2QxM8!n zg=zIxW+lM2Q#iEqVG-qjm9C1@( z%UNG2>6KCANvVw=iaQB#4{+oHmccngDdY94DUtRdt%Mj*5*#SyOgxwp?`0=CgFQ?t zT-hCo88!g#CwXhPW9PekGJ{-ptlW$rzQa95`MKAbN|4fq<-RKv*X}ttLZ34z3aad$ zXN6um6}@~*O;0bf;_saDWo$tb>#%myeH}OWF@4MYMwc>)Fn!f-ZB%<77(7QP?H8O! zD0NLuWr&ea;XQnkn7?|eyND0r*yCF=Toaw6=lvtROqF*30;}Tt{ns8V^(`Kp0pwa*f$dLXP1rmR= zkWaL_9VxUxW{x zA1=-d@;W1OQkdza@hIi?{_)xHL<`hL`zw;`9T4t;JxCYNS9Y#p^2D2scj&b|MrKUB)JmRChPq?T0FzXzwsuEDh50a(b$0|U9$%@E_yZ1NW&byp+NncNQT zo#zEvJQ-zlecRT9hk(c0mw??i;gODg%%-?`sl|N-sLp^HZ4AHYc;zEb1E;qBT8?(Y zv8sg)AFYB273-{o^mwAuw9%giYr#Pp8seu9M!F8VV{sro$Hrx9IL8qyYYa9wsV(Sw z8HgnE2PRgDJXodqgLuVv-sz(_!|TmH+oFh|pKeNu&QPHh#`_Znw5yE4uUSE%4ORZq^`h2X>&rQI(qZg&IooGb# zyZjwW5wBLy`)*!JNeWCVj)SMd5B+=kTIzrAt(45!LcDGn z4c!aJ36Gl1AoKB#gD1phgpy5t!L%PN6mD$xokr|GrcLJwmVTt9Yc|uL*m!%SYag3W zOlA(&V-ReuMdp8u0=QlEZTW6?nNoyw$D*KZ&1{+jcj24}%r#zu>4`>RwegJ1%u;x> zjFTdcaFbw9f_5mqD+&uu?|`!Kq8ROndG_}uPJyhI4z+pOm%kb)&zVPz7U0gDFDaFC zT!*a5u~Ku+><{-bC+U}FIU-}NDOb1Pl( zCLHX!FNB;%th*Hrvh~jK!>0vnUKv2!u+mIzgah zR+;w3U90#Quvv$4%?xEr=ps!c#f#G-e0;IjlVw8HJCO}$P1i|AY$ejI#E1q|bRUX7 z8HX?hgrO*00tZfcE%!zL9! z)yax(riO%_e<+a;8&n_|wSHi$g-)vg4#2fvJArp0as?b#-wsz3ZGs*uH%1)grBkYr zLLCyH3E+c^K=DzO-KrbR256bBeKzX=yc8 zMWJ8%bEAlksNOZmWjZHKRA2(DUujXux}%JcE^MHYnL4iCN&z>rbvxPUhfdPbs^IKi zuQToNVY3%HECbxS3XN_W(qAyhjmR{}Fe=*`xoX+1DK^?0-ky}l6xyQ6!Z!HRlM;B5 z^woM65IE)a|FRSLf07SvGGKaEqk5DLgh87sMkFqr{~~SDJvWkZ=cPE@AUP}`UE>SP zQn@Wu3-#yr8FMdKP5I1-+}Wn3eN%X3K@+CJmT&)4#6hm^EFbHPqp|%&a4Raj8anbw zF-zcT+RZQG z-f0F1N`D@nqONkK?CJ{d$~SEbB%wGYmlxiP1vxHx9E=>b$$lSki5!Xsj!Nmz1iwrk z^KV{mG`i@#FUT=o9AZ}!y4_oBm>%qn6>F% zSuA{2E-G;f|Nf)IyF|pwac9oYLOJ~tl!s_Dhm(kRLt5%Mk3aeRWgFQ$TS0}G@6ii{ zKAw*S!>iL^dRy_CnLkAKCsNh^NAs5@xK`e_{k~oaOQ#U{eInRrkmFM0p~%IVWkE1W ziIwit$*Ub>lsMt<3j=FXuF;=|1v_Jj7s%D!ACxP1wYuwwvQHij*6)4dOv7BiaCPzJovppkf`HJ^9*-l2OS&c&dm^_@y4;+rV{W;K79h5FZr z38!&zGq9M>q#fy|W;h)V{xNxDa9-p?=jfwWY=jCgT<*x*gn^7oxBD6;qZpndOw74) zoB2I`aQQt0+?zL{Af<$lOTVPKUns9ly~Dc8yAQMRdc2~nbGjLsUJVfI<2i1)w}@>^ zWuQt4S?)Gp=a@BbWfa{%8*-{&1<#eUJQh*T$Dxj@~a?g=? z1I6gfJq305eknA77DMRTrwyY+8}}ZSUMXDuv?@76anWJmd8_T7O(GK#DUsHVwT2eS z@a`*G|2d9N$QKEYdtFv85i-vv5T12&!Air3#w^i`qyQ^Y*k2?*lBmU%CfdbzzE0ga zwr^$?te7l%<8+3ot!Fi{uR-bF9P+YZ<7RjZOPu9B_%*|J`IBw%kljwvNwDtMIKr4L z;-#S39l+-L-IDcs@1g{(I`1$bwPyLTc-Fz`=TY}`?k-5*U%6+~LjbL9 zOUqJ%SoEf&{>mx^!Zb(K=ztZn7&ln4Vj}Mrapi_!%|NkMT=v=G?X&G|eDoOqP_gGX z-eQuHhj*%)@ zB7rbvPlZ_=?8TE@%Ap1CVHN(28$Ea2+3JTeQ2P@AfDjlnDBcszICWx^SGvu!f@ppu2CQ*!};WT$U35*F-5aizm0yr%i?H-ID8njIaK? zFb1=tiqUggi4T;7P5zsg$V-MOc;K#8T6vKpbZOjkD_%^Bt-V3ev#U_Lw7xlddYj5x>8U_K`%YiHSYtDz z2oMCAIRqZQ3^G(iSk3)OHvhMp&a9fBVyxsY`1!-K39;eoZU!gUW3;V9vNnH_WI)9OQJh9mGbzP#ncMFp zUgJ1Hj=Jv87U#24u1d+q_CWMT9L}q_+1P2R@0w_Srw|siA$y@C?|)_k^Pmqo;+plP znn9H!-kkD!oE$0Y_9F?^-?Fp^O!GN8U>TFRFq9zlwS8*9DQKs#J@;y(Pz5FP>g5Cb`|Gi z%9F_w@$dXXTs%}5g*2^Bv5J06(x0;;1@TML>t-8&R(;lf6NbgR#ywk~Oz9-x1vS6) zmqQcQSN>u%je20@oEF(u5EXd$t_v(T+3kTOmJQCA>bGc-xs-7CRtUfI-JM z6~gj0DAW!4y%H>fTJ72IOv>mgq>|)Ko2NSV)!^O3L|J9a^eK)@ zjm&bLuLz7R%`4NG1Ah#d7vtbpBn1B9%EIh2q}fw!dGTsY3FBCut})iD;kczZnM}F7 z(cqWi%xOn2A_Cj1Q8SWU->Zq?q%s>^^EgsXTDA9?3vJmJgftchDIukNsjK(%F9}IG zS1w*#^{QRzjJqVdGhkSizj``2c%$klGNWx)!sVib39`|UWLBiqq&`Oav>x~vzGUA# zp22>Lyvi}niGZU3n4240>)v`aM_oUX7c|zat|#rzy!2b!{NbR?Ue7W8ioL_KanVdv z5vNZyAHJZ7u>5cHf{`clPOe0@OXFE+ zKZK$nh0!bq^ThF|>lV1zZrB@1lKYg-ImB=X5_89BrS)$?i{RgVxJmFnWMQ$)g@#s@ z#zhB7Bk9$&iA&d8YtkwiePIt}kNrud+=}k#(@FanG!okwu8yp3WJxlUjhDCE`l28}g1~^Ku~)8B zLknB@V;M8+2V0tYLM06rSbLJv-Dz-C({%`Q&=@}93Jbrto-#Of5JT6p`vi;UkBQ$; z7OgB;{QHb5viL~y5|lQv2??oT>kSM`C^b}XcYysYnnL7Mt@HGULUW+uKJp@mjXELR zoI%`<>Zh47f;^3Rbw9Gn>o1Q`=A+y0>m*^ju`#3=}T&eUUbf+k6ddv-Ck>F>)nGPWF<>6J~?a{E@>;7%1W2s+$Gq3ks{!L z3%^I4^LegW(R8*zre3=5vFME|aw9-GWt+CGLz&JX=Ndnxo0H|X*GqAR4i!?(aGG9b zn-9tFLInnDJ#l$a+q>?sYWOyxdPL}eMk&_N&Fp9uOK@0jRgved(x1d5DRt{krLL_^ zN5j<|G9^dymarjeqUgbRen zQ8=Wy())L)J!sh6$7ip)un|iTwI&vBr&GkyzDmk6koo+520u>{xwKV+vL|Q$?ZJYf z)2g~ow2_ugZNKNNP3uUMGce_MzEzzl}IU@Yeiwr9|)xcn@}Ho`OsWhQM+{IaDg;_C7Cx}0+*sc2F^bdfta zZjM6K|F^o^m|qQ%`#OP-x{tU4Y+dU&NZ}_->aePXESO#-UnR>l(NOn?dctvV-D-6Lv>pXl3Q85{cwoRy`Wa9m- z9dFxYgoq)a3W-$Xm+r%cT4M(Jgr(F)sdAp2Uo#gZ_pOR8}Nl%j~di^g{N;oF()ep#y9yjMSP!COXP*$>X^ z7Heu|P@0XVza-TV3;Ib#+Ia6WXFst{*}cLocsvvjcA9ehHD)K{*t~P8xlyk`xwX*M ze53vJ_8J2H2IgP11s8EZf4-GO13b4$E_{S!$HxzL9RF!brwqSR)+xY(vh}AMM7_hJ zv5dvfOL&2xxfD2<&^mEiV!?N5*&`iC!W$>G^=L!o11BhEiYb3Vv)TObd|w%q9jd9O zd|K?TUHb@UAH4O3iccl(G6LyDao4-YZRJE@T_V?4l`c8in{}H?(J-D*j=Tx@56nS7 zW%4M3@`nIOw`6}7+CF^uX-PCM7y!%a*E}e{DWNmN66w5)BAwBxSEpGE`R?3RZ$%+^ zAkMgbW3dj(j*Y5qK^5Iii_o`8Pw^VJlh##43JUYRpX*zi|Gatl^lkna_Q&;9K!JwO z-9OC7-j)k<<%cMgkPw4cX9er5t77l%PBGWgN5hTR@60r_Y!nm-c_FS~Izx0}GR0$e zw!Zm@MF-W!Rp^LgGjuj=cb1q99LgTYU=c~GNzr^%Jx9JCaJXjR-mKKAn2jM;^~+27 zYYeSMW5jkXVd|X|gRP-oY%s5MfI>_CFAw28I+||C2sB1bB3|rxhE|zPduWCx|5`M; ztH&ohx!4t*$zrBT$Xi!Th%1?K0GyC9>;VNH%g>R?gs2r$$Mt?Zko1(GSeW}R+{^!a zepxPQ1FMPNC<#Da*D` z3QsIN_1fv0x0OHGWy3%o);zD2^>)Jz-yX8VR0413C51w0gf$S3x2Ut9OqvqDa`&vR z##A_XZm%p#+Ljm4neF~0=x1Pq^&lnY`D2%{1bo{Q7~iT`w1RC3uOE9ksCo2OIuKQ+ zsHr2enx5{(MQn10q@v+nd`0cotosWFaYoBd68X#|^cIsWET+?=AA=*d39dcUVT-I{ zJ!Tfy@@?1);TAW_$+wsuaqyRZk0JF1{orP!ZRykzTcqp+^rchzBlbl|1ylsgzaPGG zCc#Z(CJww~E(ggsdKM_1tR&e8<(9UbDkZjFLAoe*$usWghv-dr*3-5>6*P17K($CF zHBP_ufE$nI@Q#2iON`1meFbOySxXTwX1$|q6%6RxhGvfX>!|=SMdZutW_g#zZ!Z%*r!^ z&8jgaze!#SASFVL<%xXz*hrWr2Yal1*RC?Stq{g|yt4aNbtLkLouS#_QvUyI?=67Z zYWsarS{w=#iWDeNT!IvDaVruiE`=b)HBf?UTU?3-0>z43DGtHmEe^qoli*gI;!aQQ zy!(}J&;HK+&g}2rnLTr6GMQ(sEXlLhnkRXF|DT}N+vK<7yfJrBWA$_*h;py}{Bo1i ztlWI@*vcX%D$h|j($SaJnnT`BO&231e-Hf?FwQZ2X^856w?mQ@LtIoi+>y>o*RJaBXJ=a9 z7}UMP+s#6{7PGtmAz+}ubGdbEzV4E)pb-6wKAN5Wd#a6oDa$X6el#)`5qC8;Rk=Sp zxZ^iiv_o2Zr!}W(9VctNH{F&zRI>&>-ocLinmhkC>gl?7X<2CD7KOB4SN<6LMnR-Y}abTrxTf><{G&Lc$HaEVQZS6G)|$T`$||x;!B}V*Y%3! zM>U1s$8HAdzM$p2FX+^Zs-*_#g7WxiRy}+!jN%YDs`!mU$gRUmfm+t+$}uVjvz=AO zj^s)oCH8acg9pxvvjHJTq}P5R6y8ugCKAZ1JT-#!(h~8fGhg>j4?s9qwMr&~_quD9 z`aXP#qi`r$7V0*A(YqXZGgV`0Zu)sF{hX&aPw1r^J*lpqdVm{w>DRXJV^2V=F*aL# zuMQE@+S*I}4DjEa>YZMn1We0}J->>*EtF7~p-)-5}kjczC!*q9^VXe?% z(Ggvi!nm^yTS+PH}z1iIH#WzbkQLSco*6VprZE7?5{xBQ6M0L}w@3k*YdH0UriE*L*r>J_{40Q`v z$&(Y^_H}DCIJJEfeYf%My7m_afLr<(hQXTIx!^C1`BIPTp*A6`VPAKf9FF@eWou!5 zQXv4}3U4>dIp%_kiaOTN00ZjRMjC5Ob)LRiU*no@btEb*-z8R@x9^p-A60k$!Z;0l ziRL*<-y+|^k{NzslpgV2r~bm&pPU5##~vDhnZ>o8+k1F*1|s@TOCsi56!f07I8l-oTtpSU~*VU z9n#S~?3VN;N9&Fw{(!z@{(r#8Xa=KM;47X*yax7mp+p4o){{571y}{l8^8z*{~?|i z!sVBa4yy!ieDdyK-lNe|&VzRKEzO&rx&Xy4gY=aVWI6~>d)|JOCA)-Rsj`fog;RWb zN@Yf}w?Jolgt(D?(^>7gAsGwj%-)gpu*@;IpHSXjOWNROHIluNDCR1S@RZ7%momtv zSIa$qhB*J2dD`T9O6aC=$T4Vl*6X$FijXo05x~gFJmC^pCyVy(41iodY<(L^Lz|Fn z=KD-jR|%N6#ou!~AEv^-QQ&|5qxWYuebluVVTKRZH#Ih-DoZVSk}F9G`((Fo@y#pj zwbQ#I>O1RH6Wn44{ql-!Xrl~lrz*ZYx^zg? z2yKoc>YlXXfUbptAE?Oc5MUce>1Ps!T(M8f5G1VyGW6(J^zK0qpq4dO@M|F#%W=Pi z8C<ErC0H*4@xsBGNZ*o}$&dm9)kDr3z7u=sf-VR!c2%s!P0QAktbh-uTo z;<177oc{}J-Zj$PTt^pqpEVVj4;Q{`0nQYRHUHoZN2!&+gP!@%9jOthAs>EAWoLZW zv5AGCm1u8!Q%L5H-|Wf3aW;Hu!&4PD?9nDXESShBxqDFigjPOnkpz*qWk!wTYSFz{ zrqM9>$%47UPu!W+__8tAo5;mhMQeJ-p|a0k?8{D)F>jT#nj#gxocwP!2etD3Ug=hW zUk{g5ZeKZl@n3L*s3Qhq+mdP1Ub+#Tr8z0z23F&^o0_QRz-_ zkE2ud*U;*K;F*QYbxywKJBLC%P23keWw9*7XP$yv}l8MZZ-X^F}YH#Y2ap^7?tY zw|RjwINVSMnN-_C$MXno5aTcVZIWSER!?PDSUoSqbsES zw}40nf{3{rvH1#4xN?KZvRXP=4R0_nI7VoP)F=%J9Kh2w`nb+XdSi+^ zN)XC3t$egvPhAr2L3pl>h~xrW-9)HJr&TU{h*LH+J2(sP0}xF)yz=-4Tm4#$EP>jv zd+pjpN>_$sSSS07yeyF=O@d5|l=tM2=W^7<#9~q<;~9$ATast59LfR+JSTall8jN$ zXkN7dl~M|dJCb&jpZj9~yhB}<5-N-cRZw5qo_N%=DS><5>qO|3U}Id7iOv|H1Xucq zlk=-L9cw}fal_ZA3$z=l!VOdTw5_eQ4|4R+*J`(ye(tW?dkTiu1!Qe>`6e8E?p)0b zbEkA>b{b@6e(L(eg{Nu4D_L|c^!Z9hlDJcxU|est(pq-ay6x*-PS-qXzxSvkk2ScN z`W&(Yr}-S2gu5>MdHrl|&O@r?F!*T_?6PEMUZ>wgcp|^OvVn)$7x%8xOpjUfs*b^*{KQlipBK>ac_{0NW z**D49IDwK3a$98NOFYE1q=KiXcPromgzlxyaJWtN?iuLVI5spWpWSyBbG%i{OM{>^ zU|MIGa{ zu-B*3y>pKE!d~!M$jPVnliJ$FUT0BB!;HGBbE3CL9Y61VIpCaXgcvrX3M+eCj0Sfg zo0u(>Yc4&|1bbb>xbPDJD^|HU;}59Z`xWi)ga_N7SAslB+lNIDX&erYV{Q~n_RBBo zd#)TK%tf!=*&%j~!J#5W)IR*5j4)p=@;f)?=acWm-m1`)MVoT_1~}EvOu|ebca={o z6V1J>cKgs>Jus(ZubsRwUarENd@5`A`%&8|$hH9Uo~lmSSC=XWT1sNK?*%g?BbuRI z7U&wckCy4$d+p?#?uXUSGnp6AB7s7UsQn~9deLvG9an-#1%>wR!0S*}85UP$_(Szx zFwv;dvp2#nj=}QNO-X(*26(l@J;$9XY`;PCm8o4Q%g$OuU-F>cO-g@35tO2jh$x>- z#NplF#IXljWh7=-0f``H_Keizer}$;-z|5B4M!9XTj4$~OOaNe=@p~Ca@$`*dic}8t52II%aV@aVZUn1@G zrMSIUfJ^T_U|{I>;Gig;?@|&;ltE1hrCb?PJh|mzRrY2@JajTvJJemt|*;W>C52?>oJSE~J z8*ClpVk#)!r%;aDOykQk8b6%9x?7Q4y_)xT++RJHJdAEvUR#DjI0uJ)x6?oMnJdW) z^nBuv~?Dj!k_;JhW-ES#q3dse6*ipO|Kx_GwEc(JK1oNa{Zpnke7Dm!grwE2-Z z-x4k%Hx6Z|au^s>_1v(i;v|eEU1tPX{}wQc1I;NTH9n>NP1}0o!Rpz^vWr{s@UXxz zHiS7?noFl=X*YbvNc|>TzAdlG@1CY#Cmkek`xk~5`0WuuM28l$`e~0xzC_6psi(gf zA`1o!Cr}FD6hA3DP@~=9rJYiP(~II<@76y+J0X&b z5VF}Dr?;Y2#f?2HY2I3n^`zd>_30&8oMcva%w(bCzF#bA-hZJ5ue1Ld0y}mmW3H$* zP9o9a)0#-)+B0hIOD;KgN&?}z%(E{Q^xGt63FEsZvsI9-;adjhV^d#dt#$b(o?BIg zH=@I0q>F(~CuQ}D6;e-YRN><26xp|}Mya7rjV(u?W0UYJYB;3^!XxRTzPIj7I0wyC zc`Nr;EmW*rvOGxDw0dn`jq>#7hJOnn%7zn$0HE=}$nP`@VLN%H5&kTl+rmG(RA<>d z7?^bnEC7-G*MmaOIt3-m2teHAQ1bUU-!2VY{vTG>pD+J1d{3&{87@JQ=o;3Q$f51Yd4%= zbS!1W%3tP-g@LwnfEa?h@5*LhHQfjtz~7<_uMbe&(HV~Ja%Tks-vgFrwKSZ8a(=#} zo^cBxw4Bifc5}FRpmV?z{jLUh=Y4Qqn(SIZ)i@zu7HW`^H($A&oX(xmM3=LXnn~@s zaz&mT<@2;qmWbkB?n^y7gF~%J;~&|{7bz4)G=Zm~<(sR!$VEQmLN3tPFXCJ>Hw5v> zhiz9^i(JIF*9eQn-LZF{#c#I**3r})d$SR`yHiL}6yhu1yyj-U)0bZu@2V`&hJK=~ ztu`JfJlfC+$ZuM9WO{v>n8(7T;VXXS4fK!30Zs@zGUta@Dpx=>%_z|6XR5?w>jh3J zP?S|DgFHIjn0WU0Q?>u6Q)$yZ-|^CeNIyI{nL2cIW~NLfncJsQ=o;q*B_jh;^8w>o zpK@WOi*xf@7jQ*s1hcdYM=M&52wE2Tv_G5{$|)zNYA3I(Vy_H1U9%H^_tzyWIZy6x zpDlPN;dMRPN4fYY(>u{Anx9)Ht#QgQCD|f(N)z5?_>#~rD=K+L>?WPUT6Fs34%^R^ zv{T{Djh@{}9#0YSUR-kpfU5cAL8B%*YOwd?_{*iNSS6_Onf0MG0qG;U!tRGy!DqANNUxt;G`18Ox z<7A;g{*czKsVC6~jxyPnX!bLJz4GaK*zvG{jVmv}l-LuUO_Y5X{I^>!bu?3dEmZLX zP08_X=w76qS4$6XXF^dF@ja_zHD;?_FX;*^ms^WiuO?9JP0Do_J+7et>XH6c9kogr z2AquiYFqprT*EsGc09@q+tm;ze*PYxCkLplD@hLY+LA@`oVYylO2M(>i{zBBX>eC6 zzM9;getC%X9Bm~dpqr&(7r)YbNMU#9yRy~lvmMLnH3+^4vGY7U*|2}nPlD3l^p{cP ztyH|@g}lC`J3A>0K{M-Ht!pi)b;I`_t|6>ncNN+S-ZO7@0;Ll-T^hOjTylXqW*1t9 zvVg=#p*X@#0?b{p6E>u$M%MT68R7}f%PIZ3+jYTXO$w)D5?HeBB60bs+Ah#R-@2q4 zYZs&P6V1XPU3XbvRNUnKW(|sgJ?hpd$&Kz8rLc0@*A6=3OKpT4`t?WJy3)cp$UlC# z{u^S`KS!T`d*lC{Z7N%?w0T!$xcGV{@~l3W~bU$3w2T`Q`HhYFcc9F?~)V z$7mz8N;#ceyr)rL#PXJU1-23V=rpqM`vG?*LFzBOKi$HxYpA!Rydliy@_ioSOO!sI z`0NeY4Ze}n8H9zi^!$;)#QWC!$9|D^4kM&JNhh`pYnsO~F7`63tvjbGmlN#_h-Y&> zRy!tXJ2ovUbsTkUETU;XS}MD8P^kC$%uROGM*nB*a=h&j@!cnFe52gfKOyVoVFIiWTm<9O_ez$>U)AHil@SsGYg#{6&L20*SFhFCsWsYr@OIXfYX!>3fftCm z@8zJwE}!1K!c{YK8>C0UWP16inwhB|ynmqTRF5R1Bq<{2IB2A6^W8IjC{{YYo8}&& z1&Jo=FvjAQA7lfm8oTGb1Q>IOp;F{fdce<&T7lX;*R1;$e@ewoEiU;Svq0vQ`vlP1S!sO^M4w zfyKi$J0`P!_{CKEgX6B+0L%CE9(}Ry<@B*^ra1-ZHWS_s(Bwxp-jlFiOWg(BrB(4k z0dyJ9O0=p4g0k@(r#yzhs}XUD++xuD+U@9GG&ZeKQGn}YJ{zve?aE|9iP-y4l%vE$ z2ud^p4T*f{E0D?DdU_Lp?LT*QF#gFCA^*|X+~lhPVH7xU7L1obCKMW=(4y@gt5hj= z`o(bM-ahZf;Bh=CGd_A3QH_AFGYA*0`bfz|n4XZlt6w*2xTC01!_2R6KYFreRc0l+ z8JCg@OpCUR8W^qPl(LLAWDk3fbuN8Rc1j+nvWO+}-OF+T-m1sOHY_=nP?Y1-Vj}+& z=_q&)mfrmbHp+lw;!&ygopwHyX#(=-F}Mu&R#|L+h2eq`n^g-qq?RBVT+)<3~&kWT|KT?%v<^C zNL0cPpH(*SYY0%rEcg;n9SO6aLgr?s=gMR+?_4;kM ze$lap>vM&CA>KnN6csuZGE!rHBnZPtKYHpqeT{ojG1N)0hT|{zKy67Cwrnlhgh|z} zpy1tBsz+g!CyFNLdT-wb80cnc&YUVswKYxRpbAA&HLFhNczKe`Q*2>kKyvZy((Sxs zc6qbpf|G*(0XI@Ri6Kha2&j7>su&?n6RfgTGv(tl)WwVF&GIp|jMpCx#Qbsr6Hw-4 zUjbC)6?KpHAJbK1ci5l z`b2W`#TAIBv`BD%4YHO-fTVlI4?@wkw(6|)vaD4Hm;d|O|Gh5%)iZ%}QQp7RrcUx( z+4UQSqfhU)%n172nkyP{$wO_iyaEqh2{~_K1Yl_2vmdY2OJs2($6p$6@JsVFy?`RD zLhC4Ag}tdn2LT9()|{28N;7f2#Zbl|y0?iPIQHSs>a5XV-l41aPr@>t8a{;e;ubZjH9 z?k9DPI+4g220J-&Tq^aGfj)`%2A%InzYu>c(5v^~-y!T$=a0JVa~OGBIh3|apuAR; z?i!y7*d;V@C_%?K^HNejvM_y*MVHS&1mMGpbBlxsBiE#OYv=U_+-}^oJ@q|b zh|##>Ur~vz;rfL^KvDK=AAS+>X+R-gsSRkjtdC1&9v3`T{6wc>`oWJb>ny5BhgL)G=$@iE zh8REi3sUE{xrk!0G5{l#Gn^u3Sd=WM8 z9^9q>QuL|CILwe9GVeLyJf#XgB@yl_3;)bhTlirOH~IUMi2awiz_G?}Sr4aCE;?=! z>XIdMxhF0Q-8S_itFB{n?sE1&bnD4wM!5`-p4d)bKXWn+6@II;c2*G;*l@$yYZ!a1a)OqKYoS2J zYp;u_;e<*5MEXuK*;I{B;Z9x(y)p3+Ct|LEs`b{Y&HJ~3bem$tuzs(*uO9LoWG*wa zpPtPcq%2pjf1XQCG>RJVzAUyYrU-3BSh>jf>XzKK%<+7t(@O;LJy@%+dHXWwQNeGz zER{zKM9%UoP-x6`XF%VDb8v2nt}bfBY!so8zHj(RH1j2tI*xLtyOA_Y7wuvbl4EY= zW0j@n>%T)iLJrB*(C6nPN*>WvJ6ozVS1U|HWgz)jN2LAld*4?dK1b(hAFU=xHieBqhWDO0lJfrUj62PV(D8Oyj_&;EHvHE6b2ri@WqozbYjat2gbR3U#2b2JapBKe09P6sj z9phmew>s{3k3IlVhq{G2nM|=zmvMfeXiogr(8sl=J>Yf0-~P(imxy@s61U4tB08*1Rt;Iq9=}>v2Abf>b<%rb)Co~4HaecNxBtlQuRI|n z>~UrTHzA7;NLioaYCL2BYA<1~$&ukA^mIKeJ8(Bk9+owf#yja}Svo%`O5u_-1F4uX z)-unz$-jY2mw}Rbo6S?{Sv~u+O$gxtvn#bpq5N#GX$z&+Tw!E8M5<}oJ3l4Q*ago) z$Id)ZPE)ro`~wfxhi$z)WU#~HtFMKJZNE1{{lzz7Bsqttod+n`yMXr}N;*u|dMEX+7#0BgZ51LJOQXt$_%WUz$MNvwIF9@K5Ml40 z?Fu(NK)Hn8;4E^tGzgXx6^ehi?9Flw*vPHj_I{o$2ipgWtAAUF{Xnr2M{q%~JjaZP ze%nM{m4Mgk+91x@^IEmfTL_t~K@Nk@NL;Juue;!;aWc0=y-m2a6l%BN_;P|G|}T9@9d{Lf=rn4 zYX^pw*2|ye`GwxwapZ&SCewi&jXcgu^1ZIPs6BsDS)Z4$UYwy!vPeF2;e^)h2`ur{QAi}*t;ReLk}{VT`jC_503RE)Yd6CW2MtHGI&RF5xNe`=^M*ya&hR> zVd92~?6~R9fK`Fv*&I)lPreIh;pRQAYu{{VA{##eue>}HnPdN0A066#n|)cjcQ#dD z(h5qCFsT}ZbeikSrkKU=RAp0*jlU8aRq>d9A{nGJ#9UvZIcY)MR1;yjtGq4Tup%0s zUoNhHmT^||l$G)2)WvY5tnstxUCurVWkvbK{NI0ic8)`w@h?P*o zJt!n3VbhUDDJ+Au4<2Um8_dz9s!nJk+I{-9&jhD&a*BRpT_|8d{Gh3Gnx7O&^k zPoboew>21X{`xWN6B5?>nxMs*A|Za*`RIW8N6~mbsN~AFbK~#&8BF z9wN>mJC1`&_EsFhHmaYZ&QYJt2Qk{#^1IKvlQnlDmg+(hxAU*kVDn9SF$-5tZ&fOz zXJ#WlnGWmw;EZ9Y!?y885A=F9Z)6u-%ge-r4V|V^A)xmYi(*^u3L1igTFP?w+p6|{ zmiRX9d_-o6G|z1;t$i$e&{ro!7#mI=Yb1l|sQ%$R+W zeWOd-n%8TZ@8?j(^TUO6!?U(n3y}05b~bK2}Eg>i08GgR73Sp=1Aqg{A-392xHMbwHZ3e6f-K?nCzMKilD7hvxn_ zTmEa*u78}V2Y;hb&HB;hW7>eJGQ{t_x zZ-SQnUrz1t0`3)N#qDkgCTPb6k@r7MoJ(vE^{H8!ZMFe;)O9v1LT{Q;DQu>hbPC;Z z-&Cx_CPJ~95rp~?QsH{zy4ha}PgE}FB|BV8sD5m;$IdZom~?2atwB3;h8PfKfvro;}t+0JImPcJq)}1@uP8t=?`2#}|6_-rF{!Z`vhov)mjcHW+!A8MziI znTsZ!0pFR|+fC?nZOIR*!ZslJ{7$}01$A@~6rYSoe!o4lHJDQdo9G=KZWnKO<+KcJ zTy&p+hwD+ogN@O9*s1^m{A+xMZoRHJf>i8vFO`(tAzKXSahl~t5OGwKq z|Hk)f^GefAC=PS4nRlM&Cc@O}v0X5P%KiB^y*+?SSq5EqCZPj(*n8mS834NPdgCn& z;I7w0j{rzekmAOOf zO_`hyBs`NZ&YZrUp&yK`<qCMT%8TG1E`X3a0&DRi8ALX1ZH%k){TdLdXk|9K;+U_hslUq+^4%GEUS@ zW(Vc=K3WmKKKf`RkRzYPn!`v^SPa-=aua5P$WwI^Oigvd#)>j0;-n7Vh)e#HWQ zv=MjKsJ<3@{~ivVk*o!&5S*SaTI!bzW?9aL3DZ8UozDZ-N&&T4DuU!bdmcf+twz*a_&~sW^JUU6O{ql4O!@9erzLu> zufr3EJx{(fa2ZKZfxcd5fGX-08$eU$k|Nww8EB=iXZ(zg6*r5MNa7qF1F+d8Q@h8t z^d$P_@Y)=w8py!&ZSyStM;JOJpjA>gp#olqx6|`@h&v5qH}WPo!(xi zau&39Ukif?VDG@8=JqHy)2V108uzyBth75NJ$}sC-rlv2sK8DfDr= zd*4L8zBcCkW->T@Bl+Y0n6rqzoUXeV{0TNI0NjLT65x4YjqgIzFcR(2D5EP=Lac0+V+3BD zC;)=lH^e`fPymSor6>}wD0;Y8_7On{t}W)Ih@1%tHpZ@>+8V|5w(mZF0gaZ&sBV$_ zjj=4Ko$Zt4Oyvj;Dn^jK&lI7|RpN0vpB^atC)ffSGx(naFZ^Eklto;c;@nv#dz9kG zZ`CgxeX1%zvPmSaw{?OKd5$hw-yCW54^5RH2c5nv_JY&ts>~SUiS$|`-cQc*cBU;^ zE6Y8$V|lBhg#``3pqL^suWJg|MG_Wcv05|7<4!BKay-xV3iA)*Zw^`E9?JCT$ORE~ z3t;Fx0y(jcww_CbgX3Ue=RgL5Id+a<_s*}8S6uTa3a4n(i}ga?Fj>_lmp2^2JDVOR zp+I8#>;8?J8cK+9Vi6{D#BOuxKuse=S%>kIrPjw$$w=xm%zn-nklH$`#x+%MXqll}CEM=kk}7Dt*p!+L7FQaS-uad*-u138?K0z zOkoRp8x4EUcbs`@IeX5(SwBrWm{wYr?JDzryRGa;d-iF|Q^MTQtOo6%!vAn`tN&%Z z#?;Y__9Mh{Xj-s3$AO zi{p-b{CU1E*S55|XVYK3lszY{E2Gv&KfsqG^)`naeDB8LB9voL7i8&BAPuERkc8JhQ zb4UV3bHF$Ucv`<9hC6qjB4~$w4*>Nhzao!SYBiQLsIvCtt$kx?!xsf$tw(;X2Uvbj zOO9ACkZ34q+s1{b>X}YtuNYT!Drp(lpmb}Skid~s` zg8dDdl*ce-9D$E@Z$V>B+ZH%}eRjv+^Kt!vYe)}_wTLYD$iFtY?|sgY$q4W1+|s>P zBIkCy7dq5iFOid(N{#crkm$@NdCj4QMpvS{@G+$Os=r_G0T`rFzTLe^g*N1Zlowh( zO9Q>WojMA>2b2Yx^3*=uDZyPT2wq$@;KT^Rn-M-5=jjFW)P}&30c0PtrG;_-U#e06 zgUU&N$Zh{BF#Vseq4;aYQZJ4z*o__B{d{|zi1Bm rsB*`ZX|POtntl6^IPRYelYdX+Z Date: Tue, 28 Jul 2026 10:59:24 +0300 Subject: [PATCH 2/2] docs: address PR review gaps for Inferentia and ARO diffs - Add slim ROSA OVN/Neuron troubleshooting, enrich ARO vs ROSA key differences with current vLLM/parser pointers, and link them from docs/models-and-routing.md. --- PCA_Deployment_ARO/README.md | 2 ++ PCA_Deployment_ROSA/README.md | 15 +++++++++++++++ docs/models-and-routing.md | 2 ++ 3 files changed, 19 insertions(+) diff --git a/PCA_Deployment_ARO/README.md b/PCA_Deployment_ARO/README.md index 8d87dcd..bdce7ae 100644 --- a/PCA_Deployment_ARO/README.md +++ b/PCA_Deployment_ARO/README.md @@ -666,6 +666,8 @@ Full results: [`testresults_h100.md`](testresults_h100.md) · A100 sweep: [`test | Charts | `charts/` + `values-rosa.yaml` | Same `charts/` + `values-aro.yaml` | | Cloud values | Terraform sets `gitops.cloud=rosa` | Terraform sets `gitops.cloud=aro` | | NSG / network | AWS security groups | ARO-managed subnets (no customer NSG on master/worker) | +| vLLM image | Chart / RHOAI default path for the ROSA overlay | Upstream vLLM (see [Why Upstream vLLM v0.19.0](#why-upstream-vllm-v0190)) | +| Tool / reasoning parsers | Set in ROSA `values-rosa.yaml` / ServingRuntime for the chosen model | See [Model Tool Calling & Reasoning Configuration](#model-tool-calling--reasoning-configuration) (e.g. `qwen3_xml` + `qwen3` for Qwen3.6) | Azure GPU choice depends on quota and model size (A100 80 GB vs H100 NVL 94 GB). Prefer a SKU with native FP8 and enough VRAM for your context window; see [docs/benchmarks.md](../docs/benchmarks.md). diff --git a/PCA_Deployment_ROSA/README.md b/PCA_Deployment_ROSA/README.md index d7c0e46..04af973 100644 --- a/PCA_Deployment_ROSA/README.md +++ b/PCA_Deployment_ROSA/README.md @@ -754,6 +754,8 @@ Edit `charts/pca-ai-serving/values-rosa.yaml` → update `llmInferenceService.re terraform apply -var="inferentia_pool_enabled=true" ``` +The chart ships a Neuron ServingRuntime template at `charts/pca-ai-serving/templates/vllm-neuron-runtime-template.yaml`. If Inferentia pods fail to start networking on ROSA HCP, see [Inferentia / Neuron pods fail networking (OVN annotation race)](#inferentia--neuron-pods-fail-networking-ovn-annotation-race). + ### Adding a new developer user 1. Add to `devspaces_users` in `terraform.tfvars` and run `terraform apply` (creates the HTPasswd user) @@ -829,6 +831,19 @@ oc get node -o json | jq '.metadata.labels | with_entries(select(.key oc label node feature.node.kubernetes.io/pci-10de.present=true ``` +### Inferentia / Neuron pods fail networking (OVN annotation race) + +On ROSA HCP with OVN-Kubernetes, multi-core Inferentia pods can fail to get a network if the AWS `neuron-scheduler` overwrites the `k8s.ovn.org/pod-networks` annotation during device allocation. + +**Symptoms:** Pod stays Pending or never becomes Ready; events or CNI logs show missing / invalid OVN pod-network annotation after Neuron scheduling. + +**Mitigations:** + +1. **MutatingAdmissionWebhook (verified workaround)** — Admit Neuron-scheduled pods in a way that **preserves** the OVN `k8s.ovn.org/pod-networks` annotation when the scheduler updates device annotations. +2. **Neuron DRA (preferred long-term)** — Use the Neuron Dynamic Resource Allocation path so the classic `neuron-scheduler` annotation race is avoided when DRA is available in your Neuron / OpenShift stack. + +Neuron ServingRuntime template: `charts/pca-ai-serving/templates/vllm-neuron-runtime-template.yaml`. Prefer NVIDIA GPUs for MoE code models unless you specifically need Inferentia overflow capacity. + ### Model pod stuck in Pending ```bash diff --git a/docs/models-and-routing.md b/docs/models-and-routing.md index 60b2718..b00f52a 100644 --- a/docs/models-and-routing.md +++ b/docs/models-and-routing.md @@ -33,6 +33,8 @@ Shared system prompts and similar file context benefit from prefix-cache-aware r For MoE code models (e.g. Qwen3-Coder / Qwen3.6 MoE), prefer NVIDIA with FP8. Dense models may be more cost-competitive on Inferentia when prefix caching is available. +Optional ROSA Inferentia pool: enable with Terraform `inferentia_pool_enabled=true`. If Neuron pods fail networking on OVN-Kubernetes, see [Inferentia / Neuron pods fail networking (OVN annotation race)](../PCA_Deployment_ROSA/README.md#inferentia--neuron-pods-fail-networking-ovn-annotation-race). + ## RHCL front door IDE traffic should use **`pca-ai-gateway`**, not raw vLLM. Auth is API-key based per Dev Spaces namespace. The llm-d gateway is left unmanaged by OpenShift AI AuthPolicies that would conflict with RHCL (`opendatahub.io/managed=false` where applicable).