diff --git a/docs/architecture/compress-ml-models-for-inference.md b/docs/architecture/compress-ml-models-for-inference.md deleted file mode 100644 index ec41ec9fa..000000000 --- a/docs/architecture/compress-ml-models-for-inference.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -version: 1.0 -submitted_by: navveenb -published_date: 2022-11-10 -category: ai -description: Large-scale AI/ML models require significant storage space and take more resources to run as compared to optimized models. -tags: - - ai - - machine-learning - - size:small - - persona:ai-ml-engineer - - persona:data-engineer ---- - -# Optimize the size of AI/ML models - -## Description - -Large-scale AI/ML models require significant storage space and take more resources to run as compared to optimized models. - - -## Solution -Optimizing the size of the AI/ML model can save on storage space and take up less memory. Apply strategies like quantization and evaluate the optimization changes against the desired accuracy. - - -## SCI Impact -`SCI = (E * I) + M per R` - -[Software Carbon Intensity Spec](https://grnsft.org/sci) - -Optimizing the AI/ML model impacts SCI as follows: -- `E`: Having an optimized AI model would reduce the energy consumption for your AI/ML inference, save storage space and network bandwidth and consequently, the E number should decrease. - -## Assumptions -None - -## Considerations -None - -## References -[Model optimization](https://www.tensorflow.org/lite/performance/model_optimization) diff --git a/docs/architecture/system-topology/efficient-hardware-ai-workloads.md b/docs/architecture/system-topology/efficient-hardware-ai-workloads.md new file mode 100644 index 000000000..2e1673120 --- /dev/null +++ b/docs/architecture/system-topology/efficient-hardware-ai-workloads.md @@ -0,0 +1,75 @@ +--- +version: 1.0 +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Architecture +tags: + - ai + - machine-learning + - compute + - cloud + - persona:infrastructure-engineer + - persona:devops-engineer + - persona:ai-ml-engineer + - persona:solution-architect + - size:medium +personas: Infrastructure Engineer, DevOps Engineer, AI/ML Engineer, Solution Architect +description: Match AI workloads to the most energy-efficient hardware accelerator or instance type to improve utilisation and reduce energy consumption per inference or training run. +--- + +# Select efficient accelerators and instance types for AI workloads + +**Applicable Role:** Provider and Consumer + +## Description + +AI workloads such as training, fine-tuning, and inference require significant compute resources. The type of hardware used, including CPUs, GPUs, TPUs, and specialized accelerators, has a direct impact on energy efficiency and performance. + +Different hardware options vary in their ability to execute AI workloads efficiently. Selecting appropriate hardware and compute resources, combined with intelligent workload orchestration across heterogeneous platforms, improves utilization, reduces execution time, and lowers overall energy consumption.. + +## Solution + +- Profile workloads based on latency, throughput, and execution characteristics before selecting compute resources +- Choose hardware optimized for the specific workload, such as CPUs, GPUs, TPUs, NPUs, or other specialized accelerators +- Maintain a catalogue of supported accelerator types and their suitability for different workload classes +- Use orchestration and scheduling systems to automatically dispatch workloads to the most energy-efficient available compute platform +- Implement closed-loop monitoring and resource allocation to continuously optimize workload placement and avoid over-provisioning +- Monitor utilization and dynamically adjust resource allocation to improve efficiency +- Evaluate performance-per-watt benchmarks and runtime telemetry when selecting hardware and instance types + +## SCI Impact + +**SCI = (E × I) + M per R** + +**E (Energy):** Efficient hardware reduces compute time and energy consumption for AI workloads. + +**M (Embodied Carbon):** Better utilization can reduce the number of required machines and associated embodied emissions. + +## Cost Impact + +- **Hardware costs:** Instance type choice affects hourly compute rates significantly +- **Utilization efficiency:** Better hardware-workload fit reduces per-inference cost +- **Reserved instance savings:** Efficient hardware selection enables better RI negotiation +- **Power and cooling costs:** Specialized accelerators may have lower operational energy costs +- **Trade-off:** Premium or specialized accelerators (GPUs, TPUs, NPUs, neuromorphic processors, or in-memory computing systems) may cost more upfront but can deliver significantly lower cost-per-inference and energy consumption for suitable workloads + +## Assumptions + +- Suitable hardware options are available for the workload +- Performance benchmarks reflect real-world usage + +## Considerations + +- Specialized hardware may not be available in all regions +- Costs may vary across hardware options +- Migration to different hardware may require changes in software or frameworks +- Underutilized hardware can negate efficiency gains + +## References + +- [MLPerf Benchmarks — ML Hardware Performance](https://mlcommons.org/benchmarks/inference/) +- [Google Cloud TPU — Purpose-built AI Accelerator](https://cloud.google.com/tpu) +- [NVIDIA GPU Benchmarks for AI](https://developer.nvidia.com/deep-learning-performance-training-inference) +- [Spec Power Benchmark — Server Energy Efficiency](https://www.spec.org/power_ssj2008/) +- [NeuroBench — Benchmarking Neuromorphic Algorithms and Systems](https://neurobench.ai/) diff --git a/docs/architecture/system-topology/energy-efficent-ai-edge.md b/docs/architecture/system-topology/energy-efficent-ai-edge.md deleted file mode 100644 index 433199cc1..000000000 --- a/docs/architecture/system-topology/energy-efficent-ai-edge.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -version: 1.0 -submitted_by: navveenb -published_date: 2022-11-10 -category: ai -description: Data computation for ML workloads and ML inference is a significant contributor to the carbon footprint of the ML application. Also, if the ML model is running on the cloud, the data needs to be transferred and processed on the cloud to the required format that can be used by the ML model for inference. -tags: - - ai - - machine-learning - - size:small - - persona:devops-engineer - - persona:solution-architect - - persona:ai-ml-engineer ---- - -# Run AI models at the edge - -## Description -Data computation for ML workloads and ML inference is a significant contributor to the carbon footprint of the ML application. Also, if the ML model is running on the cloud, the data needs to be transferred and processed on the cloud to the required format that can be used by the ML model for inference. - - - -## Solution -Evaluate and run AI models at the edge, based on your application requirements. Also running data and compute processing tasks (i.e. data cleansing, feature generation) directly on the edge resources, ensure better utilization, and low latency and limit the transfer of data over the network to the cloud. - - -## SCI Impact -`SCI = (E * I) + M per R` - -[Software Carbon Intensity Spec](https://grnsft.org/sci) - -Running energy efficient AI at the edge would impact SCI as follows: -- `E`: An energy efficient AI at the edge would reduce energy consumption by providing local computing and storage for data. Running the inference at the edge in this way would reduce the network transfer to the cloud, reducing the overall energy consumed. - -## Assumptions -None - -## Considerations -Consider the operational and embodied emissions of the edge devices as part of your overall solution and how it can reduce the carbon impact of your overall application. - -## References -- [Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of Things](https://ieeexplore.ieee.org/document/9520303) diff --git a/docs/architecture/system-topology/on-demand-execution-ai-agent-workloads.md b/docs/architecture/system-topology/on-demand-execution-ai-agent-workloads.md new file mode 100644 index 000000000..474f0cfe1 --- /dev/null +++ b/docs/architecture/system-topology/on-demand-execution-ai-agent-workloads.md @@ -0,0 +1,83 @@ +--- +version: 1.0 +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Architecture +tags: + - ai + - machine-learning + - compute + - serverless + - cloud + - persona:devops-engineer + - persona:software-engineer + - persona:ai-ml-engineer + - size:medium +personas: DevOps Engineer, Software Engineer, AI/ML Engineer +description: Trigger AI and agent workloads only when needed using serverless or event-driven platforms to eliminate idle compute and reduce unnecessary energy consumption. +--- + +# Use on-demand execution for AI and agent workloads + +**Applicable Role:** Consumer + +## Description + +AI systems increasingly operate as dynamic, multi-step workflows, especially in agentic architectures where models interact with tools, data sources, and other models. + +Keeping compute resources or agent workflows active when not required leads to unnecessary energy consumption. This includes idle infrastructure, continuously running agents, and long-lived orchestration processes. + +Using on-demand execution ensures that compute and workflows are triggered only when needed, reducing idle time and improving overall efficiency. + +## Solution + +- Use serverless or event-driven platforms to execute workloads only when triggered +- Design agent workflows to run only when required and terminate after task completion +- Avoid long-running or always-on agent processes unless continuously needed +- Trigger model calls and tool usage conditionally rather than continuously +- Use orchestration frameworks that support event-driven execution, workload dispatching, and efficient workflow management +- Prioritize the use of existing available resources through orchestration before provisioning additional capacity +- Dynamically scale resources only when workload demand exceeds available capacity +- Monitor workload execution and adapt resource allocation continuously to optimize utilization and efficiency + +## SCI Impact + +**SCI = (E × I) + M per R** + +**E (Energy):** Reduces energy consumption by eliminating idle compute and unnecessary agent execution. + +**I (Carbon Intensity):** On-demand execution can be combined with carbon-aware scheduling (Pattern 5) to trigger workloads during low-carbon periods. + +**M (Embodied Carbon):** Improved utilization of shared infrastructure reduces overall hardware demand. + +**R (Functional Unit):** For event-driven systems, the functional unit may be expressed as events processed, workflows completed, or agent tasks executed. Event payload size, throughput, and latency requirements can significantly influence overall efficiency. + +## Cost Impact + +- **Compute costs:** Reduced by eliminating idle infrastructure and always-on processes +- **Cold start overhead:** Serverless platforms may incur higher per-invocation costs than reserved instances +- **Provisioned concurrency:** Can mitigate cold starts but adds baseline cost +- **State management:** Event-driven architectures often require explicit state persistence, recovery, logging, and lifecycle management. Additional storage, messaging, and orchestration components may increase operational complexity and cost. +- **Trade-off:** Per-invocation serverless pricing vs. reserved instance baseline; evaluate break-even point + +## Assumptions + +- Workloads and agent workflows can be structured as event-driven processes with appropriate state management, lifecycle instrumentation, and mechanisms for efficiently processing asynchronous event payloads +- Execution environments support dynamic scaling and orchestration, and workloads can be safely interrupted and resumed without losing state or requiring expensive recomputation + +## Considerations + +- Cold start latency may impact responsiveness +- Complex workflows may require careful orchestration design +- Not all workloads are suitable for on-demand execution +- Inefficient agent design can still lead to excessive compute even in serverless environments; orchestration and workload scheduling can improve placement efficiency but cannot fully compensate for poorly designed agents +- Trade-offs between responsiveness, cost, and carbon should be evaluated +- Synchronous and asynchronous event-processing models present different trade-offs in latency, scalability, and energy efficiency + +## References + +- [AWS Lambda for ML Inference](https://aws.amazon.com/lambda/) +- [Google Cloud Functions for Serverless AI](https://cloud.google.com/functions) +- [Azure Functions — Event-driven Serverless Compute](https://azure.microsoft.com/en-us/products/functions) +- [Knative — Kubernetes-based Serverless](https://knative.dev/) diff --git a/docs/architecture/system-topology/run-ai-models-edge.md b/docs/architecture/system-topology/run-ai-models-edge.md new file mode 100644 index 000000000..353cd6fc8 --- /dev/null +++ b/docs/architecture/system-topology/run-ai-models-edge.md @@ -0,0 +1,80 @@ +--- +version: 1.0 +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Architecture +tags: + - ai + - machine-learning + - compute + - networking + - deployment + - persona:infrastructure-engineer + - persona:solution-architect + - persona:ai-ml-engineer + - size:large +personas: Infrastructure Engineer, Solution Architect, AI/ML Engineer +description: Deploy AI inference on edge devices or local infrastructure to reduce data transfer, network energy use, and reliance on centralised cloud compute. +--- + +# Run AI models at the edge + +**Applicable Role:** Provider and Consumer + +## Description + +AI and ML workloads often rely on centralized cloud infrastructure for training and inference. This requires data to be transferred from source systems to the cloud, increasing network usage, latency, and energy consumption. + +Running AI models closer to where data is generated or consumed, such as on edge devices or local infrastructure, reduces data movement and enables more efficient processing. This is especially relevant for real-time, high-frequency, or latency-sensitive inference workloads. + +Providers also deploy edge inference capabilities through on-device ML SDKs and embedded models, making this pattern applicable to both roles. + +## Solution + +- Classify workloads and edge devices based on latency requirements, data volume, compute intensity, memory constraints, and power availability to determine suitable deployment targets +- Evaluate hybrid architectures that combine edge and cloud processing based on workload requirements +- Deploy models on edge devices or local infrastructure when doing so reduces data transfer, latency, or centralized compute requirements +- Perform workload-specific data preprocessing tasks such as filtering, cleansing, aggregation, and feature generation locally +- Use edge inference for real-time, high-frequency, or latency-sensitive applications +- Limit transmission of raw data by sending only necessary, filtered, or aggregated results to the cloud +- For applications using external AI services, consider on-device or local inference options to reduce repeated remote calls + +## SCI Impact + +**SCI = (E × I) + M per R** + +**E (Energy):** Reduced data transfer and localized processing lower energy consumption associated with network and centralized compute. + +**I (Carbon Intensity):** Edge devices run on local power grids which may have different carbon intensity than centralized data center regions. This should be measured and factored into the SCI calculation. + +**M (Embodied Carbon):** Edge deployments may increase device footprint, but can reduce reliance on large centralized infrastructure. + +## Cost Impact + +- **Cloud compute costs:** Reduced by moving inference to edge devices +- **Network costs:** Lower data transfer to centralized systems +- **Edge device costs:** Hardware choices range from conventional CPUs and GPUs to specialized low-power accelerators and emerging architectures, with different cost, performance, and energy-efficiency trade-offs +- **Model management costs:** Higher due to complexity of distributed model updates +- **Trade-off:** Cloud cost savings offset by edge device and management overhead + +## Assumptions + +- Edge or local devices have sufficient memory, compute capacity, and power to support both workload-specific preprocessing and model execution, potentially using optimized, compressed, or quantized models +- Workloads can be partitioned effectively between edge and cloud + +## Considerations + +- Embodied emissions of edge devices must be accounted for +- Edge environments may have limited compute, storage, connectivity, availability, and responsiveness +- Model updates and lifecycle management can be more complex in distributed systems +- Workload partitioning between edge and cloud should consider latency, energy consumption, network usage, device capabilities, operational constraints, and carbon intensity differences +- Not all workloads are suitable for edge deployment; trade-offs between latency, cost, carbon, and operational complexity should be evaluated + +## References + +- [TensorFlow Lite — On-device ML Framework](https://www.tensorflow.org/lite) +- [ONNX Runtime — Cross-platform Inference Engine](https://onnxruntime.ai/) +- [NVIDIA Jetson Platform — Edge AI Computing](https://developer.nvidia.com/embedded-computing) +- [Green AI for IIoT: Energy Efficient Intelligent Edge Computing](https://arxiv.org/abs/2205.02343) +- [SCI-AI Specification — Green Software Foundation](https://github.com/Green-Software-Foundation/sci-ai/blob/dev/SPEC.md) diff --git a/docs/architecture/technology-selection/efficent-format-for-model-training.md b/docs/architecture/technology-selection/efficent-format-for-model-training.md deleted file mode 100644 index 836010496..000000000 --- a/docs/architecture/technology-selection/efficent-format-for-model-training.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -version: 1.0 -submitted_by: navveenb -published_date: 2022-11-10 -category: ai -description: Efficient storage of the model becomes extremely important to manage the data used for ML model development. -tags: - - ai - - machine-learning - - size:small - - persona:ai-ml-engineer - - persona:data-engineer ---- - -# Use efficient file formats for AI/ML development - -## Description -Data processing and storage constitute a significant portion of AI/ML development and impact the carbon footprint of your application. Variety and volumes of data might need to be captured and pre-processed for building the ML model. Efficient storage of the model becomes extremely important to manage the data used for ML model development. - - -## Solution -Use efficient file formats for building your ML models. For instance, column-oriented data file formats like Parquet provide efficient data storage and retrieval as compared to formats like CSV. - - -## SCI Impact -`SCI = (E * I) + M per R` - -[Software Carbon Intensity Spec](https://grnsft.org/sci) - -Using efficient file formats for ML development impacts SCI as follows: -- `E`: A more efficient file format for ML development means more efficient data storage and retrieval, resulting in lower overall energy consumption. -- `M`: A more efficient file format for ML development reduces the amount of storage space and number of servers needed, resulting in a lower overall embodied carbon. - -## Assumptions -None - -## Considerations -Evaluate and consider the most energy efficient formats required for your application. - -## References -[Apache Parquet](https://parquet.apache.org/) diff --git a/docs/architecture/technology-selection/right-hardware-type.md b/docs/architecture/technology-selection/right-hardware-type.md deleted file mode 100644 index e61392084..000000000 --- a/docs/architecture/technology-selection/right-hardware-type.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -version: 1.0 -submitted_by: navveenb -published_date: 2022-11-10 -category: ai -description: Selecting the right hardware/VM instance types for training is one of the choices you should make as part of your energy-efficient AI/ML process. -tags: - - ai - - machine-learning - - size:small - - persona:ai-ml-engineer - - persona:devops-engineer - - persona:infrastructure-engineer ---- - -# Select the right hardware/VM instance types for AI/ML training - -## Description -Training an AI model has a significant carbon footprint. Selecting the right hardware/VM instance types for training is one of the choices you should make as part of your energy-efficient AI/ML process. For instance, custom application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs) are provided or supported by cloud vendors which provide better energy efficiency and inference for AI models than conventional chips. - - -## Solution -Evaluate and leverage the right hardware/VM instance types for training and inference of AI/ML development. - -## SCI Impact -`SCI = (E * I) + M per R` - -[Software Carbon Intensity Spec](https://grnsft.org/sci) - -Selecting the right hardware/VM types impacts SCI as follows: -- `E`: The right hardware/VM type provides better energy efficiency and inference for AI models, reducing the energy consumption of your AI/ML processes overall. -- `M`: By reducing the total number of servers required to run a process, the total embodied carbon is lower. - -## Assumptions -None - -## Considerations -None - -## References -- [Energy and Policy Considerations for Deep Learning in NLP](https://arxiv.org/pdf/1906.02243.pdf) -- [Deploy ML models to field-programmable gate arrays (FPGAs) with Azure Machine Learning](https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-deploy-fpga-web-service) -- [Quantifying the performance of the TPU, our first machine learning chip](https://cloud.google.com/blog/products/gcp/quantifying-the-performance-of-the-tpu-our-first-machine-learning-chip) diff --git a/docs/development/data-handling/optimize-data-storage-ai-training.md b/docs/development/data-handling/optimize-data-storage-ai-training.md new file mode 100644 index 000000000..89e8e908c --- /dev/null +++ b/docs/development/data-handling/optimize-data-storage-ai-training.md @@ -0,0 +1,74 @@ +--- +version: 1.0 +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Development +tags: + - ai + - machine-learning + - storage + - persona:data-engineer + - persona:ai-ml-engineer + - size:medium +personas: Data Engineer, AI/ML Engineer +description: Use efficient storage formats, compression, and indexing strategies for AI datasets and embeddings to reduce storage footprint, data transfer, and retrieval compute. +--- + +# Optimize data storage formats for AI training and inference + +**Applicable Role:** Provider and Consumer + +## Description + +Data storage and access form a significant part of AI and ML systems. During development, large datasets are collected and processed for training. During runtime, especially in retrieval-augmented generation (RAG) systems, data and embeddings are frequently accessed for retrieval and inference. + +Inefficient data storage formats and access patterns increase storage requirements, data transfer volumes, and processing overhead. This leads to higher energy consumption and infrastructure usage. + +Using efficient data storage and access patterns improves data retrieval performance and reduces the overall resource footprint of both training and runtime systems. + +## Solution + +- Use columnar storage formats such as Parquet or ORC for structured datasets +- Avoid text-based formats like CSV for large-scale workloads when more efficient alternatives are available +- Compress data where appropriate to reduce storage and transfer size +- Use efficient serialization and deserialization formats and techniques to reduce processing overhead and data transfer costs, particularly in distributed or edge-cloud environments +- Optimize data schemas to reduce redundancy and improve access efficiency +- Use storage systems that support efficient querying, indexing, and partial reads +- For retrieval systems, use optimized vector storage and indexing techniques to reduce compute during similarity search + +## SCI Impact + +**SCI = (E × I) + M per R** + +**E (Energy):** Efficient storage, retrieval, and vector search reduce compute required for data processing and runtime inference. + +**M (Embodied Carbon):** Reduced storage requirements decrease infrastructure needs and associated embodied emissions. + +## Cost Impact + +- **Storage costs:** Reduced through efficient formats (Parquet vs. CSV) and compression +- **Data transfer costs:** Lower egress charges due to smaller data sizes +- **Compute costs:** Reduced query and retrieval costs from optimized indexing +- **Tooling costs:** Vector DB licensing (Milvus, Pinecone) may add operational expense +- **Trade-off:** Storage efficiency gains offset by vector indexing infrastructure costs + +## Assumptions + +- Data storage formats and systems can be updated without breaking downstream applications +- Compression and indexing strategies do not introduce excessive processing overhead + +## Considerations + +- Compatibility with existing tools and pipelines must be evaluated, as data format conversion and migration may introduce additional compute, tooling, and operational costs +- Retrieval workloads may require both efficient storage and optimized indexing strategies +- Compression should be balanced with decompression cost +- Vector storage and indexing choices can significantly impact retrieval performance and energy usage + +## References + +- [Apache Parquet — Columnar Storage Format](https://parquet.apache.org/) +- [Apache ORC — Optimized Row Columnar Format](https://orc.apache.org/) +- [FAISS — Facebook AI Similarity Search](https://github.com/facebookresearch/faiss) +- [Milvus — Open-source Vector Database](https://milvus.io/) +- [Pinecone — Managed Vector Database](https://www.pinecone.io/) diff --git a/docs/development/leverage-sustainable-regions.md b/docs/development/leverage-sustainable-regions.md deleted file mode 100644 index 1708c6f0a..000000000 --- a/docs/development/leverage-sustainable-regions.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -version: 1.0 -submitted_by: navveenb -published_date: 2022-11-10 -category: ai -description: Depending on the model parameters and training iterations, training an AI/ML model consumes a lot of power and requires many servers which contribute to embodied emissions. -tags: - - ai - - machine-learning - - size:small - - persona:ai-ml-engineer - - persona:infrastructure-engineer ---- - -# Use sustainable regions for AI/ML training - -## Description -Training an AI model has a significant carbon footprint. Depending on the model parameters and training iterations, training an AI/ML model consumes a lot of power and requires many servers which contribute to embodied emissions. - - -## Solution -Use a cloud region which has a lower carbon intensity value for running your AI/ML training workloads. - - -## SCI Impact -`SCI = (E * I) + M per R` - -[Software Carbon Intensity Spec](https://grnsft.org/sci) - -Using a lower carbon intensity region for AI/ML training impacts SCI as follows: -- `E`: Using a lower carbon intensity region for ML training would reduce the carbon emissions of ML applications, therefore decreasing the amount of energy consumed. - -## Assumptions -The migration of workloads to other regions assumes you have taken into consideration privacy, security, or data sovereignty based on your application requirements. - -## Considerations -Consider the trade-offs between carbon footprint, cost, and latency when selecting a region. - -## References -- [Faster, cheaper, greener? Pick the Google Cloud region that’s right for you](https://cloud.google.com/blog/topics/sustainability/google-cloud-region-picker-helps-you-make-the-green-choice) -- [Amazon’s sustainability regions](https://sustainability.aboutamazon.com/around-the-globe?energyType=true) -- [Azure sustainability](https://azure.microsoft.com/en-us/explore/global-infrastructure/sustainability/) -- [Google Cloud sustainability regions](https://cloud.google.com/sustainability/region-carbon) diff --git a/docs/development/optimize-agent-orchestration-reduce-model-calls.md b/docs/development/optimize-agent-orchestration-reduce-model-calls.md new file mode 100644 index 000000000..380f8403e --- /dev/null +++ b/docs/development/optimize-agent-orchestration-reduce-model-calls.md @@ -0,0 +1,84 @@ +--- +version: 1.0 +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Development +tags: + - ai + - machine-learning + - compute + - persona:ai-ml-engineer + - persona:software-engineer + - size:medium +personas: AI/ML Engineer, Software Engineer +description: Design agentic AI workflows to minimise redundant model invocations and unnecessary compute through caching, conditional logic, and efficient orchestration patterns. +--- + +# Optimize agent orchestration to reduce unnecessary model calls + +**Applicable Role:** Consumer + +## Description + +AI systems increasingly operate as multi-step workflows and agentic architectures where models interact with tools, data sources, and other models to accomplish complex tasks. Orchestration frameworks and patterns determine how these interactions are coordinated and how efficiently the system calls models. + +Inefficient orchestration design leads to redundant model invocations, unnecessary API calls, repeated processing of identical inputs, and wasted compute. This increases energy consumption without advancing toward the desired outcome. + +Optimizing agent orchestration and workflow design minimizes unnecessary model calls, reduces computational waste, and improves the overall efficiency of AI systems. + +## Solution + +- Design agent workflows to minimize redundant model calls and repeated computations +- Use caching mechanisms to avoid re-processing identical inputs or identical tool results +- Implement conditional logic to skip unnecessary model calls when prior results can be reused +- Prefer direct tool calls or API integrations over calling models to transform simple data +- Use streaming, progressive results, and event-driven processing patterns where appropriate to reduce unnecessary computation and improve responsiveness +- Implement thought/action batching to reduce the number of model invocations per task +- Design workflows to halt agent loops when goals are achieved rather than running fixed iterations +- Monitor and profile agent execution to identify and eliminate inefficient patterns +- Use simpler models or heuristics for routing and filtering decisions before invoking larger models +- Document and test agent workflows to ensure they perform necessary steps without backtracking or rework + +## SCI Impact + +**SCI = (E × I) + M per R** + +**E (Energy):** Reducing unnecessary model calls directly decreases compute and energy consumption. Optimized workflow design eliminates wasted computation per functional unit. + +**I (Carbon Intensity):** Orchestration optimization can be combined with carbon-aware scheduling (see related pattern) to defer non-urgent agent tasks to low-carbon periods. + +**M (Embodied Carbon):** Reduced compute requirements lower overall infrastructure demand. + +## Cost Impact + +- **Compute costs:** Directly reduced by eliminating unnecessary model calls and redundant processing +- **API/model costs:** Lower per-task cost due to fewer model invocations +- **Infrastructure costs:** Reduced due to lower overall compute demand +- **Development costs:** Initial investment in profiling and optimization; ongoing monitoring required +- **Trade-off:** More efficient workflows may require more thoughtful design and testing upfront + +## Assumptions + +- Workflows can be analyzed, profiled, and calibrated using telemetry from deployed systems to identify inefficiencies +- Caching and conditional logic can be implemented without breaking workflow functionality +- Tool integrations and APIs are available as alternatives to model invocations for certain tasks + +## Considerations + +## Considerations + +- Complex multi-step workflows and different agent design patterns (for example, ReAct or Tree of Thought) may require different optimization strategies +- Caching strategies must account for data freshness and accuracy requirements +- Some tasks genuinely require multiple model calls; avoid over-optimizing at the expense of output quality or responsiveness +- Monitoring, profiling, and telemetry are required to identify inefficient execution patterns and support adaptive orchestration +- Trade-offs between latency, cost, and efficiency should be evaluated for each use case + +## References + +- [LangChain — LLM Application Framework](https://www.langchain.com/) +- [LlamaIndex — Data Framework for LLMs](https://www.llamaindex.ai/) +- [AutoGen — Multi-Agent Framework (Microsoft)](https://github.com/microsoft/autogen) +- [CrewAI — Multi-Agent Orchestration](https://www.crewai.com/) +- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629) +- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/abs/2305.10601) diff --git a/docs/development/pre-trained-transfer-learning.md b/docs/development/pre-trained-transfer-learning.md index fe996177a..10c572b21 100644 --- a/docs/development/pre-trained-transfer-learning.md +++ b/docs/development/pre-trained-transfer-learning.md @@ -1,40 +1,71 @@ --- version: 1.0 -submitted_by: navveenb -published_date: 2022-11-10 -category: ai -description: As part of your AI/ML process, you should evaluate using a pre-trained model and use transfer learning to avoid training a new model from scratch. +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Development tags: - ai - machine-learning - - size:small + - compute - persona:ai-ml-engineer - persona:data-engineer + - size:medium +personas: AI/ML Engineer, Data Engineer +description: Fine-tune existing pre-trained models instead of training from scratch to dramatically reduce the compute, energy, and time required for model development. --- -# Leverage pre-trained models and transfer learning for AI/ML development +# Leverage pre-trained models and transfer learning + +**Applicable Role:** Provider ## Description -Training an AI model has a significant carbon footprint. As part of your AI/ML process, you should evaluate using a pre-trained model and use transfer learning to avoid training a new model from scratch. +Training AI and ML models from scratch requires significant compute, data, and time, leading to high energy consumption and carbon emissions. In many cases, models can be initialized from pre-trained versions and adapted to specific tasks through fine-tuning. + +Leveraging pre-trained models avoids redundant training effort and reduces the overall resource footprint of model development. ## Solution -Evaluate and select pre-trained models and use transfer learning to avoid training a new model from scratch. + +- Select pre-trained models that are relevant to the target task +- Fine-tune models instead of training from scratch where possible +- Reuse existing model weights and representations to reduce training effort +- Evaluate whether full training is necessary before starting new model development +- Use domain-adapted or task-specific pre-trained models when available ## SCI Impact -`SCI = (E * I) + M per R` -[Software Carbon Intensity Spec](https://grnsft.org/sci) +**SCI = (E × I) + M per R** + +**E (Energy):** Avoiding full training significantly reduces compute and energy consumption. + +**M (Embodied Carbon):** Reduced infrastructure usage lowers embodied emissions associated with training. -Leveraging a pre-trained model would impact SCI as follows: -- `E`: Having a pre-trained model reduces energy consumption for your AI/ML development as you don’t need to train the entire model from scratch. -- `M`: Transfer learning does not require as many servers as you don’t need to train the entire model from scratch. By reducing the total number of servers required to run a process, the total embodied carbon is lower. +**R (Functional Unit):** For providers using per FLOP or per training token as the functional unit, transfer learning dramatically reduces the total FLOPs and tokens required, lowering total carbon (C) while R scales proportionally, resulting in a more favorable SCI score. + +## Cost Impact + +- **Training costs:** Dramatically reduced by avoiding full model training +- **Compute time:** Significantly lower for fine-tuning vs. training from scratch +- **Pre-trained model licensing:** Potential licensing costs for commercial model access +- **Data costs:** May be lower if transfer learning requires less training data +- **Trade-off:** Pre-trained model licensing may offset training cost savings ## Assumptions -None + +- Suitable pre-trained models are available for the target use case +- Fine-tuning can achieve the required performance ## Considerations -None + +- Pre-trained models may introduce biases or limitations from their original training data +- Fine-tuning large foundation models can still require substantial compute resources comparable to training from scratch; evaluate the true cost-benefit of fine-tuning vs. full training for your use case +- Licensing and usage restrictions of pre-trained models must be evaluated +- Suitable domain-specific pre-trained models may not always be available; adapting and validating models for specialized domains may require additional data, compute resources, and evaluation effort ## References -- [Transfer learning and fine-tuning](https://www.tensorflow.org/tutorials/images/transfer_learning) + +- [Hugging Face Model Hub — Pre-trained Model Repository](https://huggingface.co/models) +- [Transfer Learning in NLP (Ruder et al., 2019)](https://arxiv.org/abs/1902.10547) +- [Foundation Models — Opportunities and Risks (Bommasani et al., 2021)](https://arxiv.org/abs/2108.07258) +- [Energy and Policy Considerations for Deep Learning in NLP (Strubell et al., 2019)](https://arxiv.org/abs/1906.02243) diff --git a/docs/development/right-sized-energy-efficient-ai-models.md b/docs/development/right-sized-energy-efficient-ai-models.md new file mode 100644 index 000000000..91f0446d5 --- /dev/null +++ b/docs/development/right-sized-energy-efficient-ai-models.md @@ -0,0 +1,80 @@ +--- +version: 1.0 +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Development +tags: + - ai + - machine-learning + - compute + - persona:ai-ml-engineer + - persona:software-engineer + - persona:solution-architect + - size:medium +personas: AI/ML Engineer, Software Engineer, Solution Architect +description: Select and optimize AI models that are appropriately sized for the task to reduce compute, memory, and energy consumption during training and inference. +--- + +# Use right-sized and energy-efficient AI models + +**Applicable Role:** Provider and Consumer + +## Description + +AI and ML models vary significantly in size, architecture, complexity, and resource requirements. Larger models typically require more compute, memory, and storage, leading to higher energy consumption during both training and inference. + +Using models that are appropriately sized and architecturally efficient for the task avoids unnecessary resource usage. This includes selecting smaller or task-specific models, choosing energy-efficient architectures at equivalent capability levels, and applying optimization techniques to reduce model footprint without sacrificing required performance. + +## Solution + +- Evaluate model options based on task requirements, deployment constraints, and available model catalogs before selecting a model +- Select smaller or task-specific models where they provide sufficient performance +- Choose base models that provide the required capability with lower compute requirements +- Prefer optimized or distilled versions of larger models for fine-tuning and inference +- Apply model compression techniques such as quantization, pruning, and knowledge distillation +- Remove redundant or inactive parameters where possible +- Evaluate model options based on performance and energy efficiency using benchmarks representative of the target deployment environment +- Continuously evaluate newer model variants that offer improved efficiency +- Avoid defaulting to the largest available model when simpler alternatives can achieve similar outcomes + +## SCI Impact + +**SCI = (E × I) + M per R** + +**E (Energy):** Smaller or optimized models reduce compute requirements, memory usage, and data movement during training and inference. + +**M (Embodied Carbon):** Reduced infrastructure and storage needs lower embodied emissions over time. + +**R (Functional Unit):** When the functional unit is per inference or per token, right-sizing a model reduces the energy cost per functional unit, directly lowering the SCI score. However, if optimization reduces output quality and more functional units are needed to achieve the same outcome, the net effect on SCI should be evaluated. + +## Cost Impact + +- **Compute costs:** Reduced due to smaller model sizes and faster inference +- **Infrastructure costs:** Lower due to reduced memory and storage requirements +- **Benchmarking and evaluation:** Requires additional effort and cost but is essential for identifying the most efficient model for a given use case +- **Trade-off:** Optimization for efficiency may require initial investment in model compression tooling + +## Assumptions + +- Smaller or optimized models can meet the functional requirements of the application +- Model performance can be validated against application-specific functional and quality thresholds +- Efficiency improvements are evaluated against acceptable trade-offs in output quality, accuracy, latency, and cost + +## Considerations + +- Trade-offs between model size, accuracy, latency, and efficiency should be evaluated in the context of task requirements, deployment hardware, data characteristics, preprocessing requirements, and operational constraints +- Some complex or domain-specific tasks may still require larger models or fine-tuning +- Model suitability depends on task requirements, deployment hardware, data characteristics, preprocessing requirements, and operational constraints +- Over-optimization can degrade performance or output quality +- Monitoring and periodic re-evaluation introduce overhead and should balance observability benefits with resource consumption +- Benchmarking should include both performance and resource usage in representative deployment environments + +## References + +- [Quantization and Pruning Techniques — Hugging Face Optimum](https://huggingface.co/docs/optimum) +- [Knowledge Distillation — DistilBERT (Sanh et al., 2019)](https://arxiv.org/abs/1910.01108) +- [ML CO2 Impact — Estimate carbon emissions from ML compute](https://mlco2.github.io/impact/) +- [Green AI (Schwartz et al., 2020) — Efficiency in AI Research](https://arxiv.org/abs/1907.10597) +- [Efficient Transformers: A Survey (Tay et al., 2022)](https://arxiv.org/abs/2009.06732) +- [ISO/IEC 21031:2024 — Software Carbon Intensity (SCI) Specification](https://www.iso.org/standard/86612.html) diff --git a/docs/development/select-efficient-ml-frameworks-inference-runtimes.md b/docs/development/select-efficient-ml-frameworks-inference-runtimes.md new file mode 100644 index 000000000..8cdabf95d --- /dev/null +++ b/docs/development/select-efficient-ml-frameworks-inference-runtimes.md @@ -0,0 +1,79 @@ +--- +version: 1.0 +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Development +tags: + - ai + - machine-learning + - compute + - persona:ai-ml-engineer + - size:medium +personas: AI/ML Engineer +description: Choose ML frameworks and inference runtimes that best match your hardware and workload to reduce compute overhead and improve energy efficiency across training and production inference. +--- + +# Select efficient ML frameworks and inference runtimes + +**Applicable Role:** Provider and Consumer + +## Description + +Machine learning frameworks and inference runtimes are the core execution engines for AI and ML workloads. These tools determine how efficiently models and algorithms utilize available hardware, manage memory, and optimize compute across CPU, GPU, TPU, and specialized accelerators. + +Different frameworks and runtimes vary significantly in their ability to leverage hardware capabilities, execute operations efficiently, and minimize computational overhead. Inefficient framework choices can lead to unnecessary compute consumption, poor hardware utilization, and increased energy expenditure for the same workload. + +Selecting efficient ML frameworks and inference runtimes improves model execution performance and reduces the carbon footprint of AI training and inference. + +## Solution + +- Choose frameworks that efficiently utilize available hardware (GPUs, TPUs, specialized accelerators) +- Prefer frameworks with native support for hardware acceleration and parallel processing +- Evaluate inference runtimes (ONNX Runtime, TensorRT, OpenVINO) that are optimized for model execution +- Use optimized inference layers that reduce latency and compute overhead compared to training frameworks +- Select frameworks with strong compiler optimization and memory management capabilities +- Benchmark framework options under your actual workload conditions before committing to production +- Keep frameworks and runtime dependencies updated to benefit from performance and efficiency improvements +- Consider compilation frameworks and interoperable model formats (for example, ONNX or OpenVINO) that optimize models for specific hardware targets and enable portability across heterogeneous environments + +## SCI Impact + +**SCI = (E × I) + M per R** + +**E (Energy):** Efficient framework selection, optimized hardware utilization, and reduced latency directly lower energy consumption per inference or training operation. + +**M (Embodied Carbon):** Improved hardware utilization can reduce the need for additional infrastructure and associated embodied emissions. + +## Cost Impact + +- **Compute costs:** Reduced through efficient execution and better hardware utilization; faster inference reduces per-operation cost +- **Development costs:** May increase due to framework migration or retraining teams on new tools +- **Infrastructure costs:** Lower due to improved utilization and reduced resource requirements +- **Licensing costs:** Framework and runtime licensing vary by choice (most open-source options are free) +- **Trade-off:** Long-term compute savings must be weighed against upfront engineering investment and team ramp time + +## Assumptions + +- Selected frameworks and runtimes are compatible with application requirements +- Performance benchmarks reflect real-world workload behavior and hardware configurations +- Team has capacity to evaluate and learn new frameworks if migration is needed + +## Considerations + +- Framework migration may require significant effort and refactoring of existing code +- Compatibility with existing tools, libraries, pipelines, infrastructure, and runtime dependencies must be evaluated +- Existing infrastructure investments and backward compatibility requirements may constrain framework selection and migration options +- Some optimized runtimes may be hardware-specific (for example, NVIDIA TensorRT or Apple Metal) +- Training framework efficiency may differ from inference runtime efficiency; choose accordingly for your use case +- Framework maturity, ecosystem support, and portability should factor into the decision +- Performance gains must be validated under actual workload conditions, not just benchmarks + +## References + +- [ONNX Runtime — Cross-platform Inference](https://onnxruntime.ai/) +- [NVIDIA TensorRT — High-Performance Deep Learning Inference](https://developer.nvidia.com/tensorrt) +- [PyTorch — Machine Learning Framework](https://pytorch.org/) +- [TensorFlow — End-to-End ML Platform](https://www.tensorflow.org/) +- [OpenVINO — Intel AI Inference Toolkit](https://docs.openvino.ai/) +- [MLPerf Benchmarks — Framework Performance Comparison](https://mlcommons.org/benchmarks/) diff --git a/docs/operations/carbon-aware-ai-scheduling.md b/docs/operations/carbon-aware-ai-scheduling.md new file mode 100644 index 000000000..ed2e9cdd1 --- /dev/null +++ b/docs/operations/carbon-aware-ai-scheduling.md @@ -0,0 +1,87 @@ +--- +version: 1.0 +submitted_by: Naveen Balani +submitted_by_linkedin: https://www.linkedin.com/in/naveenbalani/ +published_date: +category: Operations +tags: + - ai + - machine-learning + - cloud + - compute + - persona:devops-engineer + - persona:infrastructure-engineer + - persona:solution-architect + - size:large +personas: DevOps Engineer, Infrastructure Engineer, Solution Architect +description: Reduce the carbon impact of AI workloads by running them in cloud regions with lower grid carbon intensity and scheduling deferrable jobs during periods of high renewable energy availability. +--- + +# Use carbon-aware scheduling and region selection for AI workloads + +**Applicable Role:** Provider and Consumer + +## Description + +AI workloads such as training, fine-tuning, and inference consume significant amounts of energy. The carbon impact of this energy depends on two factors: where the workload is executed (spatial) and when it is executed (temporal). + +Different cloud regions and data centers operate on energy grids with varying carbon intensity. Within any single region, carbon intensity also fluctuates by time of day and season as the share of renewable energy on the grid changes. + +By selecting low-carbon regions and scheduling deferrable workloads during periods of high renewable energy availability, organizations can significantly reduce emissions without changing the workload itself. Combining spatial and temporal shifting maximizes the carbon reduction effect on the I factor in the SCI equation. + +## Solution + +- Choose cloud regions that use a higher proportion of renewable or low-carbon energy +- Evaluate carbon intensity as a factor alongside cost, latency, and availability when selecting regions +- Run training, fine-tuning, and batch workloads in low-carbon regions where latency is less critical +- For inference workloads, balance user proximity with carbon-efficient regions +- Use carbon-aware scheduling tools to shift deferrable workloads to low-carbon time windows within a region +- Schedule training, fine-tuning, and batch processing during periods of high renewable energy availability +- Integrate carbon intensity signals from grid APIs into job scheduling and orchestration systems +- Pause and resume long-running training jobs based on carbon intensity thresholds where feasible +- Design pipelines to support flexible execution windows for non-time-critical workloads +- Re-evaluate region and scheduling choices periodically as energy mixes and cloud offerings evolve + +## SCI Impact + +**SCI = (E × I) + M per R** + +**I (Carbon Intensity):** Both spatial and temporal shifting directly reduce the carbon intensity factor in the SCI equation. This is the primary lever in this pattern. + +**E (Energy):** Energy consumption remains largely unchanged for the same workload, though pausing and resuming may introduce minor checkpoint overhead. + +## Cost Impact + +- **Compute costs:** May decrease in low-cost regions; varies by cloud provider pricing +- **Data transfer costs:** May increase due to cross-region data movement +- **Monitoring costs:** Carbon-aware scheduling tools add operational cost +- **SLA costs:** Potential cost increases if scheduling flexibility impacts performance SLAs +- **Trade-off:** Regional cost arbitrage may offset environmental gains; evaluate full cost picture + +## Assumptions + +- Workloads can be executed in alternative regions without violating data sovereignty or compliance constraints +- Carbon intensity data for regions and time periods is available and reliable +- Some workloads are deferrable and can tolerate flexible scheduling windows + +## Considerations + +- Trade-offs between carbon, latency, cost, and data sovereignty must be evaluated +- Moving workloads across regions may introduce data transfer overhead +- Some services or hardware may not be available in all regions +- Regulatory and compliance requirements may restrict region selection +- Not all workloads can be deferred; latency-sensitive inference requires immediate execution +- Pausing and resuming training may introduce checkpoint overhead and minor efficiency loss +- Carbon intensity data quality and granularity vary by region and provider +- Organizational SLAs and deadlines may constrain scheduling flexibility + +## References + +- [Google Cloud Sustainability Regions](https://cloud.google.com/sustainability/region-carbon) +- [AWS Customer Carbon Footprint Tool](https://aws.amazon.com/aws-cost-management/aws-customer-carbon-footprint-tool/) +- [Azure Emissions Impact Dashboard](https://www.microsoft.com/en-us/sustainability/emissions-impact-dashboard) +- [Electricity Maps — Real-time Carbon Intensity Data](https://www.electricitymaps.com/) +- [WattTime — Automated Emissions Reduction](https://watttime.org/) +- [Carbon Aware SDK — Green Software Foundation](https://github.com/Green-Software-Foundation/carbon-aware-sdk) +- [Google Carbon-Intelligent Computing](https://blog.google/outreach-initiatives/sustainability/carbon-intelligent-computing/) +- [Carbon-Aware Computing (Radovanovic et al., 2022)](https://arxiv.org/abs/2106.11750) \ No newline at end of file diff --git a/docusaurus.config.js b/docusaurus.config.js index 4abe70fcc..794e38788 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -11,8 +11,8 @@ const config = { tagline: "Practical green software patterns, organised by role and lifecycle stage, to reduce your software emissions.", // Change to site url - url: "https://patterns.greensoftware.foundation/", - baseUrl: "/", + url: "https://russelltrow.github.io/", + baseUrl: "/gsf-patterns/", onBrokenLinks: "throw", markdown: { hooks: { @@ -281,20 +281,20 @@ const config = { { from: "/catalog/cloud/shed-lower-priority-traffic", to: "/requirements/shed-lower-priority-traffic" }, // AI → Architecture (root) - { from: "/catalog/ai/compress-ml-models-for-inference", to: "/architecture/compress-ml-models-for-inference" }, + { from: "/catalog/ai/compress-ml-models-for-inference", to: "/development/right-sized-energy-efficient-ai-models" }, // AI → Architecture: System Topology - { from: "/catalog/ai/energy-efficent-ai-edge", to: "/architecture/system-topology/energy-efficent-ai-edge" }, + { from: "/catalog/ai/energy-efficent-ai-edge", to: "/architecture/system-topology/run-ai-models-edge" }, { from: "/catalog/ai/serverless-model-development", to: "/architecture/system-topology/serverless-model-development" }, // AI → Architecture: Technology Selection - { from: "/catalog/ai/efficent-format-for-model-training", to: "/architecture/technology-selection/efficent-format-for-model-training" }, + { from: "/catalog/ai/efficent-format-for-model-training", to: "/development/data-handling/optimize-data-storage-ai-training" }, { from: "/catalog/ai/energy-efficent-framework", to: "/architecture/technology-selection/energy-efficent-framework" }, { from: "/catalog/ai/energy-efficent-models", to: "/architecture/technology-selection/energy-efficent-models" }, - { from: "/catalog/ai/right-hardware-type", to: "/architecture/technology-selection/right-hardware-type" }, + { from: "/catalog/ai/right-hardware-type", to: "/architecture/system-topology/efficient-hardware-ai-workloads" }, // AI → Development (root) - { from: "/catalog/ai/leverage-sustainable-regions", to: "/development/leverage-sustainable-regions" }, + { from: "/catalog/ai/leverage-sustainable-regions", to: "/operations/carbon-aware-ai-scheduling" }, { from: "/catalog/ai/pre-trained-transfer-learning", to: "/development/pre-trained-transfer-learning" }, // Web → Development: Data Handling