< Previous Challenge - Home - Next Challenge >
In this challenge, you'll build an advanced agentic Retrieval-Augmented Generation (RAG) system using Azure AI Search. Unlike traditional RAG approaches that only retrieve and concatenate content, agentic retrieval allows the AI to decide what to retrieve, when to retrieve it, and how to iteratively refine its search strategy to answer complex, multi-part questions.
You will implement a conversational AI system that can:
- Dynamically search a knowledge base
- Reason about intermediate retrieval results
- Perform iterative refinement when information is incomplete
- Synthesize a grounded answer with cited sources
Throughout the process, you can inspect activities to understand how the agent planned, searched, and constructed its final response.
Before diving into the implementation, let's understand the key concepts that make agentic retrieval powerful for modern AI applications.
Traditional RAG typically features:
- Fixed similarity search per user prompt
- Limited reasoning about missing context
- Static chunk retrieval (one-shot)
- Blind concatenation of results
Agentic Retrieval enables:
- Dynamic Query Planning – the agent decomposes or reformulates queries
- Iterative Refinement – additional searches when gaps remain
- Contextual Reasoning – awareness of sufficiency vs. insufficiency
- Grounded Synthesis – combines sources while preserving attribution
graph TD
A[User Query] --> B[Agent Planning]
B --> C[Dynamic Search Query Generation]
C --> D[Knowledge Source Retrieval]
D --> E[Result Evaluation]
E --> F{Need More Info?}
F -->|Yes| C
F -->|No| G[Synthesis & Response]
G --> H[User Response with Sources]
Azure AI Search provides native support for agentic retrieval through the following capabilities:
- Knowledge Sources: Structured data repositories an agent can query
- Knowledge Agents: Orchestrators that retrieve, reason, and synthesize
- Retrieval Orchestration: Coordination of multi-step query execution
- Activity Tracking: Logs of decision paths and retrieval operations
Agentic retrieval enables enterprise scenarios such as:
- Multi-Source Knowledge: Unified retrieval across documents, databases, APIs
- Contextual Understanding: Domain-aware reasoning about user intent
- Auditability: Trace how each answer was formed
- Source Attribution: Cite underlying documents for verification
- Security Alignment: Honor data governance and access control boundaries
In this challenge, you'll complete a partially implemented agentic RAG system that demonstrates how AI agents can intelligently search through a knowledge base about "Earth at Night" using NASA data. The system showcases advanced retrieval capabilities in which the agent decides what to search for and how to synthesize the results.
You'll work with the starter project. The conversational interface is already implemented; your objective is to complete the core agentic retrieval workflow powered by Azure AI Search.
Tip: If you get stuck, refer to the Coach solution for inspiration.
Create and configure the required Azure services for agentic retrieval.
-
Create Azure AI Search Service:
- Create a new Azure AI Search service in the Azure portal
- Choose a Standard tier or higher to support vector search and semantic capabilities
-
Deploy required models:
- Chat completion model (for reasoning and synthesis), for example
gpt-4.1 - Embedding model (for vector search), for example
text-embedding-3-large
- Chat completion model (for reasoning and synthesis), for example
Azure AI Search agentic retrieval requires that the Search service can call Azure OpenAI. Configure these permissions:
For Azure AI Search:
- Enable role-based access control (RBAC) on the Search service.
- Enable a system-assigned managed identity.
- Assign these roles to yourself (development convenience):
Search Service Contributor,Search Index Data Contributor,Search Index Data Reader.
For Azure OpenAI:
- Assign the
Cognitive Services OpenAI Userrole to the Search service's managed identity.
Navigate to the starter project and configure your Azure service settings.
- Open the starter project:
cd Student/Resources/Challenge-11/python-
Update environment variables with your Azure service configurations. Copy
.env.sampleto.envand fill in your values:# Azure OpenAI AZURE_OPENAI_ENDPOINT=https://your-openai-service.openai.azure.com AZURE_OPENAI_API_KEY=your-openai-api-key AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4.1 AZURE_OPENAI_MODEL=gpt-4.1 AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME=text-embedding-3-large AZURE_OPENAI_EMBEDDINGS_MODEL=text-embedding-3-large # Azure AI Search AZURE_AI_SEARCH_ENDPOINT=https://your-search-service.search.windows.net AZURE_AI_SEARCH_KEY=your-search-admin-key AZURE_AI_SEARCH_INDEX_NAME=nasa-earth-night-index AZURE_AI_SEARCH_KNOWLEDGE_SOURCE_NAME=nasa-knowledge-source AZURE_AI_SEARCH_KNOWLEDGE_AGENT_NAME=earth-night-agent
Implement four core components:
- Search Index – Create index fields (id, chunk content, embedding vector, page number) plus semantic and vector configurations.
- Data Ingestion – Upload NASA "Earth at Night" dataset into the index.
- Knowledge Source – Reference the index and specify included fields.
- Knowledge Agent – Bridge OpenAI deployments and knowledge source for multi-step retrieval & synthesis.
Reference Guide: For detailed implementation steps and code examples, you can reference the Azure AI Search agentic retrieval quickstart guide which provides comprehensive examples for each component.
Create a search index. The index schema contains fields for document identification and page content, embeddings, and numbers. The schema also includes configurations for semantic ranking and vector search, which uses your embeddings deployment to vectorize text and match documents based on semantic or conceptual similarity.
You can create the search index programmatically (as shown below) or by using the Azure portal. In the Azure portal, navigate to your Azure AI Search resource, select Indexes, and use the UI to define fields, semantic settings, and vector search configurations. This provides a visual way to set up your index if you prefer not to use code.
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
SearchIndex,
SearchField,
SearchFieldDataType,
SimpleField,
VectorSearch,
VectorSearchProfile,
HnswAlgorithmConfiguration,
AzureOpenAIVectorizer,
AzureOpenAIVectorizerParameters,
SemanticConfiguration,
SemanticSearch,
SemanticPrioritizedFields,
SemanticField
)
from azure.core.credentials import AzureKeyCredential
# Define fields for the index
fields = [
SimpleField(
name="id",
type=SearchFieldDataType.String,
key=True,
filterable=True,
sortable=True,
facetable=True
),
SearchField(
name="page_chunk",
type=SearchFieldDataType.String,
filterable=False,
sortable=False,
facetable=False
),
SearchField(
name="page_embedding_text_3_large",
type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
vector_search_dimensions=3072,
vector_search_profile_name="hnsw_text_3_large"
),
SimpleField(
name="page_number",
type=SearchFieldDataType.Int32,
filterable=True,
sortable=True,
facetable=True
)
]
# Define a vectorizer
vectorizer = AzureOpenAIVectorizer(
vectorizer_name="azure_openai_text_3_large",
parameters=AzureOpenAIVectorizerParameters(
resource_url=aoai_endpoint,
deployment_name=aoai_embedding_deployment,
model_name=aoai_embedding_model
)
)
# Define a vector search profile and algorithm
vector_search = VectorSearch(
profiles=[
VectorSearchProfile(
name="hnsw_text_3_large",
algorithm_configuration_name="alg",
vectorizer_name="azure_openai_text_3_large"
)
],
algorithms=[
HnswAlgorithmConfiguration(name="alg")
],
vectorizers=[vectorizer]
)
# Define a semantic configuration
semantic_config = SemanticConfiguration(
name="semantic_config",
prioritized_fields=SemanticPrioritizedFields(
content_fields=[SemanticField(field_name="page_chunk")]
)
)
semantic_search = SemanticSearch(
default_configuration_name="semantic_config",
configurations=[semantic_config]
)
# Create the index
index = SearchIndex(
name=index_name,
fields=fields,
vector_search=vector_search,
semantic_search=semantic_search
)
# Create the index client and create or update the index
index_client = SearchIndexClient(
endpoint=search_endpoint,
credential=AzureKeyCredential(search_key)
)
index_client.create_or_update_index(index)
print(f"Index '{index_name}' created or updated successfully.")You can upload data to your Azure AI Search index in two main ways:
-
Using an Indexer via the Azure Portal (Pull Method): This is known as the "pull" method, where Azure AI Search automatically pulls data from supported data sources (such as Azure Blob Storage, SQL Database, or Cosmos DB) using an indexer. You can configure and schedule indexers in the Azure portal, making it ideal for ongoing or large-scale data ingestion scenarios. The portal provides a visual interface for mapping fields and managing indexing jobs.
-
Uploading Data Programmatically (Push Method): This is the "push" method, where you directly push data into the search index using code. For simplicity in this challenge, you'll upload a single file containing NASA "Earth at Night" data directly to the index programmatically. This approach is quick and effective for small datasets or initial prototyping.
from azure.search.documents import SearchClient
import requests
import json
# Upload sample documents from the GitHub URL
url = "https://raw.githubusercontent.com/Azure-Samples/azure-search-sample-data/refs/heads/main/nasa-e-book/earth-at-night-json/documents.json"
response = requests.get(url)
response.raise_for_status()
documents = response.json()
search_client = SearchClient(
endpoint=search_endpoint,
index_name=index_name,
credential=AzureKeyCredential(search_key)
)
# Upload documents in batches
result = search_client.upload_documents(documents=documents)
print(f"Documents uploaded to index '{index_name}' successfully.")Configure a knowledge source that the agent can query. A knowledge source is a reusable reference to your source data. The following code defines a knowledge source that targets your index.
from azure.search.documents.indexes.models import (
SearchIndexKnowledgeSource,
SearchIndexKnowledgeSourceParameters
)
# Create a knowledge source
index_knowledge_source = SearchIndexKnowledgeSource(
name=knowledge_source_name,
search_index_parameters=SearchIndexKnowledgeSourceParameters(
search_index_name=index_name,
source_data_select="id,page_chunk,page_number"
)
)
index_client.create_or_update_knowledge_source(index_knowledge_source)
print(f"Knowledge source '{knowledge_source_name}' created or updated successfully.")Set up the intelligent agent that will perform agentic retrieval.
To target your knowledge source and your model deployment at query time, you need a knowledge agent. A knowledge agent connects your Azure OpenAI deployment with one or more knowledge sources, enabling advanced retrieval and synthesis capabilities.
Tip: You can add multiple knowledge sources to a single knowledge agent. This allows the agent to retrieve and reason across different datasets, indexes, or repositories, making your retrieval system more flexible and powerful.
from azure.search.documents.indexes.models import (
KnowledgeAgent,
KnowledgeAgentAzureOpenAIModel,
KnowledgeAgentOutputConfiguration,
KnowledgeAgentOutputConfigurationModality,
KnowledgeSourceReference
)
# Create a knowledge agent
openai_parameters = AzureOpenAIVectorizerParameters(
resource_url=aoai_endpoint,
deployment_name=aoai_gpt_deployment,
model_name=aoai_gpt_model
)
agent_model = KnowledgeAgentAzureOpenAIModel(
azure_open_ai_parameters=openai_parameters
)
output_config = KnowledgeAgentOutputConfiguration(
modality=KnowledgeAgentOutputConfigurationModality.ANSWER_SYNTHESIS,
include_activity=True
)
agent = KnowledgeAgent(
name=knowledge_agent_name,
models=[agent_model],
knowledge_sources=[
KnowledgeSourceReference(
reference_name=knowledge_source_name,
include_references=True,
include_reference_source_data=True,
reranker_threshold=2.5
)
],
output_configuration=output_config
)
index_client.create_or_update_knowledge_agent(agent)
print(f"Knowledge agent '{knowledge_agent_name}' created or updated successfully.")You can now run agentic retrieval by sending a user query (single or multi-part) to your knowledge agent. Given conversation history and retrieval parameters, the agent:
- Analyzes prior turns to infer the current information need
- Decomposes complex prompts into focused subqueries
- Executes subqueries (often in parallel) against the knowledge source
- Applies semantic ranking to refine relevance
- Synthesizes grounded, cited natural-language output
Use the following code template to send a user query and retrieve results from your knowledge agent. Update the query content as needed for your scenario:
from azure.search.documents.agents import KnowledgeAgentRetrievalClient
from azure.search.documents.agents.models import (
KnowledgeAgentRetrievalRequest,
KnowledgeAgentMessage,
KnowledgeAgentMessageTextContent
)
from azure.identity import DefaultAzureCredential
# Create retrieval client
agent_client = KnowledgeAgentRetrievalClient(
endpoint=search_endpoint,
agent_name=knowledge_agent_name,
credential=DefaultAzureCredential()
)
messages.append({
"role": "user",
"content": """Why do suburban belts display larger December brightening than urban cores even though absolute light levels are higher downtown?
Why is the Phoenix nighttime street grid is so sharply visible from space, whereas large stretches of the interstate between midwestern cities remain comparatively dim?"""
})
# Convert messages to the required format
agent_messages = [
KnowledgeAgentMessage(
content=[KnowledgeAgentMessageTextContent(text=msg["content"])],
role=msg["role"]
)
for msg in messages if msg["role"] != "system"
]
retrieval_result = agent_client.retrieve(
retrieval_request=KnowledgeAgentRetrievalRequest(messages=agent_messages)
)
# Add assistant response to conversation history
assistant_response = retrieval_result.response[0].content[0].text
messages.append({
"role": "assistant",
"content": assistant_response
})Inspect the agent's internal reasoning and retrieval trace. Key elements:
- Response – Final synthesized answer
- Activity – Step-by-step actions taken
- Results – Source documents and retrieval metadata
Use the following code to display the agent's response, activity log, and source references:
import json
# Print the response, activity, and results
print("Response:")
print(retrieval_result.response[0].content[0].text)
print("\nActivity:")
for activity in retrieval_result.activity:
print(f"Activity Type: {type(activity).__name__}")
activity_json = json.dumps(
activity.as_dict(),
indent=2,
default=str
)
print(activity_json)
print("\nResults:")
for reference in retrieval_result.references:
print(f"Reference Type: {type(reference).__name__}")
reference_json = json.dumps(
reference.as_dict(),
indent=2,
default=str
)
print(reference_json)Test the agentic retrieval with varied question types to observe planning depth and refinement.
-
Question Types:
- Factual Questions: "What is the Earth at night project about?"
- Analytical Questions: "How does light pollution affect astronomical observations?"
- Comparative Questions: "What are the differences between urban and rural nighttime lighting?"
- Complex Queries: "Explain the relationship between economic development and nighttime lighting patterns"
-
Observe Behavior:
- Note retrieval strategy evolution
- Review activity logs for decision points
- Examine reference metadata and synthesis quality
Pay attention to:
- How the agent reformulates queries for better recall
- When it determines more information is required
- How it fuses multiple sources without hallucination
- The precision and clarity of source attribution
- ✅ Azure AI Search and Azure OpenAI services are provisioned and accessible
- ✅ Application configuration is complete, including endpoints, API keys, and deployment names
- ✅ Search index is created with vector and semantic settings, and NASA data is successfully ingested
- ✅ Knowledge source and knowledge agent are set up and working
- ✅ Agent performs multi-step retrieval, as shown in the activity log
- ✅ Multi-turn conversations maintain context across user turns
- ✅ Retrieval decisions can be explained and audited using activity logs and references