RAG Pipelines on Azure for AI-103: Azure AI Search, Vector Search, Hybrid Retrieval, Semantic Ranking, Chunking Strategies, Embedding Models, Integrated Vectorization, Skillsets, and Building Production RAG

Leave a Comment / AI/ML, Azure

Table of Contents

In the previous post, we covered prompt engineering — the foundation for making models respond reliably. But even the best prompt cannot answer questions about YOUR company’s data if the model has never seen it. RAG (Retrieval Augmented Generation) solves this: retrieve relevant documents first, then generate answers grounded in those documents. This is the most tested pattern in AI-103’s largest domain (30-35%).

Analogy — A research librarian. Without RAG, the model is like a professor answering questions from memory — knowledgeable but sometimes wrong or outdated. With RAG, the model becomes a research librarian: when you ask a question, the librarian first searches the catalog (Azure AI Search), retrieves the relevant books and papers (document chunks), reads the relevant sections (context window), and then answers your question with citations. The librarian’s answer is grounded in what they actually read, not what they vaguely remember from years ago.

What Is RAG and Why It Matters

RAG = Retrieval Augmented Generation

The problem RAG solves:
  Language models are trained on public data up to a cutoff date.
  They do NOT know:
    - Your company's internal documents
    - Data added after their training cutoff
    - Private databases, policies, or procedures
    - Real-time information (stock prices, live status)

  Without RAG: the model guesses or halluccinates
  With RAG: the model retrieves YOUR data and answers from it

RAG vs other approaches:
  Fine-tuning: teaches the model new patterns/style (expensive, slow)
  RAG: gives the model access to data at query time (fast, updatable)
  Most applications need RAG, not fine-tuning.

  Fine-tune for: tone, format, domain vocabulary
  RAG for: factual answers from specific documents

  The default order: prompting → RAG → fine-tuning
  Most problems are solved at the RAG stage.

The RAG Architecture — Six Steps

The RAG pipeline has two phases: INGESTION (offline) and QUERY (real-time).

INGESTION (runs once or on schedule):
  Step 1: INGEST
    Source: Blob Storage, SQL, Cosmos DB, SharePoint, web pages
    → Pull documents into the pipeline

  Step 2: CHUNK
    Break large documents into smaller pieces (200-1000 tokens each)
    → Each chunk is independently searchable

  Step 3: EMBED
    Convert each chunk into a vector (array of numbers)
    Using: text-embedding-3-large or text-embedding-3-small
    → Vectors capture the MEANING of the text

  Step 4: INDEX
    Store chunks + vectors + metadata in Azure AI Search
    → Searchable index ready for queries

QUERY (runs for every user question):
  Step 5: RETRIEVE
    User question → embed the question → search the index
    → Return the top-K most relevant chunks (e.g., top 5)

  Step 6: GENERATE
    Combine: system message + retrieved chunks + user question
    Send to GPT-4o → generate grounded answer
    → Answer is based on YOUR documents, not model memory

  Complete flow:
    User asks: "What is our refund policy?"
    → Question embedded as vector
    → AI Search finds top 5 matching chunks from policy documents
    → Chunks included in prompt as context
    → GPT-4o reads chunks and generates answer: "Our refund policy allows..."
    → Answer is grounded in the actual policy document

Analogy — A data pipeline you already know. RAG ingestion is just another ETL pipeline. Extract (pull documents from Blob Storage), Transform (chunk, embed, enrich), Load (index in AI Search). The only difference is the “T” includes embedding text into vectors. If you build ADF or Databricks pipelines, you already understand the pattern.

Azure AI Search — The Retrieval Engine

Azure AI Search (formerly Azure Cognitive Search) is the retrieval engine
that stores and searches your documents for RAG.

What it provides:
  - Full-text search (keyword matching with BM25 scoring)
  - Vector search (semantic similarity using embeddings)
  - Hybrid search (keyword + vector combined)
  - Semantic ranking (AI re-ranking for relevance)
  - Indexers (automated data ingestion from Azure sources)
  - Skillsets (AI enrichment during indexing: OCR, NER, embedding)
  - Integrated vectorization (chunking + embedding built-in)

Azure AI Search is NOT:
  - A database (it is a search index, not a transactional store)
  - A vector-only database (it supports text, vectors, and structured data)
  - A replacement for Azure SQL or Cosmos DB (different purpose)

Pricing tiers:
  Free: 3 indexes, 50 MB, for testing only
  Basic: 15 indexes, 2 GB, small workloads
  Standard (S1/S2/S3): production workloads, scaling, replicas
  Storage Optimized (L1/L2): large document collections

For AI-103:
  Know that AI Search is the default retrieval engine for RAG
  Know the difference between full-text, vector, and hybrid search
  Know that semantic ranking is an add-on feature (additional cost)

Creating a Search Index

A search index defines the schema for your searchable content: what fields exist, which are searchable, filterable, or sortable, and which contain vectors.

# Create an index with text and vector fields
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    SearchIndex, SearchField, SearchFieldDataType,
    VectorSearch, HnswAlgorithmConfiguration, VectorSearchProfile,
    SemanticConfiguration, SemanticSearch, SemanticPrioritizedFields, SemanticField
)
from azure.identity import DefaultAzureCredential

client = SearchIndexClient(
    endpoint="https://your-search.search.windows.net",
    credential=DefaultAzureCredential()
)

index = SearchIndex(
    name="documents-index",
    fields=[
        # Text fields
        SearchField(name="id", type=SearchFieldDataType.String, key=True),
        SearchField(name="title", type=SearchFieldDataType.String, searchable=True),
        SearchField(name="content", type=SearchFieldDataType.String, searchable=True),
        SearchField(name="source", type=SearchFieldDataType.String, filterable=True),
        SearchField(name="page_number", type=SearchFieldDataType.Int32, filterable=True),

        # Vector field (for semantic similarity search)
        SearchField(
            name="content_vector",
            type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
            searchable=True,
            vector_search_dimensions=3072,  # text-embedding-3-large
            vector_search_profile_name="vector-profile"
        ),
    ],
    vector_search=VectorSearch(
        algorithms=[HnswAlgorithmConfiguration(name="hnsw-config")],
        profiles=[VectorSearchProfile(name="vector-profile", algorithm_configuration_name="hnsw-config")]
    ),
    semantic_search=SemanticSearch(
        configurations=[SemanticConfiguration(
            name="semantic-config",
            prioritized_fields=SemanticPrioritizedFields(
                content_fields=[SemanticField(field_name="content")]
            )
        )]
    )
)

client.create_index(index)
Index field types:

  SearchFieldDataType.String:     text content, titles, descriptions
  SearchFieldDataType.Int32:      numbers (page numbers, counts)
  SearchFieldDataType.DateTimeOffset:  timestamps
  SearchFieldDataType.Boolean:    flags (is_active, is_published)
  SearchFieldDataType.Collection(Single):  vector field (float array)

Field attributes:
  searchable:  included in full-text search (text fields)
  filterable:  can be used in filter expressions (source = "HR")
  sortable:    can be sorted on (date, score)
  facetable:   can be used for faceted navigation (category counts)
  key:         unique identifier (required, one per index)

Indexers and Data Sources — Automated Ingestion

Indexers automatically pull data from Azure sources, process it through skillsets, and populate your search index.

Supported data sources:
  Azure Blob Storage (PDFs, Word, text, JSON)
  Azure SQL Database
  Azure Cosmos DB
  Azure Data Lake Storage Gen2
  Azure Table Storage
  SharePoint Online

Indexer workflow:
  1. Data Source: defines WHERE to pull data from (connection string + container)
  2. Skillset: defines HOW to process data (chunk, embed, extract entities)
  3. Index: defines WHERE to store results (field schema)
  4. Indexer: ties it all together + runs on a schedule

Scheduling:
  Run once: manual trigger
  Recurring: every 5 minutes, hourly, daily
  Change detection: only process new or modified documents
  High watermark: tracks last processed timestamp

For AI-103:
  Know the indexer pipeline: Data Source → Skillset → Index
  Know which data sources are supported
  Know that change detection enables incremental indexing
  Know that indexers can run on a schedule for automated refresh

Chunking Strategies — Breaking Documents into Pieces

Chunking is the most important decision in a RAG pipeline. How you split documents directly affects retrieval quality.

Analogy — Cutting a textbook into flashcards. If you cut by page (fixed-size), some flashcards end mid-sentence and lose meaning. If you cut by section (semantic), each flashcard contains a complete concept. If you cut by paragraph (sentence-based), you get coherent units but some may be too short for context. The right strategy depends on your documents and your questions.

Chunking strategies:

  1. FIXED-SIZE (simplest):
     Split every N tokens (e.g., 500 tokens per chunk)
     Overlap: include last 100 tokens of previous chunk (context continuity)
     Pros: simple, predictable chunk sizes, easy to implement
     Cons: may split mid-sentence, mid-paragraph, or mid-concept
     Best for: uniform documents (logs, records, structured text)

     Azure: Text Split skill with textSplitMode = "pages"
            maximumPageLength = 2000 (characters), pageOverlapLength = 500

  2. SENTENCE-BASED:
     Split on sentence boundaries
     Group sentences into chunks of target size (e.g., 5-10 sentences)
     Pros: respects sentence boundaries, more coherent
     Cons: chunk sizes vary, may still split concepts
     Best for: general documents (articles, policies, manuals)

  3. SEMANTIC / PARAGRAPH:
     Split on paragraph, section, or heading boundaries
     Preserves document structure and conceptual units
     Pros: most coherent chunks, preserves context
     Cons: chunk sizes vary widely, some sections may be very large
     Best for: structured documents with clear sections (legal, technical)

  4. DOCUMENT LAYOUT:
     Uses AI to understand document structure (headings, tables, lists)
     Azure: Document Layout skill (preview)
     Preserves tables and structured content as complete units
     Best for: PDFs with complex formatting, tables, multi-column layouts

Chunk size guidelines:
  Too small (< 100 tokens): not enough context, poor answers
  Sweet spot (200-800 tokens): good balance of context and precision
  Too large (> 2000 tokens): too much irrelevant content, dilutes the answer

  Overlap: 10-25% of chunk size prevents information loss at boundaries

For AI-103:
  Know all four strategies and when to use each
  Know that chunk size affects retrieval quality
  Know that overlap prevents information loss at chunk boundaries
  Know the Text Split skill and Document Layout skill in Azure

Embedding Models — Converting Text to Vectors

Embeddings convert text into dense vector arrays where similar meanings are represented by similar numbers. This enables searching by meaning rather than exact keywords.

Analogy — Translating language into GPS coordinates. Each text chunk gets a GPS coordinate in a high-dimensional space. Similar texts end up near each other: “Azure Data Factory pipeline” and “ADF data pipeline” would be close together, even though they use different words. When you search, your question also gets a GPS coordinate, and the search finds the nearest chunks.

Embedding models in Azure:

  text-embedding-3-large (OpenAI):
    Dimensions: 3072 (or configurable: 256, 1024, 3072)
    Quality: highest
    Cost: $0.13 per 1M tokens
    Best for: production RAG with high accuracy requirements

  text-embedding-3-small (OpenAI):
    Dimensions: 1536
    Quality: good (slightly lower than large)
    Cost: $0.02 per 1M tokens (6.5x cheaper)
    Best for: cost-sensitive applications, development/testing

  Cohere embed-v3:
    Dimensions: 1024
    Quality: good, strong multilingual support
    Best for: multilingual RAG applications

Key concepts:
  Dimensions: higher = more expressive but more storage and slower search
  Cosine similarity: measures how similar two vectors are (-1 to 1, higher = more similar)
  The SAME embedding model must be used for indexing AND querying
  (embed documents with text-embedding-3-large → embed queries with text-embedding-3-large)

For AI-103:
  Know text-embedding-3-large vs text-embedding-3-small
  Know that the same model must be used for indexing and querying
  Know what dimensions mean and the tradeoff (quality vs cost/speed)

Vector Search — Finding Similar Meaning

Vector search finds documents with similar meaning to the query, even if they use different words.

How vector search works:

  1. User query: "How do I load data into a data warehouse?"
  2. Query embedding: [0.021, -0.045, 0.083, ...] (3072 numbers)
  3. Search: find the K nearest vectors in the index (cosine similarity)
  4. Results: chunks about data loading, even if they use words like
     "ingest," "ETL," "COPY INTO," or "bulk insert" instead of "load"

  This is different from keyword search:
    Keyword: "load data warehouse" → only finds documents with those exact words
    Vector: "load data warehouse" → finds documents about data loading, ETL,
            ingestion, bulk insert — matching MEANING, not keywords

Vector search algorithms in Azure AI Search:
  HNSW (Hierarchical Navigable Small World):
    Default algorithm, good for most scenarios
    Approximate nearest neighbors (fast, slight accuracy tradeoff)
    Configurable: efConstruction, efSearch, metric (cosine, dotProduct, euclidean)

  Exhaustive KNN:
    Checks every vector (exact, but slow for large indexes)
    Use for: small indexes or when recall is critical

Query example:
  results = search_client.search(
      search_text=None,  # No keyword search
      vector_queries=[
          VectorizedQuery(
              vector=query_embedding,  # Your embedded question
              k_nearest_neighbors=5,   # Return top 5 matches
              fields="content_vector"  # Which vector field to search
          )
      ]
  )

Semantic Search — AI-Powered Relevance

Semantic ranking is an AI-powered re-ranking layer that improves relevance by understanding meaning, not just keyword matches.

How semantic ranking works:

  1. Initial retrieval: keyword or hybrid search returns top 50 results
  2. Semantic ranker: a CROSS-ENCODER model reads each result + the query together
  3. Re-ranking: results are re-ordered by semantic relevance
  4. Output: top results are the most meaningfully relevant

  Cross-encoder vs bi-encoder:
    Bi-encoder (vector search): query and document encoded separately, compared by cosine
    Cross-encoder (semantic ranker): query and document encoded TOGETHER, deeper understanding
    Cross-encoders catch nuances that vector similarity misses (negation, complex relationships)

  Example:
    Query: "What happens when you DROP a managed table?"
    Vector search might return chunks about CREATE TABLE (related but wrong)
    Semantic ranker would promote chunks specifically about DROP behavior

Configuration:
  Enabled per index via SemanticConfiguration
  Applied at query time with query_type="semantic"
  Additional cost per query (semantic ranker is a premium feature)
  Available on Standard tier and above

For AI-103:
  Know that semantic ranking is a RE-RANKING layer, not a search type
  Know it uses a cross-encoder (different from vector bi-encoder)
  Know it improves relevance for complex queries
  Know it requires Standard tier or above (additional cost)

Hybrid Search — The Best of All Worlds

Hybrid search combines keyword search, vector search, and optionally semantic ranking in a single query. This is the recommended approach for production RAG.

Three search types combined:

  1. Keyword (BM25): exact term matching, good for specific terms, names, codes
  2. Vector: semantic similarity, good for meaning and paraphrases
  3. Semantic ranker: AI re-ranking for relevance (optional layer on top)

Why hybrid beats individual approaches:
  Keyword alone: misses paraphrases ("load data" doesn't find "ingest records")
  Vector alone: misses exact matches (product codes, error messages, names)
  Hybrid: catches both paraphrases AND exact terms

  Research shows hybrid search outperforms either alone by 10-30% on recall

Reciprocal Rank Fusion (RRF):
  Azure AI Search uses RRF to combine keyword and vector results
  Each result gets a score from both keyword and vector search
  RRF merges the two score lists into a single ranked result set
  Results that score well on BOTH methods rank highest
# Hybrid search: keyword + vector + semantic ranking
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery

results = search_client.search(
    search_text="How do I load data into a warehouse?",  # Keyword search
    vector_queries=[
        VectorizedQuery(
            vector=query_embedding,          # Vector search
            k_nearest_neighbors=10,
            fields="content_vector"
        )
    ],
    query_type="semantic",                   # Semantic re-ranking
    semantic_configuration_name="semantic-config",
    top=5                                    # Return top 5 results
)

for result in results:
    print(f"Score: {result['@search.score']:.3f}")
    print(f"Reranker: {result['@search.reranker_score']:.3f}")
    print(f"Content: {result['content'][:100]}...")

Integrated Vectorization — Azure Handles Everything

Integrated vectorization automates the entire RAG ingestion pipeline: chunking, embedding, and indexing — all within Azure AI Search.

Without integrated vectorization (manual pipeline):
  1. YOU write code to chunk documents
  2. YOU call the embedding API for each chunk
  3. YOU push vectors + text to the search index
  4. YOU manage the pipeline (scheduling, error handling, updates)

With integrated vectorization (Azure manages it):
  1. Configure a data source (Blob Storage)
  2. Configure a skillset (Text Split skill + embedding skill)
  3. Configure an indexer (connects data source → skillset → index)
  4. Azure handles chunking, embedding, and indexing automatically
  5. Schedule the indexer for automatic refresh

Components:
  Text Split skill: chunks documents (configurable size and overlap)
  AzureOpenAIEmbedding skill: generates vectors using your embedding deployment
  Document Layout skill: AI-powered document structure analysis
  Vectorizer: embeds queries at search time (same model as indexing)

Benefits:
  - No custom code for the ingestion pipeline
  - Automatic change detection (only re-index modified documents)
  - Built-in scheduling (hourly, daily)
  - Consistent chunking and embedding across all documents

For AI-103:
  Know that integrated vectorization eliminates custom ingestion code
  Know the three key skills: Text Split, AzureOpenAIEmbedding, Document Layout
  Know that a vectorizer handles query-time embedding automatically
  Know this is the RECOMMENDED approach for new RAG implementations

Skillsets — AI Enrichment During Indexing

Skillsets run AI processing on documents during indexing. They add intelligence to the ingestion pipeline.

Built-in skills:
  Text Split:           chunk documents into smaller pieces
  AzureOpenAIEmbedding: generate vector embeddings
  OCR:                  extract text from images in documents
  Entity Recognition:   extract people, places, organizations
  Key Phrase Extraction: identify important phrases
  Language Detection:    identify the document language
  Sentiment Analysis:    detect positive/negative sentiment
  PII Detection:         find and redact personal information
  Document Layout:       AI-powered structure analysis (tables, sections)
  Image Analysis:        describe images in documents

Custom skills:
  Call YOUR API during indexing (Azure Function or any REST endpoint)
  Use for: domain-specific processing, custom embedding models, business logic

Skillset pipeline example:
  Document (PDF) 
    → OCR skill (extract text from scanned pages)
    → Text Split skill (chunk into 500-token pieces)
    → Entity Recognition skill (extract people and organizations)
    → AzureOpenAIEmbedding skill (generate vectors)
    → Index (store text, vectors, entities, metadata)

On Your Data — Quick RAG Without Code

"On Your Data" is Azure OpenAI's built-in RAG feature.
Connect an AI Search index directly to your model deployment -- no code needed.

Setup in Foundry Playground:
  1. Open Playground → Chat
  2. Click "Add your data"
  3. Select: Azure AI Search
  4. Choose your search index
  5. Configure: search type (vector, hybrid, semantic)
  6. Start chatting -- model automatically retrieves and cites your data

What On Your Data does:
  - Automatically embeds user queries
  - Searches the connected AI Search index
  - Includes retrieved chunks in the prompt
  - Model generates grounded answers with citations

Limitations:
  - Less control than a custom RAG pipeline
  - Cannot customize prompt templates
  - Limited to Azure AI Search as retrieval source
  - Cannot chain multiple retrieval steps
  - Good for: prototyping, simple use cases, demos
  - Use Prompt Flow or Agent Service for: production, complex orchestration

For AI-103:
  Know On Your Data as a quick RAG setup (no code)
  Know its limitations vs custom RAG
  Know it is useful for prototyping but not production-grade orchestration

Building a Production RAG Pipeline

Production RAG architecture:

  INGESTION (automated, scheduled):
    Blob Storage (PDFs, docs)
      → Azure AI Search Indexer
        → Skillset:
            Text Split skill (chunk: 500 tokens, overlap: 100)
            AzureOpenAIEmbedding skill (text-embedding-3-large)
        → Search Index (text + vectors + metadata)
    Schedule: daily at 2 AM, change detection enabled

  QUERY (real-time, per user request):
    User question
      → Prompt Flow or application code:
          1. Embed the question (text-embedding-3-large)
          2. Hybrid search (keyword + vector + semantic ranking)
          3. Retrieve top 5 chunks
          4. Format prompt: system message + chunks + question
          5. Call GPT-4o
          6. Content Safety check on response
          7. Return grounded answer with citations

  Key design decisions:
    Chunk size: 500 tokens with 100-token overlap (start here, tune later)
    Embedding model: text-embedding-3-large (highest quality)
    Search type: hybrid + semantic ranking (best recall)
    Top-K: 5 chunks (balance between context and noise)
    Model: GPT-4o (best reasoning for complex questions)

  Monitoring:
    Track: retrieval precision (are the right chunks returned?)
    Track: groundedness (is the answer based on chunks?)
    Track: user satisfaction (are users getting helpful answers?)
    Track: latency (is the pipeline fast enough?)

Evaluating RAG Quality

RAG quality depends on TWO stages: retrieval quality and generation quality.

Retrieval metrics (is the search returning the right chunks?):
  Recall: what percentage of relevant chunks were retrieved?
  Precision: what percentage of retrieved chunks were actually relevant?
  Mean Reciprocal Rank (MRR): how high does the first relevant result appear?

Generation metrics (is the model answering well?):
  Groundedness: is the answer based on the retrieved chunks?
  Relevance: does the answer address the user's question?
  Coherence: is the answer well-structured and readable?
  Fluency: is the language natural and grammatically correct?

Evaluating in Foundry:
  1. Create a test dataset: questions + expected answers + source documents
  2. Run evaluation flow: system processes each question through the RAG pipeline
  3. Built-in evaluators score: groundedness, relevance, coherence, fluency
  4. Review results: which questions scored low? Why?
  5. Iterate: adjust chunking, retrieval, prompts based on evaluation

Common causes of poor RAG quality:
  Low groundedness → chunks are irrelevant (fix retrieval or chunking)
  Low relevance → good chunks but bad prompt (fix system message)
  Low coherence → model struggles to synthesize multiple chunks (reduce top-K)

Common Mistakes

  1. Using keyword search only for RAG. Keyword search misses paraphrases and synonyms. A user asking “how to ingest data” will not find documents about “loading records” or “ETL pipelines.” Always use hybrid search (keyword + vector) for production RAG. Research shows 10-30% improvement in recall over keyword alone.

  2. Using different embedding models for indexing and querying. If you embed documents with text-embedding-3-large (3072 dimensions) and queries with text-embedding-3-small (1536 dimensions), the vectors are incompatible and search returns garbage results. Always use the SAME embedding model for both indexing and querying.

  3. Chunking too large or too small. Chunks of 50 tokens lack context and produce vague answers. Chunks of 5000 tokens include too much irrelevant content and dilute the answer. Start with 500 tokens and 100-token overlap, then tune based on evaluation metrics.

  4. Not using overlap between chunks. Without overlap, information at chunk boundaries is split across two chunks. If the answer spans a boundary, neither chunk contains the full answer. Use 10-25% overlap to ensure continuity.

  5. Skipping semantic ranking for production RAG. Vector search alone may rank results that are topically similar but not directly relevant. Semantic ranking re-scores results using a cross-encoder that reads the query and document together, catching nuances that vector similarity misses. Enable it for production workloads.

  6. Returning too many chunks to the model. Sending 20 chunks to GPT-4o floods the context window with information, much of it irrelevant. The model struggles to find the answer in the noise. Start with top-5 chunks. Only increase if evaluation shows the answer is not in the retrieved set.

  7. Not evaluating RAG quality systematically. Without evaluation, you do not know if retrieval returns the right chunks or if the model generates grounded answers. Create a test dataset of 50-100 questions with expected answers, run evaluations in Foundry, and track groundedness, relevance, and coherence over time.

  8. Building a custom ingestion pipeline when integrated vectorization would suffice. Writing custom code to chunk, embed, and push documents to Azure AI Search is unnecessary for standard scenarios. Integrated vectorization handles chunking (Text Split skill), embedding (AzureOpenAIEmbedding skill), and indexing automatically. Use custom code only for non-standard document processing.

Interview Questions

Q: What is RAG and why is it preferred over fine-tuning for most applications? A: RAG (Retrieval Augmented Generation) retrieves relevant documents from a search index and includes them in the prompt so the model generates answers grounded in specific data. It is preferred over fine-tuning because: data can be updated instantly (re-index, no retraining), it costs less (no GPU time for training), it provides citations (you know which documents informed the answer), and it works with any model (no model-specific training). Fine-tuning is better for changing the model’s style or domain vocabulary, not for providing specific factual knowledge.

Q: What are the three search types in Azure AI Search and when would you use each? A: Keyword search (BM25) matches exact terms and is best for specific identifiers, product codes, and error messages. Vector search uses embeddings to find semantically similar content even with different wording and is best for natural language questions. Hybrid search combines both, using Reciprocal Rank Fusion to merge results, and is the recommended default because it catches both exact matches and paraphrases. Semantic ranking can be added on top of any search type to re-score results using a cross-encoder for deeper relevance understanding.

Q: What chunking strategy would you recommend for a RAG pipeline? A: Start with fixed-size chunking at 500 tokens with 100-token overlap using the Text Split skill. This works for most document types. For documents with clear section structure (legal contracts, technical manuals), use semantic or paragraph-based chunking to preserve conceptual units. For PDFs with tables and complex layouts, use the Document Layout skill. Always include overlap (10-25% of chunk size) to prevent information loss at boundaries. Evaluate retrieval quality and adjust chunk size based on whether the right content is being retrieved.

Q: What is integrated vectorization and why does Azure recommend it? A: Integrated vectorization automates the entire RAG ingestion pipeline within Azure AI Search. It combines an indexer (data source connection), skillset (Text Split for chunking, AzureOpenAIEmbedding for vectorization), and vectorizer (query-time embedding) into a single pipeline. Azure manages chunking, embedding, indexing, and change detection automatically. It is recommended because it eliminates custom ingestion code, ensures consistent processing, supports scheduled refresh, and reduces maintenance. Custom pipelines are only needed for non-standard processing requirements.

Q: What is the difference between On Your Data and a custom RAG pipeline? A: On Your Data is Azure OpenAI’s built-in RAG feature that connects an AI Search index directly to a model deployment with zero code. It handles query embedding, retrieval, and grounding automatically. A custom RAG pipeline (built with Prompt Flow or application code) offers full control over prompt templates, retrieval logic, multi-step orchestration, and response processing. Use On Your Data for prototyping and simple use cases. Use custom pipelines for production where you need control over prompts, multiple retrieval sources, custom reranking, or complex orchestration.

Q: How do you evaluate RAG quality? A: Evaluate two stages independently. Retrieval quality: measure recall (are relevant chunks retrieved?), precision (are retrieved chunks relevant?), and mean reciprocal rank (how high is the first relevant result?). Generation quality: use Foundry built-in evaluators for groundedness (is the answer based on retrieved chunks?), relevance (does it answer the question?), coherence (is it well-structured?), and fluency (is the language natural?). Create a test dataset of questions with expected answers and source documents. Run evaluations systematically and iterate on chunking, retrieval, and prompts based on low-scoring areas.

Q: Why must you use the same embedding model for indexing and querying? A: Each embedding model maps text into a specific vector space with specific dimensions and relationships. text-embedding-3-large produces 3072-dimensional vectors with its own learned relationships. text-embedding-3-small produces 1536-dimensional vectors in a different space. Mixing models means the query vector exists in a different space than the document vectors, so cosine similarity comparisons are meaningless. Always deploy one embedding model and use it consistently for both the ingestion pipeline (document embedding) and the query pipeline (question embedding).

Wrapping Up

RAG is the bridge between prompt engineering and production AI. The prompt tells the model how to behave. RAG tells it what to know. Together, they produce grounded, accurate, citeable answers from YOUR documents. The architecture is straightforward for data engineers: ingest documents into Azure AI Search (chunk, embed, index), retrieve relevant chunks at query time (hybrid search with semantic ranking), and generate grounded answers (GPT-4o with retrieved context).

The key decisions are chunking strategy (start at 500 tokens with overlap), embedding model (text-embedding-3-large for quality, small for cost), search type (hybrid + semantic for production), and evaluation (groundedness, relevance, coherence). Get these right, and your RAG pipeline delivers reliable answers. Get them wrong, and users get hallucinations with citations.

In the next post, we cover AI Agents — autonomous AI systems that can use tools, call functions, maintain memory, and orchestrate multi-step workflows using the Responses API.

Related posts: – Prompt Engineering – Microsoft Foundry Platform – AI-103 Study Guide – Security & Responsible AI – Data File Formats

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top