Context Engineering Research Report: File Types, RAG, and MCP Integration

Research Date: November 26, 2025
Scope: VS Code Copilot Ecosystem & AI Agent Systems
Status: Comprehensive Analysis Complete


Executive Summary

This report documents how different file types (.agent.md, .instructions.md, .prompt.md) and RAG (Retrieval-Augmented Generation) work together in context engineering for AI systems, particularly within the VS Code Copilot ecosystem. The research reveals a sophisticated multi-layered context injection system with clear hierarchies, integration patterns, and the emerging role of Model Context Protocol (MCP) in extending capabilities.


1. File Type Purposes and Structure

1.1 .agent.md Files

Purpose: Define specialized AI agents with specific roles, capabilities, and behavioral boundaries.

Structure:

````chatagent
---
name: AgentName                    # Unique identifier (PascalCase)
description: Brief description     # Used for routing/discovery
argument-hint: User input guide    # Instruction for users
tools: ['tool1', 'tool2', ...]    # Authorized capabilities
handoffs:                          # Inter-agent transitions
  - label: Button text
    agent: target_agent
    prompt: Handoff instructions
    send: true/false              # Auto-execute flag
---
{System Prompt}                    # Agent identity and role

<stopping_rules>
{Boundary conditions}              # Prevent scope creep
</stopping_rules>

<workflow>
{Step-by-step execution logic}     # Structured process
</workflow>

<guidelines>
{Specific operational rules}       # Style guides, standards
</guidelines>
````

Key Characteristics: - Single Responsibility: Each agent focuses on one task type (Planning, Implementation, Review) - Tool Authorization: Explicit permission model for capabilities - State Management: Handoffs enable complex multi-agent workflows - Behavioral Constraints: Stopping rules prevent agents from exceeding their mandate

Location: Typically in VS Code extension directories (e.g., extensions/github.copilot-chat-*/assets/agents/)

Real-World Example (Plan.agent.md):

---
name: Plan
description: Researches and outlines multi-step plans
tools: ['search', 'runSubagent', 'usages', 'problems', 'changes']
handoffs:
  - label: Start Implementation
    agent: agent
    prompt: Start implementation
---
You are a PLANNING AGENT, NOT an implementation agent.

<stopping_rules>
STOP IMMEDIATELY if you consider starting implementation.
Plans describe steps for the USER to execute later.
</stopping_rules>

<workflow>
## 1. Context gathering:
MANDATORY: Run #tool:runSubagent for autonomous research.

## 2. Present plan:
Follow <plan_style_guide> and pause for user feedback.

## 3. Handle feedback:
Restart <workflow> to refine plan based on new information.
</workflow>


1.2 .instructions.md Files

Purpose: Provide targeted, context-aware guidelines that automatically apply to specific files or patterns.

Structure:

---
description: Optional hover text in Chat view
applyTo: '**/*.cs'  # Glob pattern for file matching
---

# Instructions Body (Markdown)

- Write clear comments for each function
- Use PascalCase for class names
- Add newline before opening braces

Characteristics: - Automatic Detection: VS Code Copilot detects relevant files based on applyTo patterns - Scoped Application: Different instructions for different file types/domains - Silent Operation: Instructions inject into context without explicit UI indication - Reference Tracking: Used instructions appear in Chat response references

Standard Locations: 1. Global Instructions: .github/copilot-instructions.md (applies to all prompts) 2. Targeted Instructions: .github/instructions/*.instructions.md (scoped by applyTo)

Activation: - Requires opt-in via VS Code settings: github.copilot.chat.codeGeneration.useInstructionFiles - Frontmatter applyTo field uses glob patterns for file matching

Example - C# Coding Standards:

---
applyTo: '**/*.cs'
description: C# coding conventions for this project
---

## Naming Conventions
- Use PascalCase for public members
- Use camelCase for private fields with `_` prefix

## Code Organization
- Group members: fields → constructors → properties → methods
- Add XML documentation for public APIs


1.3 .prompt.md Files

Purpose: Reusable, shareable prompt templates with embedded references for frequently used queries.

Structure:

Analyze the #file:src/services/OrderService.cs for:
1. Error handling patterns
2. Dependency injection usage
3. Async/await best practices

Compare with #file:src/services/BaseService.cs and suggest improvements.

Characteristics: - Reference Embedding: Use #file:, #symbol:, #prompt: syntax for dynamic context - Composability: Prompt files can reference other prompt files - Shareability: Stored in version control for team collaboration - IDE Integration: Autocomplete via #prompt: prefix in chat

Standard Location: .github/prompts/*.prompt.md

Usage Pattern: 1. Create in chat: Write prompt → Copy to .prompt.md file 2. Invoke in chat: Type #prompt: → Select from list → Add details → Send

Example - API Review Template:

<!-- File: .github/prompts/api-review.prompt.md -->

Review #selection for RESTful API compliance:

## Check:
- HTTP method semantics (GET/POST/PUT/DELETE)
- Status code appropriateness
- Request/response validation
- Error handling consistency

## Compare with:
- #file:docs/api-guidelines.md
- Existing patterns in #file:src/controllers/

Suggest improvements with code examples.


2. Context Injection Pipeline

2.1 Context Hierarchy (Priority Order)

┌─────────────────────────────────────────────────────────────┐
│ HIGHEST PRIORITY: Explicit User Context                     │
│ - #file: references                                         │
│ - #symbol: references                                       │
│ - Selected code                                             │
│ - Attached images/errors                                    │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ IMPLICIT CONTEXT: Current Environment                       │
│ - Active editor file                                        │
│ - Open tabs                                                 │
│ - Chat history (current session)                            │
│ - Cursor position/surrounding code                          │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ INSTRUCTIONS: Scoped Rules                                  │
│ - Matched .instructions.md (via applyTo patterns)           │
│ - Global copilot-instructions.md                            │
│ - Agent-specific guidelines                                 │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ SEMANTIC CONTEXT: RAG Retrieval                             │
│ - Vector search on codebase index                           │
│ - Semantic similarity matches                               │
│ - Cross-file relationship analysis                          │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ SYSTEM PROMPT: Base Capabilities                            │
│ - Model capabilities (GPT-4, Claude, etc.)                  │
│ - Base programming knowledge                                │
│ - General best practices                                    │
└─────────────────────────────────────────────────────────────┘

2.2 Override Rules

Principle: More specific context overrides general context.

  1. User Explicit > All: Direct #file: references trump everything
  2. Instructions > System Defaults: Custom .instructions.md override base rules
  3. Agent Guidelines > Generic Instructions: Agent-specific rules take precedence
  4. Local > Remote: Workspace instructions override global settings

Example Scenario:

User prompt: "Optimize this function"
+ Selected code: function calculateTotal() {...}
+ .instructions.md: "Use LINQ for collections"
+ System knowledge: "Use loops for performance"

Result: Applies LINQ (per instructions) > generic loop optimization


3. RAG (Retrieval-Augmented Generation) Mechanisms

3.1 Codebase Indexing

Remote Indexing (GitHub/Azure DevOps): - Vector embeddings capture semantic relationships - Cloud-based index for fast cross-repository search - Automatic updates on code changes

Local Indexing (Other Sources): - Built on-demand for non-cloud repositories - Stored in VS Code workspace cache - Refreshed periodically or on file changes

Index Components:

Code Embeddings:
├── Function signatures
├── Class definitions
├── Comment/documentation text
├── Variable names
├── Import/dependency relationships
└── Code structure patterns

3.2 Semantic Search Process

Trigger Conditions: - Copilot determines insufficient context from prompt alone - User asks about unfamiliar concepts - Cross-file analysis required

Search Flow:

1. Query Vectorization
   User prompt → Vector embedding (e.g., 1536 dimensions)

2. Similarity Computation
   Compare query vector with code embedding vectors
   Cosine similarity: sim(A,B) = (A·B) / (|A||B|)

3. Ranking & Retrieval
   Top-K files with highest semantic similarity
   Typical K = 5-20 files depending on context window

4. Context Augmentation
   Retrieved files added to prompt context
   Prioritized by relevance score

Semantic vs. Traditional Search:

Aspect Semantic Search Traditional (grep)
Match Type Meaning-based Keyword exact
Example Query "authentication logic" grep "auth"
Finds Login, JWT validation, OAuth Only literal "auth"
Ranking Relevance score None (binary match)

3.3 RAG Integration with Agent Workflow

Agent Tool Interaction:

Agent receives prompt
    ↓
Checks available tools: ['semantic_search', 'read_file', 'grep_search']
    ↓
Decision logic:
  - Concept-based query → semantic_search
  - Exact pattern search → grep_search
  - Known file location → read_file
    ↓
RAG retrieves relevant code
    ↓
Agent synthesizes response with retrieved context
    ↓
Cites sources in response (file paths, line numbers)

Example from Plan.agent.md:

<workflow>
## 1. Context gathering and research:
MANDATORY: Run #tool:runSubagent tool, instructing it to:
  - Use semantic_search for conceptual understanding
  - Follow with grep_search for specific patterns
  - Read identified files with targeted queries

Stop research at 80% confidence threshold.
</workflow>


4. Model Context Protocol (MCP) Integration

4.1 MCP Architecture

Purpose: Standardized protocol for connecting AI agents to external tools, resources, and data sources.

Core Components:

┌─────────────────────────────────────────────────────────┐
│                    AI Agent (Copilot)                   │
└────────────────────────┬────────────────────────────────┘
                         │
                         │ MCP Client
                         ↓
┌─────────────────────────────────────────────────────────┐
│              Model Context Protocol (MCP)               │
│  ┌─────────────┬──────────────┬─────────────────────┐  │
│  │   Tools     │  Resources   │      Prompts        │  │
│  │ (Actions)   │ (Data Access)│  (Templates)        │  │
│  └─────────────┴──────────────┴─────────────────────┘  │
└────────────────────────┬────────────────────────────────┘
                         │
        ┌────────────────┼────────────────┐
        │                │                │
        ↓                ↓                ↓
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ MCP Server 1 │ │ MCP Server 2 │ │ MCP Server N │
│  (GitHub)    │ │ (Azure Docs) │ │  (Dataverse) │
└──────────────┘ └──────────────┘ └──────────────┘

MCP Server Capabilities:

  1. Tools: Functions the agent can invoke (e.g., get_issue, query_docs, deploy_resource)
  2. Resources: File-like data for context (e.g., API responses, documentation)
  3. Prompts: Predefined templates for common tasks (e.g., PR review, commit message)

4.2 MCP Configuration

Configuration File: .mcp.json (workspace or user level)

Example Structure:

{
  "mcpServers": {
    "github": {
      "command": "mcp-server-github",
      "args": ["--token", "${GITHUB_TOKEN}"],
      "disabled": false
    },
    "microsoft-docs": {
      "type": "sse",
      "url": "https://mcp-docs.microsoft.com",
      "disabled": false
    }
  }
}

Authentication: - OAuth integration for remote servers - Keychain storage for credentials - Per-server authorization management

4.3 MCP in Agent Workflows

Tool Discovery:

# In .agent.md
tools: [
  'search',                              # Built-in VS Code tool
  'github/github-mcp-server/get_issue',  # MCP tool from GitHub server
  'microsoft-docs/docs_search'           # MCP tool from Microsoft server
]

Dynamic Tool Invocation:

Agent determines: "Need to search Microsoft docs"
    ↓
Checks MCP servers for relevant tools
    ↓
Finds: microsoft-docs/docs_search
    ↓
Requests user approval (first use)
    ↓
Invokes tool with parameters
    ↓
Receives structured response
    ↓
Integrates into agent response

MCP Prompt Templates:

User types: #mcp:github/pr-review
    ↓
MCP server provides template:
"""
Review #pr:123 for:
- Code quality issues
- Security vulnerabilities
- Breaking changes

Format: {template_structure}
"""
    ↓
User fills in PR number → Agent executes

4.4 MCP vs. Built-in Tools

Aspect Built-in Tools MCP Tools
Scope VS Code workspace External systems
Examples read_file, grep_search azure_deploy, github_get_issue
Auth Local permissions OAuth/API keys
Discovery Static list Dynamic registration
Updates VS Code releases MCP server updates

4.5 MCP Context Management

Instructions for MCP Tools:

Stored in .instructions.md:

---
applyTo: '**'
---

You have access to MCP tools:
- `microsoft_docs_search`: Query official Microsoft documentation
- `github/get_issue`: Retrieve GitHub issue details

For questions about Microsoft technologies, ALWAYS use microsoft_docs_search
before relying on training data.

Context File Approach (Alternative):

<!-- scp-mcp-context.md -->
# Security Copilot MCP Tools Context

When building agents, use these MCP tools:
1. `get_agent_skills`: List available Security Copilot skills
2. `generate_agent_yaml`: Create agent definition from description
3. `deploy_agent`: Deploy agent to Security Copilot

Workflow:
1. Describe intent → generate_agent_yaml
2. Review YAML → iterate with user
3. Finalize → deploy_agent

Context Injection Methods: 1. Global Instructions: Add to copilot-instructions.md 2. Explicit Reference: Use #file:scp-mcp-context.md in prompts 3. Agent Guidelines: Include in agent's <workflow> section


5. Priority Hierarchy and Override Rules

5.1 Context Source Priority Matrix

Priority  Source Type              Scope         Override Power
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1         User Explicit Context     Single prompt    Overrides all
          (#file, #symbol, selection)

2         Agent Stopping Rules      Agent session    Overrides workflows
          (behavioral boundaries)

3         Targeted Instructions     File pattern     Overrides global
          (.instructions.md + applyTo)

4         Global Instructions       Workspace        Overrides system
          (copilot-instructions.md)

5         Agent Guidelines          Agent instance   Overrides generic
          (<workflow>, <guidelines>)

6         MCP Prompt Templates      Tool-specific    Overrides generic
          (from MCP servers)

7         RAG Retrieved Context     Semantic match   Augments knowledge
          (vector search results)

8         Chat History              Session          Provides continuity
          (previous conversation)

9         System Prompt             Model base       Foundational rules
          (model capabilities)

5.2 Conflict Resolution Examples

Scenario 1: Coding Style Conflict

System: "Use snake_case for Python"
Global .instructions.md: "Use camelCase for consistency"
File-specific .instructions.md (applyTo: '**/*.py'): "Use snake_case per PEP8"

Result: snake_case (file-specific wins over global)

Scenario 2: Agent vs. User Intent

Agent stopping rule: "STOP if attempting file edits"
User prompt: "Edit this file to add logging"

Result: Agent refuses and suggests handoff to implementation agent
        (stopping rules protect agent integrity)

Scenario 3: RAG vs. Instructions

RAG retrieves: Old code pattern (deprecated approach)
.instructions.md: "Use new async/await pattern"

Result: Copilot suggests async/await pattern with reference to instructions
        (instructions guide interpretation of retrieved code)

5.3 Context Merging Strategy

Additive Merging (Default): - Multiple .instructions.md files combine if applyTo patterns overlap - All matching instructions added to context

Replacement Merging (Explicit): - Agent guidelines override generic instructions - Stopping rules replace permissive defaults

Example Merged Context:

FINAL CONTEXT for `src/services/PaymentService.cs`:

[System Prompt]
You are GitHub Copilot, an AI programming assistant...

[Global Instructions from copilot-instructions.md]
- Follow SOLID principles
- Write unit tests for business logic

[C# Instructions from .github/instructions/csharp.instructions.md]
- Use PascalCase for public members
- Add XML documentation

[Agent Guidelines from CodeReviewer.agent.md]
<stopping_rules>
STOP if attempting to fix code—only document issues.
</stopping_rules>

[RAG Retrieved Context]
File: src/services/OrderService.cs (similar payment logic)
[...code snippet...]

[User Query]
"Review the ProcessPayment method"


6. Real-World Integration Patterns

6.1 Multi-Agent Workflow with Context Sharing

Scenario: Complex feature implementation

┌─────────────────────────────────────────────────────────────┐
│ 1. Plan Agent                                               │
│    - Tools: semantic_search, runSubagent                    │
│    - Context: .instructions.md, RAG                         │
│    - Output: Structured plan in plan.prompt.md             │
└────────────────────┬────────────────────────────────────────┘
                     │ Handoff: #prompt:plan-implementation.prompt.md
                     ↓
┌─────────────────────────────────────────────────────────────┐
│ 2. Implementation Agent                                     │
│    - Tools: replace_string_in_file, run_in_terminal        │
│    - Context: Plan + .instructions.md + RAG                │
│    - Output: Code changes + test files                     │
└────────────────────┬────────────────────────────────────────┘
                     │ Handoff: "Run tests"
                     ↓
┌─────────────────────────────────────────────────────────────┐
│ 3. Test Agent                                               │
│    - Tools: run_in_terminal, get_errors                    │
│    - Context: Test results + error logs                    │
│    - Output: Test report + fix suggestions                 │
└─────────────────────────────────────────────────────────────┘

Context Continuity Mechanisms: 1. Prompt Files: Plan saved as .prompt.md, referenced in handoff 2. Chat History: Previous agent responses available to next agent 3. File References: Modified files tracked and auto-referenced

6.2 RAG-Powered Code Search Pattern

Use Case: "Find all authentication implementations"

# Pseudo-code for agent logic

def search_authentication():
    # 1. Semantic Search (RAG)
    semantic_results = semantic_search(
        query="authentication login user validation",
        max_results=10
    )

    # 2. Validate with grep_search
    confirmed_files = []
    for file in semantic_results:
        if grep_search(
            query="authenticate|login|verify",
            file=file,
            isRegexp=True
        ):
            confirmed_files.append(file)

    # 3. Read relevant files
    code_snippets = []
    for file in confirmed_files[:5]:  # Limit token usage
        content = read_file(file)
        code_snippets.append({
            'file': file,
            'content': extract_auth_functions(content)
        })

    # 4. Synthesize response
    return f"""
    Found authentication in {len(confirmed_files)} files:

    {format_findings(code_snippets)}

    Common patterns:
    - JWT token validation
    - Password hashing with bcrypt
    - OAuth2 integration

    References: {list_file_paths(confirmed_files)}
    """

6.3 MCP-Enhanced Documentation Workflow

Use Case: Answer question using latest Azure docs

User: "How do I create an Azure Storage account with CLI?"
    ↓
Agent checks MCP tools: microsoft_docs_search available
    ↓
Agent calls: microsoft_docs_search(query="Azure Storage account CLI create")
    ↓
MCP returns:
{
  "results": [
    {
      "title": "Quickstart: Create storage account",
      "url": "https://learn.microsoft.com/azure/storage/...",
      "excerpt": "Use az storage account create to..."
    }
  ]
}
    ↓
Agent fetches full content: microsoft_docs_fetch(url=...)
    ↓
Agent synthesizes:
"""
To create an Azure Storage account using CLI:

```bash
az storage account create \
  --name mystorageaccount \
  --resource-group myresourcegroup \
  --location eastus \
  --sku Standard_LRS

Reference: Official Azure Documentation """

### 6.4 Context-Aware File Generation

**Use Case:** Generate code matching project conventions
User: "Create a new service class for Orders" ↓ Context gathered: ├── .instructions.md (applyTo: '**/*.cs') │ └── "Use dependency injection, add XML docs" ├── semantic_search("service class pattern") │ └── Finds: UserService.cs, ProductService.cs ├── read_file(UserService.cs) │ └── Extracts: Constructor pattern, interface usage └── Global instructions └── "Follow SOLID principles" ↓ Agent generates: """
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;

namespace MyApp.Services
{
    /// <summary>
    /// Service for managing order operations.
    /// </summary>
    public interface IOrderService
    {
        Task<Order> GetOrderAsync(int orderId);
    }

    public class OrderService : IOrderService
    {
        private readonly ILogger<OrderService> _logger;
        private readonly IOrderRepository _repository;

        public OrderService(
            ILogger<OrderService> logger,
            IOrderRepository repository)
        {
            _logger = logger;
            _repository = repository;
        }

        // Implementation follows...
    }
}

Generated based on: - Dependency injection pattern from UserService.cs - XML documentation from .instructions.md - SOLID principles from global instructions """

---

## 7. Code Examples and Configuration Patterns

### 7.1 Complete .instructions.md Example

```markdown
<!-- .github/instructions/typescript.instructions.md -->
---
description: TypeScript coding standards for React components
applyTo: 'src/**/*.tsx'
---

# React Component Guidelines

## Naming Conventions
- Use PascalCase for component files: `UserProfile.tsx`
- Use camelCase for utility files: `formatDate.ts`

## Component Structure
```typescript
// 1. Imports (grouped: React → third-party → local)
import React, { useState, useEffect } from 'react';
import { Button } from '@mui/material';
import { useAuth } from '@/hooks/useAuth';

// 2. Type definitions
interface UserProfileProps {
  userId: string;
  onUpdate?: (user: User) => void;
}

// 3. Component
export const UserProfile: React.FC<UserProfileProps> = ({ userId, onUpdate }) => {
  // 3.1 Hooks (useState → useEffect → custom hooks)
  const [user, setUser] = useState<User | null>(null);
  const { isAuthenticated } = useAuth();

  // 3.2 Event handlers
  const handleUpdate = () => { /* ... */ };

  // 3.3 Render
  return <div>{/* ... */}</div>;
};

State Management

  • Use useState for component-local state
  • Use context for cross-component state
  • Never mutate state directly—use spread operators

Testing Requirements

  • Every component must have a corresponding .test.tsx file
  • Test file structure: Arrange → Act → Assert
  • Use Testing Library queries: getByRole, getByText
    ### 7.2 .prompt.md Template Examples
    
    ```markdown
    <!-- .github/prompts/feature-implementation.prompt.md -->
    
    Implement a new feature: [FEATURE_NAME]
    
    ## Context
    - Related files: #file:[CORE_FILE_PATH]
    - Similar features: #file:[REFERENCE_FEATURE]
    - API docs: #file:docs/api-guidelines.md
    
    ## Requirements
    1. Follow patterns from #file:[REFERENCE_FEATURE]
    2. Add unit tests in `tests/` directory
    3. Update documentation in #file:README.md
    
    ## Checklist
    - [ ] Implementation matches coding standards in #file:.github/instructions/
    - [ ] Error handling covers edge cases
    - [ ] Tests achieve >80% coverage
    - [ ] No breaking changes to existing APIs
    
    Generate:
    1. Implementation files
    2. Test files
    3. Documentation updates
    

<!-- .github/prompts/bug-investigation.prompt.md -->

Investigate bug: [BUG_DESCRIPTION]

## Error Context
[PASTE ERROR LOG]
## Investigation Steps
1. Search for error message: Use semantic_search
2. Check recent changes: Review #changes
3. Examine test failures: Check #testFailure
4. Review related code: #file:[AFFECTED_FILE]

## Analysis
- Root cause hypothesis
- Affected components
- Reproduction steps
- Suggested fix approach

## Next Steps
- [ ] Verify fix in isolation
- [ ] Add regression test
- [ ] Update error handling

7.3 MCP Configuration Examples

// .mcp.json (User-level MCP configuration)
{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}"
      },
      "disabled": false
    },

    "microsoft-docs": {
      "type": "sse",
      "url": "https://mcp-docs-server.microsoft.com/api/mcp",
      "headers": {
        "Authorization": "Bearer ${MS_DOCS_TOKEN}"
      },
      "disabled": false
    },

    "azure-resources": {
      "command": "azure-mcp-server",
      "args": ["--subscription", "${AZURE_SUBSCRIPTION_ID}"],
      "disabled": false
    }
  },

  "mcpToolApprovals": {
    "github/get_issue": "always",
    "microsoft-docs/docs_search": "always",
    "azure-resources/deploy": "prompt"
  }
}

7.4 Agent Definition with Full Context Integration

<!-- InfoGatherer.agent.md -->
````chatagent
---
name: InfoGatherer
description: Researches codebases and synthesizes findings into reports
argument-hint: Describe what information to gather (e.g., "Find all API endpoints")
tools: [
  'semantic_search',
  'grep_search',
  'read_file',
  'list_dir',
  'fetch_webpage',
  'github/github-mcp-server/get_issue',
  'microsoft-docs/docs_search'
]
handoffs:
  - label: Generate Documentation
    agent: agent
    prompt: '#createFile Create a report from the gathered information'
---
You are an INFORMATION GATHERING AGENT specializing in codebase research and analysis.

Your role is to autonomously search, read, and synthesize information from multiple sources, NOT to implement changes.

<stopping_rules>
STOP IMMEDIATELY if you attempt to edit files, run terminal commands, or make code changes.

If you identify areas needing changes, DOCUMENT them for handoff—do not implement.
</stopping_rules>

<workflow>
## 1. Query Analysis
- Parse user request to identify key concepts
- Determine information sources: codebase, docs, issues
- Set confidence threshold (target: 90%)

## 2. Multi-Source Research
Execute searches in parallel:

### A. Codebase Search (RAG)
1. semantic_search(query, maxResults=15) 2. Validate top results with grep_search 3. read_file for confirmed matches
### B. Documentation Search (MCP)
If query relates to frameworks/technologies:
1. Check MCP tools: microsoft_docs_search available? 2. Call MCP tool with refined query 3. Fetch full docs if needed
### C. Issue Tracker (MCP)
If query involves bugs/features:
1. Use github/get_issue for related issues 2. Extract context from issue descriptions 3. Cross-reference with code findings
## 3. Synthesis
Organize findings:
```markdown
## Summary (2-3 sentences)

## Detailed Findings
### Source 1: [Codebase]
- Location: #file:path/to/file.ts:123
- Key points: ...

### Source 2: [Documentation]
- Reference: [Title](url)
- Key points: ...

### Source 3: [Issues]
- Issue #123: ...
- Related: #456, #789

## Patterns Identified
- Common approach: ...
- Variations: ...
- Edge cases: ...

## Recommendations
1. [Actionable suggestion]
2. [Alternative approach]

## References
- [List of all sources consulted]

MANDATORY: Pause for user feedback before handoff.

RAG Usage

  • Start broad (semantic_search) → narrow down (grep_search)
  • Read files selectively—only confirmed matches
  • Track token budget: stop at 80% if approaching limit

MCP Integration

  • Query external docs for framework-specific questions
  • Use issue data for historical context
  • Combine MCP results with local findings

Instructions Awareness

  • Check for .instructions.md matching searched files
  • Note project-specific patterns to inform synthesis
  • Reference instructions in recommendations

- Use markdown with clear hierarchy (##, ###) - Link to files: FileName - Quote code with language tags: typescript - Cite sources at end of sections - Keep summary under 100 words </output_format> ````


8. Technical Implementation Details

8.1 Vector Embedding Process (RAG)

Model: Typically text-embedding-ada-002 or similar (1536 dimensions)

Indexing Pipeline: ```python

Conceptual implementation

def index_codebase(repo_path): code_chunks = []

for file in walk_directory(repo_path):
    if is_code_file(file):
        # Parse file into semantic chunks
        chunks = split_into_chunks(
            file_content=read_file(file),
            strategy='ast-aware',  # Respect function/class boundaries
            max_tokens=512
        )

        for chunk in chunks:
            # Generate embedding
            embedding = embed_model.encode(chunk.text)

            code_chunks.append({
                'file': file.path,
                'start_line': chunk.start_line,
                'end_line': chunk.end_line,
                'content': chunk.text,
                'embedding': embedding,
                'metadata': {
                    'type': chunk.type,  # function/class/module
                    'name': chunk.name,
                    'language': file.language
                }
            })

# Store in vector database (e.g., FAISS, Pinecone)
vector_db.upsert(code_chunks)
return vector_db

def semantic_search(query, max_results=10): # Embed query query_embedding = embed_model.encode(query)

# Vector similarity search
results = vector_db.search(
    query_vector=query_embedding,
    top_k=max_results,
    metric='cosine'
)

# Re-rank by additional signals
results = rerank(results, signals=[
    'recency',          # Newer code prioritized
    'centrality',       # Core logic prioritized
    'quality_score'     # Well-documented code
])

return results

```

Chunk Splitting Strategy: ``` File: src/services/UserService.ts (2000 lines) ↓ Chunking (AST-aware): ├── Chunk 1: Import statements + class declaration (lines 1-50) ├── Chunk 2: getUserById method + dependencies (lines 51-150) ├── Chunk 3: createUser method + validation (lines 151-300) ├── ... └── Chunk N: Helper functions (lines 1900-2000)

Each chunk: - Self-contained (includes necessary context) - Respects syntax boundaries (complete functions) - Max 512 tokens to fit embedding model - Overlaps by 50 tokens to preserve continuity ```

8.2 Context Window Management

Token Budget Allocation: ``` Total Context Window: 128,000 tokens (GPT-4 Turbo)

Allocation: ├── System Prompt: ~2,000 tokens ├── Instructions: ~5,000 tokens │ ├── Global .instructions.md │ └── Targeted .instructions.md (multiple) ├── Agent Definition: ~3,000 tokens │ ├── Frontmatter + tools │ ├── Workflow + guidelines │ └── Stopping rules ├── RAG Retrieved Context: ~30,000 tokens │ ├── 10-20 code files │ └── Ranked by relevance ├── Chat History: ~10,000 tokens │ └── Last 5-10 exchanges ├── User Context: ~20,000 tokens │ ├── Explicit references (#file, #symbol) │ └── Selected code └── Response Generation: ~58,000 tokens (remaining) ```

Overflow Handling: 1. Priority Pruning: Remove lower-priority context first - Chat history older than 5 messages - RAG results below relevance threshold 2. Summarization: Compress lengthy files before adding 3. Chunked Reading: Read files in sections with offset/limit 4. User Notification: "Context too large, please narrow scope"

8.3 Instruction File Matching Algorithm

```python

Conceptual implementation of applyTo matching

def find_applicable_instructions(file_path, workspace_root): applicable = []

# 1. Load all .instructions.md files
instruction_files = glob(
    f"{workspace_root}/.github/instructions/*.instructions.md"
)

for inst_file in instruction_files:
    frontmatter = parse_yaml_frontmatter(inst_file)
    apply_to_pattern = frontmatter.get('applyTo', '**')

    # 2. Match glob pattern
    if glob_match(file_path, apply_to_pattern, workspace_root):
        applicable.append({
            'file': inst_file,
            'content': read_file(inst_file),
            'description': frontmatter.get('description', ''),
            'priority': calculate_priority(apply_to_pattern)
        })

# 3. Sort by specificity (most specific pattern wins)
# e.g., 'src/api/**/*.ts' > 'src/**/*.ts' > '**/*.ts'
applicable.sort(key=lambda x: x['priority'], reverse=True)

return applicable

def calculate_priority(pattern): """More specific patterns get higher priority.""" specificity = 0 specificity += pattern.count('/') # Deeper paths specificity += len(pattern.replace('', '')) # Non-wildcard chars specificity -= pattern.count('*') # Penalize recursive wildcards return specificity

Example usage

file_path = "src/api/controllers/UserController.ts" instructions = find_applicable_instructions(file_path, workspace_root)

Result (sorted by priority):

[ { file: 'src-api-controllers.instructions.md', priority: 25 }, # Most specific { file: 'typescript.instructions.md', priority: 10 }, { file: 'copilot-instructions.md', priority: 0 } # Global ] ```


9. Best Practices and Recommendations

9.1 For File-Based Context Engineering

✅ DO: - Specific Patterns: Use precise applyTo globs (e.g., src/api/**/*.ts vs. **/*.ts) - Scoped Instructions: Create separate .instructions.md for distinct domains (API, UI, tests) - Version Control: Store all context files in repo for team consistency - Clear Descriptions: Add description frontmatter for discoverability - Test Instructions: Verify instructions apply by checking Chat response references

❌ DON'T: - Overlap Carelessly: Avoid conflicting instructions in overlapping patterns - Overload Global: Keep copilot-instructions.md concise (<2000 words) - Hardcode Secrets: Never put tokens/keys in .mcp.json (use env vars) - Ignore Hierarchy: Remember user explicit context always wins - Forget Updates: Review and prune outdated instructions regularly

9.2 For RAG Optimization

Query Formulation: ``` ❌ Poor: "Find login code" ✅ Better: "Find user authentication and session management logic" ✅ Best: "Find implementations of JWT token validation and refresh logic in API routes"

Rationale: More specific queries → better semantic matches ```

Search Strategy: ```python

Recommended search sequence

1. Broad semantic search (conceptual understanding)

semantic_search("authentication logic", max_results=20)

2. Filter with grep (validation)

grep_search("authenticate|login|jwt", isRegexp=True)

3. Targeted reads (deep inspection)

read_file("src/auth/AuthService.ts") ```

Token Efficiency: - Request summaries instead of full files when possible - Use read_file with offset and limit for large files - Parallelize independent searches - Cache frequently accessed context

9.3 For Agent Design

Single Responsibility: ❌ Bad: "UniversalAgent" with 50 tools doing everything ✅ Good: Specialized agents - PlanAgent: Research + outline (read-only tools) - ImplementAgent: Code changes (edit + terminal tools) - ReviewAgent: Analysis + feedback (read + error tools)

Stopping Rules Clarity: ``` ❌ Weak: "Don't do things outside your scope" ✅ Strong: STOP IMMEDIATELY if you attempt to: 1. Edit files (use handoff to ImplementAgent) 2. Run terminal commands 3. Start implementation without approved plan

If you catch yourself writing code, STOP and present a plan instead. ```

Workflow Explicitness: ``` ❌ Vague: "Gather context and respond" ✅ Explicit:

1. Context Gathering (MANDATORY)

  • Run semantic_search for concepts
  • Run grep_search for exact matches
  • Read top 5 files

MANDATORY: Achieve 80% confidence or ask clarifying questions.

2. Synthesis

  • Organize findings by theme
  • Identify patterns and anti-patterns

3. User Feedback (MANDATORY)

Present findings and WAIT for feedback. DO NOT proceed to implementation. ```

9.4 For MCP Integration

Tool Selection: ```yaml

Principle: Least privilege

❌ Grant all tools by default ✅ Grant tools based on agent role:

PlanAgent: tools: ['semantic_search', 'github/get_issue'] # Read-only

ImplementAgent: tools: ['replace_string', 'run_terminal'] # Write access

DocAgent: tools: ['microsoft-docs/search', 'fetch_webpage'] # External docs ```

Authentication Security: ```json // ❌ DON'T: Hardcode tokens { "env": { "GITHUB_TOKEN": "ghp_abc123..." } }

// ✅ DO: Use environment variables { "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } }

// Set in shell: // export GITHUB_TOKEN=ghp_abc123... ```

Error Handling: ```markdown

If MCP tool unavailable:

  1. Log warning: "MCP server not configured"
  2. Fallback to built-in tools (e.g., fetch_webpage)
  3. Inform user of limitation

If MCP call fails:

  1. Retry once with exponential backoff
  2. If still fails, document error and continue
  3. Suggest manual alternative

Example: "Unable to fetch latest Azure docs via MCP. Providing answer based on training data (may be outdated). Consider checking [URL] manually." ```


10.1 Emerging Patterns

1. Agentic Workflows (Multi-Agent Systems) - Specialized agents coordinate via handoffs - Context passed through .prompt.md files - Workflow orchestration via "conductor" agents

2. Dynamic Tool Discovery - MCP servers dynamically register new tools - Agents query capabilities at runtime - User approvals cached per-session

3. Context Personalization - Learn user preferences over time - Store in "Copilot memories" (.editorconfig, CONTRIBUTING.md) - Auto-generate project-specific instructions

4. Hybrid RAG + LLM Reasoning - RAG provides facts, LLM provides reasoning - Multi-hop reasoning across retrieved documents - Agentic frameworks (LangChain, Semantic Kernel)

10.2 MCP Ecosystem Growth

Current State: - ~50 community MCP servers (GitHub, Slack, databases) - Standardization in progress (v1.0 spec)

Near Future (2025-2026): - Enterprise MCP marketplaces - Pre-certified security-reviewed servers - Native OS integration (Windows File Explorer MCP) - Cross-platform agent frameworks (Microsoft Agent Framework)

10.3 Context Engineering Best Practices (Next Gen)

1. Context Tiering Tier 1: Immediate (current file, selection) → Always included Tier 2: Proximate (open files, recent history) → Included if space Tier 3: Repository (semantic search) → Included on-demand Tier 4: External (docs, issues via MCP) → Included with approval

2. Proactive Context Injection - Copilot predicts needed context before user asks - Pre-fetches related files based on intent detection - Suggests additional references ("Also consider: #file:...")

3. Context Explainability - Show context sources in UI ("Based on: .instructions.md + 3 files via RAG") - Allow users to exclude specific sources - Provide "why this context?" explanations


11. Conclusion

Key Findings Summary

  1. File Types Serve Distinct Roles:
  2. .agent.md: Define agent behavior and tool authorization
  3. .instructions.md: Provide scoped, automatic context injection
  4. .prompt.md: Enable reusable, composable query templates

  5. Context Hierarchy is Strict:

  6. User explicit context (selections, #file:) overrides all
  7. Instructions override system defaults
  8. RAG augments but doesn't override explicit guidance

  9. RAG is Semantic Search + Context Augmentation:

  10. Vector embeddings capture code meaning
  11. Retrieval ranked by relevance, not keyword matching
  12. Integrated into agent workflows via tools

  13. MCP Bridges AI Agents and External Systems:

  14. Standardized protocol for tools, resources, prompts
  15. Dynamic discovery and invocation
  16. Growing ecosystem of servers (GitHub, Azure, Dataverse)

  17. Priority System Ensures Correctness:

  18. More specific context wins (file-level > global)
  19. Behavioral boundaries (stopping rules) protect integrity
  20. Conflicts resolved by source hierarchy

Practical Takeaways

For Developers: - Start with global .instructions.md, specialize with targeted files - Design agents with single responsibilities and clear handoffs - Use RAG for discovery, explicit references for precision - Integrate MCP for external data/tools

For Teams: - Version-control all context files for consistency - Document agent roles and handoff patterns - Review and prune instructions regularly - Share prompt templates for common workflows

For Organizations: - Establish MCP governance (approved servers, security reviews) - Train teams on context engineering best practices - Monitor token usage and optimize context efficiency - Build internal MCP servers for proprietary systems


12. References and Resources

Official Documentation

MCP Resources

Code Samples

Community


Report Version: 1.0
Last Updated: November 26, 2025
Next Review: Q1 2026 (track MCP v1.0 release)