Context Engineering Research Report: File Types, RAG, and MCP Integration
Research Date: November 26, 2025
Scope: VS Code Copilot Ecosystem & AI Agent Systems
Status: Comprehensive Analysis Complete
Executive Summary
This report documents how different file types (.agent.md, .instructions.md, .prompt.md) and RAG (Retrieval-Augmented Generation) work together in context engineering for AI systems, particularly within the VS Code Copilot ecosystem. The research reveals a sophisticated multi-layered context injection system with clear hierarchies, integration patterns, and the emerging role of Model Context Protocol (MCP) in extending capabilities.
1. File Type Purposes and Structure
1.1 .agent.md Files
Purpose: Define specialized AI agents with specific roles, capabilities, and behavioral boundaries.
Structure:
````chatagent
---
name: AgentName # Unique identifier (PascalCase)
description: Brief description # Used for routing/discovery
argument-hint: User input guide # Instruction for users
tools: ['tool1', 'tool2', ...] # Authorized capabilities
handoffs: # Inter-agent transitions
- label: Button text
agent: target_agent
prompt: Handoff instructions
send: true/false # Auto-execute flag
---
{System Prompt} # Agent identity and role
<stopping_rules>
{Boundary conditions} # Prevent scope creep
</stopping_rules>
<workflow>
{Step-by-step execution logic} # Structured process
</workflow>
<guidelines>
{Specific operational rules} # Style guides, standards
</guidelines>
````
Key Characteristics: - Single Responsibility: Each agent focuses on one task type (Planning, Implementation, Review) - Tool Authorization: Explicit permission model for capabilities - State Management: Handoffs enable complex multi-agent workflows - Behavioral Constraints: Stopping rules prevent agents from exceeding their mandate
Location: Typically in VS Code extension directories (e.g., extensions/github.copilot-chat-*/assets/agents/)
Real-World Example (Plan.agent.md):
---
name: Plan
description: Researches and outlines multi-step plans
tools: ['search', 'runSubagent', 'usages', 'problems', 'changes']
handoffs:
- label: Start Implementation
agent: agent
prompt: Start implementation
---
You are a PLANNING AGENT, NOT an implementation agent.
<stopping_rules>
STOP IMMEDIATELY if you consider starting implementation.
Plans describe steps for the USER to execute later.
</stopping_rules>
<workflow>
## 1. Context gathering:
MANDATORY: Run #tool:runSubagent for autonomous research.
## 2. Present plan:
Follow <plan_style_guide> and pause for user feedback.
## 3. Handle feedback:
Restart <workflow> to refine plan based on new information.
</workflow>
1.2 .instructions.md Files
Purpose: Provide targeted, context-aware guidelines that automatically apply to specific files or patterns.
Structure:
---
description: Optional hover text in Chat view
applyTo: '**/*.cs' # Glob pattern for file matching
---
# Instructions Body (Markdown)
- Write clear comments for each function
- Use PascalCase for class names
- Add newline before opening braces
Characteristics:
- Automatic Detection: VS Code Copilot detects relevant files based on applyTo patterns
- Scoped Application: Different instructions for different file types/domains
- Silent Operation: Instructions inject into context without explicit UI indication
- Reference Tracking: Used instructions appear in Chat response references
Standard Locations:
1. Global Instructions: .github/copilot-instructions.md (applies to all prompts)
2. Targeted Instructions: .github/instructions/*.instructions.md (scoped by applyTo)
Activation:
- Requires opt-in via VS Code settings: github.copilot.chat.codeGeneration.useInstructionFiles
- Frontmatter applyTo field uses glob patterns for file matching
Example - C# Coding Standards:
---
applyTo: '**/*.cs'
description: C# coding conventions for this project
---
## Naming Conventions
- Use PascalCase for public members
- Use camelCase for private fields with `_` prefix
## Code Organization
- Group members: fields → constructors → properties → methods
- Add XML documentation for public APIs
1.3 .prompt.md Files
Purpose: Reusable, shareable prompt templates with embedded references for frequently used queries.
Structure:
Analyze the #file:src/services/OrderService.cs for:
1. Error handling patterns
2. Dependency injection usage
3. Async/await best practices
Compare with #file:src/services/BaseService.cs and suggest improvements.
Characteristics:
- Reference Embedding: Use #file:, #symbol:, #prompt: syntax for dynamic context
- Composability: Prompt files can reference other prompt files
- Shareability: Stored in version control for team collaboration
- IDE Integration: Autocomplete via #prompt: prefix in chat
Standard Location: .github/prompts/*.prompt.md
Usage Pattern:
1. Create in chat: Write prompt → Copy to .prompt.md file
2. Invoke in chat: Type #prompt: → Select from list → Add details → Send
Example - API Review Template:
<!-- File: .github/prompts/api-review.prompt.md -->
Review #selection for RESTful API compliance:
## Check:
- HTTP method semantics (GET/POST/PUT/DELETE)
- Status code appropriateness
- Request/response validation
- Error handling consistency
## Compare with:
- #file:docs/api-guidelines.md
- Existing patterns in #file:src/controllers/
Suggest improvements with code examples.
2. Context Injection Pipeline
2.1 Context Hierarchy (Priority Order)
┌─────────────────────────────────────────────────────────────┐
│ HIGHEST PRIORITY: Explicit User Context │
│ - #file: references │
│ - #symbol: references │
│ - Selected code │
│ - Attached images/errors │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ IMPLICIT CONTEXT: Current Environment │
│ - Active editor file │
│ - Open tabs │
│ - Chat history (current session) │
│ - Cursor position/surrounding code │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ INSTRUCTIONS: Scoped Rules │
│ - Matched .instructions.md (via applyTo patterns) │
│ - Global copilot-instructions.md │
│ - Agent-specific guidelines │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ SEMANTIC CONTEXT: RAG Retrieval │
│ - Vector search on codebase index │
│ - Semantic similarity matches │
│ - Cross-file relationship analysis │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ SYSTEM PROMPT: Base Capabilities │
│ - Model capabilities (GPT-4, Claude, etc.) │
│ - Base programming knowledge │
│ - General best practices │
└─────────────────────────────────────────────────────────────┘
2.2 Override Rules
Principle: More specific context overrides general context.
- User Explicit > All: Direct
#file:references trump everything - Instructions > System Defaults: Custom
.instructions.mdoverride base rules - Agent Guidelines > Generic Instructions: Agent-specific rules take precedence
- Local > Remote: Workspace instructions override global settings
Example Scenario:
User prompt: "Optimize this function"
+ Selected code: function calculateTotal() {...}
+ .instructions.md: "Use LINQ for collections"
+ System knowledge: "Use loops for performance"
Result: Applies LINQ (per instructions) > generic loop optimization
3. RAG (Retrieval-Augmented Generation) Mechanisms
3.1 Codebase Indexing
Remote Indexing (GitHub/Azure DevOps): - Vector embeddings capture semantic relationships - Cloud-based index for fast cross-repository search - Automatic updates on code changes
Local Indexing (Other Sources): - Built on-demand for non-cloud repositories - Stored in VS Code workspace cache - Refreshed periodically or on file changes
Index Components:
Code Embeddings:
├── Function signatures
├── Class definitions
├── Comment/documentation text
├── Variable names
├── Import/dependency relationships
└── Code structure patterns
3.2 Semantic Search Process
Trigger Conditions: - Copilot determines insufficient context from prompt alone - User asks about unfamiliar concepts - Cross-file analysis required
Search Flow:
1. Query Vectorization
User prompt → Vector embedding (e.g., 1536 dimensions)
2. Similarity Computation
Compare query vector with code embedding vectors
Cosine similarity: sim(A,B) = (A·B) / (|A||B|)
3. Ranking & Retrieval
Top-K files with highest semantic similarity
Typical K = 5-20 files depending on context window
4. Context Augmentation
Retrieved files added to prompt context
Prioritized by relevance score
Semantic vs. Traditional Search:
| Aspect | Semantic Search | Traditional (grep) |
|---|---|---|
| Match Type | Meaning-based | Keyword exact |
| Example Query | "authentication logic" | grep "auth" |
| Finds | Login, JWT validation, OAuth | Only literal "auth" |
| Ranking | Relevance score | None (binary match) |
3.3 RAG Integration with Agent Workflow
Agent Tool Interaction:
Agent receives prompt
↓
Checks available tools: ['semantic_search', 'read_file', 'grep_search']
↓
Decision logic:
- Concept-based query → semantic_search
- Exact pattern search → grep_search
- Known file location → read_file
↓
RAG retrieves relevant code
↓
Agent synthesizes response with retrieved context
↓
Cites sources in response (file paths, line numbers)
Example from Plan.agent.md:
<workflow>
## 1. Context gathering and research:
MANDATORY: Run #tool:runSubagent tool, instructing it to:
- Use semantic_search for conceptual understanding
- Follow with grep_search for specific patterns
- Read identified files with targeted queries
Stop research at 80% confidence threshold.
</workflow>
4. Model Context Protocol (MCP) Integration
4.1 MCP Architecture
Purpose: Standardized protocol for connecting AI agents to external tools, resources, and data sources.
Core Components:
┌─────────────────────────────────────────────────────────┐
│ AI Agent (Copilot) │
└────────────────────────┬────────────────────────────────┘
│
│ MCP Client
↓
┌─────────────────────────────────────────────────────────┐
│ Model Context Protocol (MCP) │
│ ┌─────────────┬──────────────┬─────────────────────┐ │
│ │ Tools │ Resources │ Prompts │ │
│ │ (Actions) │ (Data Access)│ (Templates) │ │
│ └─────────────┴──────────────┴─────────────────────┘ │
└────────────────────────┬────────────────────────────────┘
│
┌────────────────┼────────────────┐
│ │ │
↓ ↓ ↓
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ MCP Server 1 │ │ MCP Server 2 │ │ MCP Server N │
│ (GitHub) │ │ (Azure Docs) │ │ (Dataverse) │
└──────────────┘ └──────────────┘ └──────────────┘
MCP Server Capabilities:
- Tools: Functions the agent can invoke (e.g.,
get_issue,query_docs,deploy_resource) - Resources: File-like data for context (e.g., API responses, documentation)
- Prompts: Predefined templates for common tasks (e.g., PR review, commit message)
4.2 MCP Configuration
Configuration File: .mcp.json (workspace or user level)
Example Structure:
{
"mcpServers": {
"github": {
"command": "mcp-server-github",
"args": ["--token", "${GITHUB_TOKEN}"],
"disabled": false
},
"microsoft-docs": {
"type": "sse",
"url": "https://mcp-docs.microsoft.com",
"disabled": false
}
}
}
Authentication: - OAuth integration for remote servers - Keychain storage for credentials - Per-server authorization management
4.3 MCP in Agent Workflows
Tool Discovery:
# In .agent.md
tools: [
'search', # Built-in VS Code tool
'github/github-mcp-server/get_issue', # MCP tool from GitHub server
'microsoft-docs/docs_search' # MCP tool from Microsoft server
]
Dynamic Tool Invocation:
Agent determines: "Need to search Microsoft docs"
↓
Checks MCP servers for relevant tools
↓
Finds: microsoft-docs/docs_search
↓
Requests user approval (first use)
↓
Invokes tool with parameters
↓
Receives structured response
↓
Integrates into agent response
MCP Prompt Templates:
User types: #mcp:github/pr-review
↓
MCP server provides template:
"""
Review #pr:123 for:
- Code quality issues
- Security vulnerabilities
- Breaking changes
Format: {template_structure}
"""
↓
User fills in PR number → Agent executes
4.4 MCP vs. Built-in Tools
| Aspect | Built-in Tools | MCP Tools |
|---|---|---|
| Scope | VS Code workspace | External systems |
| Examples | read_file, grep_search |
azure_deploy, github_get_issue |
| Auth | Local permissions | OAuth/API keys |
| Discovery | Static list | Dynamic registration |
| Updates | VS Code releases | MCP server updates |
4.5 MCP Context Management
Instructions for MCP Tools:
Stored in .instructions.md:
---
applyTo: '**'
---
You have access to MCP tools:
- `microsoft_docs_search`: Query official Microsoft documentation
- `github/get_issue`: Retrieve GitHub issue details
For questions about Microsoft technologies, ALWAYS use microsoft_docs_search
before relying on training data.
Context File Approach (Alternative):
<!-- scp-mcp-context.md -->
# Security Copilot MCP Tools Context
When building agents, use these MCP tools:
1. `get_agent_skills`: List available Security Copilot skills
2. `generate_agent_yaml`: Create agent definition from description
3. `deploy_agent`: Deploy agent to Security Copilot
Workflow:
1. Describe intent → generate_agent_yaml
2. Review YAML → iterate with user
3. Finalize → deploy_agent
Context Injection Methods:
1. Global Instructions: Add to copilot-instructions.md
2. Explicit Reference: Use #file:scp-mcp-context.md in prompts
3. Agent Guidelines: Include in agent's <workflow> section
5. Priority Hierarchy and Override Rules
5.1 Context Source Priority Matrix
Priority Source Type Scope Override Power
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1 User Explicit Context Single prompt Overrides all
(#file, #symbol, selection)
2 Agent Stopping Rules Agent session Overrides workflows
(behavioral boundaries)
3 Targeted Instructions File pattern Overrides global
(.instructions.md + applyTo)
4 Global Instructions Workspace Overrides system
(copilot-instructions.md)
5 Agent Guidelines Agent instance Overrides generic
(<workflow>, <guidelines>)
6 MCP Prompt Templates Tool-specific Overrides generic
(from MCP servers)
7 RAG Retrieved Context Semantic match Augments knowledge
(vector search results)
8 Chat History Session Provides continuity
(previous conversation)
9 System Prompt Model base Foundational rules
(model capabilities)
5.2 Conflict Resolution Examples
Scenario 1: Coding Style Conflict
System: "Use snake_case for Python"
Global .instructions.md: "Use camelCase for consistency"
File-specific .instructions.md (applyTo: '**/*.py'): "Use snake_case per PEP8"
Result: snake_case (file-specific wins over global)
Scenario 2: Agent vs. User Intent
Agent stopping rule: "STOP if attempting file edits"
User prompt: "Edit this file to add logging"
Result: Agent refuses and suggests handoff to implementation agent
(stopping rules protect agent integrity)
Scenario 3: RAG vs. Instructions
RAG retrieves: Old code pattern (deprecated approach)
.instructions.md: "Use new async/await pattern"
Result: Copilot suggests async/await pattern with reference to instructions
(instructions guide interpretation of retrieved code)
5.3 Context Merging Strategy
Additive Merging (Default):
- Multiple .instructions.md files combine if applyTo patterns overlap
- All matching instructions added to context
Replacement Merging (Explicit): - Agent guidelines override generic instructions - Stopping rules replace permissive defaults
Example Merged Context:
FINAL CONTEXT for `src/services/PaymentService.cs`:
[System Prompt]
You are GitHub Copilot, an AI programming assistant...
[Global Instructions from copilot-instructions.md]
- Follow SOLID principles
- Write unit tests for business logic
[C# Instructions from .github/instructions/csharp.instructions.md]
- Use PascalCase for public members
- Add XML documentation
[Agent Guidelines from CodeReviewer.agent.md]
<stopping_rules>
STOP if attempting to fix code—only document issues.
</stopping_rules>
[RAG Retrieved Context]
File: src/services/OrderService.cs (similar payment logic)
[...code snippet...]
[User Query]
"Review the ProcessPayment method"
6. Real-World Integration Patterns
6.1 Multi-Agent Workflow with Context Sharing
Scenario: Complex feature implementation
┌─────────────────────────────────────────────────────────────┐
│ 1. Plan Agent │
│ - Tools: semantic_search, runSubagent │
│ - Context: .instructions.md, RAG │
│ - Output: Structured plan in plan.prompt.md │
└────────────────────┬────────────────────────────────────────┘
│ Handoff: #prompt:plan-implementation.prompt.md
↓
┌─────────────────────────────────────────────────────────────┐
│ 2. Implementation Agent │
│ - Tools: replace_string_in_file, run_in_terminal │
│ - Context: Plan + .instructions.md + RAG │
│ - Output: Code changes + test files │
└────────────────────┬────────────────────────────────────────┘
│ Handoff: "Run tests"
↓
┌─────────────────────────────────────────────────────────────┐
│ 3. Test Agent │
│ - Tools: run_in_terminal, get_errors │
│ - Context: Test results + error logs │
│ - Output: Test report + fix suggestions │
└─────────────────────────────────────────────────────────────┘
Context Continuity Mechanisms:
1. Prompt Files: Plan saved as .prompt.md, referenced in handoff
2. Chat History: Previous agent responses available to next agent
3. File References: Modified files tracked and auto-referenced
6.2 RAG-Powered Code Search Pattern
Use Case: "Find all authentication implementations"
# Pseudo-code for agent logic
def search_authentication():
# 1. Semantic Search (RAG)
semantic_results = semantic_search(
query="authentication login user validation",
max_results=10
)
# 2. Validate with grep_search
confirmed_files = []
for file in semantic_results:
if grep_search(
query="authenticate|login|verify",
file=file,
isRegexp=True
):
confirmed_files.append(file)
# 3. Read relevant files
code_snippets = []
for file in confirmed_files[:5]: # Limit token usage
content = read_file(file)
code_snippets.append({
'file': file,
'content': extract_auth_functions(content)
})
# 4. Synthesize response
return f"""
Found authentication in {len(confirmed_files)} files:
{format_findings(code_snippets)}
Common patterns:
- JWT token validation
- Password hashing with bcrypt
- OAuth2 integration
References: {list_file_paths(confirmed_files)}
"""
6.3 MCP-Enhanced Documentation Workflow
Use Case: Answer question using latest Azure docs
User: "How do I create an Azure Storage account with CLI?"
↓
Agent checks MCP tools: microsoft_docs_search available
↓
Agent calls: microsoft_docs_search(query="Azure Storage account CLI create")
↓
MCP returns:
{
"results": [
{
"title": "Quickstart: Create storage account",
"url": "https://learn.microsoft.com/azure/storage/...",
"excerpt": "Use az storage account create to..."
}
]
}
↓
Agent fetches full content: microsoft_docs_fetch(url=...)
↓
Agent synthesizes:
"""
To create an Azure Storage account using CLI:
```bash
az storage account create \
--name mystorageaccount \
--resource-group myresourcegroup \
--location eastus \
--sku Standard_LRS
Reference: Official Azure Documentation """
### 6.4 Context-Aware File Generation
**Use Case:** Generate code matching project conventions
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
namespace MyApp.Services
{
/// <summary>
/// Service for managing order operations.
/// </summary>
public interface IOrderService
{
Task<Order> GetOrderAsync(int orderId);
}
public class OrderService : IOrderService
{
private readonly ILogger<OrderService> _logger;
private readonly IOrderRepository _repository;
public OrderService(
ILogger<OrderService> logger,
IOrderRepository repository)
{
_logger = logger;
_repository = repository;
}
// Implementation follows...
}
}
Generated based on: - Dependency injection pattern from UserService.cs - XML documentation from .instructions.md - SOLID principles from global instructions """
---
## 7. Code Examples and Configuration Patterns
### 7.1 Complete .instructions.md Example
```markdown
<!-- .github/instructions/typescript.instructions.md -->
---
description: TypeScript coding standards for React components
applyTo: 'src/**/*.tsx'
---
# React Component Guidelines
## Naming Conventions
- Use PascalCase for component files: `UserProfile.tsx`
- Use camelCase for utility files: `formatDate.ts`
## Component Structure
```typescript
// 1. Imports (grouped: React → third-party → local)
import React, { useState, useEffect } from 'react';
import { Button } from '@mui/material';
import { useAuth } from '@/hooks/useAuth';
// 2. Type definitions
interface UserProfileProps {
userId: string;
onUpdate?: (user: User) => void;
}
// 3. Component
export const UserProfile: React.FC<UserProfileProps> = ({ userId, onUpdate }) => {
// 3.1 Hooks (useState → useEffect → custom hooks)
const [user, setUser] = useState<User | null>(null);
const { isAuthenticated } = useAuth();
// 3.2 Event handlers
const handleUpdate = () => { /* ... */ };
// 3.3 Render
return <div>{/* ... */}</div>;
};
State Management
- Use
useStatefor component-local state - Use context for cross-component state
- Never mutate state directly—use spread operators
Testing Requirements
- Every component must have a corresponding
.test.tsxfile - Test file structure: Arrange → Act → Assert
- Use Testing Library queries:
getByRole,getByText### 7.2 .prompt.md Template Examples ```markdown <!-- .github/prompts/feature-implementation.prompt.md --> Implement a new feature: [FEATURE_NAME] ## Context - Related files: #file:[CORE_FILE_PATH] - Similar features: #file:[REFERENCE_FEATURE] - API docs: #file:docs/api-guidelines.md ## Requirements 1. Follow patterns from #file:[REFERENCE_FEATURE] 2. Add unit tests in `tests/` directory 3. Update documentation in #file:README.md ## Checklist - [ ] Implementation matches coding standards in #file:.github/instructions/ - [ ] Error handling covers edge cases - [ ] Tests achieve >80% coverage - [ ] No breaking changes to existing APIs Generate: 1. Implementation files 2. Test files 3. Documentation updates
<!-- .github/prompts/bug-investigation.prompt.md -->
Investigate bug: [BUG_DESCRIPTION]
## Error Context
## Investigation Steps
1. Search for error message: Use semantic_search
2. Check recent changes: Review #changes
3. Examine test failures: Check #testFailure
4. Review related code: #file:[AFFECTED_FILE]
## Analysis
- Root cause hypothesis
- Affected components
- Reproduction steps
- Suggested fix approach
## Next Steps
- [ ] Verify fix in isolation
- [ ] Add regression test
- [ ] Update error handling
7.3 MCP Configuration Examples
// .mcp.json (User-level MCP configuration)
{
"mcpServers": {
"github": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}"
},
"disabled": false
},
"microsoft-docs": {
"type": "sse",
"url": "https://mcp-docs-server.microsoft.com/api/mcp",
"headers": {
"Authorization": "Bearer ${MS_DOCS_TOKEN}"
},
"disabled": false
},
"azure-resources": {
"command": "azure-mcp-server",
"args": ["--subscription", "${AZURE_SUBSCRIPTION_ID}"],
"disabled": false
}
},
"mcpToolApprovals": {
"github/get_issue": "always",
"microsoft-docs/docs_search": "always",
"azure-resources/deploy": "prompt"
}
}
7.4 Agent Definition with Full Context Integration
<!-- InfoGatherer.agent.md -->
````chatagent
---
name: InfoGatherer
description: Researches codebases and synthesizes findings into reports
argument-hint: Describe what information to gather (e.g., "Find all API endpoints")
tools: [
'semantic_search',
'grep_search',
'read_file',
'list_dir',
'fetch_webpage',
'github/github-mcp-server/get_issue',
'microsoft-docs/docs_search'
]
handoffs:
- label: Generate Documentation
agent: agent
prompt: '#createFile Create a report from the gathered information'
---
You are an INFORMATION GATHERING AGENT specializing in codebase research and analysis.
Your role is to autonomously search, read, and synthesize information from multiple sources, NOT to implement changes.
<stopping_rules>
STOP IMMEDIATELY if you attempt to edit files, run terminal commands, or make code changes.
If you identify areas needing changes, DOCUMENT them for handoff—do not implement.
</stopping_rules>
<workflow>
## 1. Query Analysis
- Parse user request to identify key concepts
- Determine information sources: codebase, docs, issues
- Set confidence threshold (target: 90%)
## 2. Multi-Source Research
Execute searches in parallel:
### A. Codebase Search (RAG)
### B. Documentation Search (MCP)
If query relates to frameworks/technologies:
### C. Issue Tracker (MCP)
If query involves bugs/features:
## 3. Synthesis
Organize findings:
```markdown
## Summary (2-3 sentences)
## Detailed Findings
### Source 1: [Codebase]
- Location: #file:path/to/file.ts:123
- Key points: ...
### Source 2: [Documentation]
- Reference: [Title](url)
- Key points: ...
### Source 3: [Issues]
- Issue #123: ...
- Related: #456, #789
## Patterns Identified
- Common approach: ...
- Variations: ...
- Edge cases: ...
## Recommendations
1. [Actionable suggestion]
2. [Alternative approach]
## References
- [List of all sources consulted]
MANDATORY: Pause for user feedback before handoff.
RAG Usage
- Start broad (semantic_search) → narrow down (grep_search)
- Read files selectively—only confirmed matches
- Track token budget: stop at 80% if approaching limit
MCP Integration
- Query external docs for framework-specific questions
- Use issue data for historical context
- Combine MCP results with local findings
Instructions Awareness
- Check for .instructions.md matching searched files
- Note project-specific patterns to inform synthesis
- Reference instructions in recommendations
typescript
- Cite sources at end of sections
- Keep summary under 100 words
</output_format>
````
8. Technical Implementation Details
8.1 Vector Embedding Process (RAG)
Model: Typically text-embedding-ada-002 or similar (1536 dimensions)
Indexing Pipeline: ```python
Conceptual implementation
def index_codebase(repo_path): code_chunks = []
for file in walk_directory(repo_path):
if is_code_file(file):
# Parse file into semantic chunks
chunks = split_into_chunks(
file_content=read_file(file),
strategy='ast-aware', # Respect function/class boundaries
max_tokens=512
)
for chunk in chunks:
# Generate embedding
embedding = embed_model.encode(chunk.text)
code_chunks.append({
'file': file.path,
'start_line': chunk.start_line,
'end_line': chunk.end_line,
'content': chunk.text,
'embedding': embedding,
'metadata': {
'type': chunk.type, # function/class/module
'name': chunk.name,
'language': file.language
}
})
# Store in vector database (e.g., FAISS, Pinecone)
vector_db.upsert(code_chunks)
return vector_db
def semantic_search(query, max_results=10): # Embed query query_embedding = embed_model.encode(query)
# Vector similarity search
results = vector_db.search(
query_vector=query_embedding,
top_k=max_results,
metric='cosine'
)
# Re-rank by additional signals
results = rerank(results, signals=[
'recency', # Newer code prioritized
'centrality', # Core logic prioritized
'quality_score' # Well-documented code
])
return results
```
Chunk Splitting Strategy: ``` File: src/services/UserService.ts (2000 lines) ↓ Chunking (AST-aware): ├── Chunk 1: Import statements + class declaration (lines 1-50) ├── Chunk 2: getUserById method + dependencies (lines 51-150) ├── Chunk 3: createUser method + validation (lines 151-300) ├── ... └── Chunk N: Helper functions (lines 1900-2000)
Each chunk: - Self-contained (includes necessary context) - Respects syntax boundaries (complete functions) - Max 512 tokens to fit embedding model - Overlaps by 50 tokens to preserve continuity ```
8.2 Context Window Management
Token Budget Allocation: ``` Total Context Window: 128,000 tokens (GPT-4 Turbo)
Allocation: ├── System Prompt: ~2,000 tokens ├── Instructions: ~5,000 tokens │ ├── Global .instructions.md │ └── Targeted .instructions.md (multiple) ├── Agent Definition: ~3,000 tokens │ ├── Frontmatter + tools │ ├── Workflow + guidelines │ └── Stopping rules ├── RAG Retrieved Context: ~30,000 tokens │ ├── 10-20 code files │ └── Ranked by relevance ├── Chat History: ~10,000 tokens │ └── Last 5-10 exchanges ├── User Context: ~20,000 tokens │ ├── Explicit references (#file, #symbol) │ └── Selected code └── Response Generation: ~58,000 tokens (remaining) ```
Overflow Handling: 1. Priority Pruning: Remove lower-priority context first - Chat history older than 5 messages - RAG results below relevance threshold 2. Summarization: Compress lengthy files before adding 3. Chunked Reading: Read files in sections with offset/limit 4. User Notification: "Context too large, please narrow scope"
8.3 Instruction File Matching Algorithm
```python
Conceptual implementation of applyTo matching
def find_applicable_instructions(file_path, workspace_root): applicable = []
# 1. Load all .instructions.md files
instruction_files = glob(
f"{workspace_root}/.github/instructions/*.instructions.md"
)
for inst_file in instruction_files:
frontmatter = parse_yaml_frontmatter(inst_file)
apply_to_pattern = frontmatter.get('applyTo', '**')
# 2. Match glob pattern
if glob_match(file_path, apply_to_pattern, workspace_root):
applicable.append({
'file': inst_file,
'content': read_file(inst_file),
'description': frontmatter.get('description', ''),
'priority': calculate_priority(apply_to_pattern)
})
# 3. Sort by specificity (most specific pattern wins)
# e.g., 'src/api/**/*.ts' > 'src/**/*.ts' > '**/*.ts'
applicable.sort(key=lambda x: x['priority'], reverse=True)
return applicable
def calculate_priority(pattern): """More specific patterns get higher priority.""" specificity = 0 specificity += pattern.count('/') # Deeper paths specificity += len(pattern.replace('', '')) # Non-wildcard chars specificity -= pattern.count('*') # Penalize recursive wildcards return specificity
Example usage
file_path = "src/api/controllers/UserController.ts" instructions = find_applicable_instructions(file_path, workspace_root)
Result (sorted by priority):
[ { file: 'src-api-controllers.instructions.md', priority: 25 }, # Most specific { file: 'typescript.instructions.md', priority: 10 }, { file: 'copilot-instructions.md', priority: 0 } # Global ] ```
9. Best Practices and Recommendations
9.1 For File-Based Context Engineering
✅ DO:
- Specific Patterns: Use precise applyTo globs (e.g., src/api/**/*.ts vs. **/*.ts)
- Scoped Instructions: Create separate .instructions.md for distinct domains (API, UI, tests)
- Version Control: Store all context files in repo for team consistency
- Clear Descriptions: Add description frontmatter for discoverability
- Test Instructions: Verify instructions apply by checking Chat response references
❌ DON'T:
- Overlap Carelessly: Avoid conflicting instructions in overlapping patterns
- Overload Global: Keep copilot-instructions.md concise (<2000 words)
- Hardcode Secrets: Never put tokens/keys in .mcp.json (use env vars)
- Ignore Hierarchy: Remember user explicit context always wins
- Forget Updates: Review and prune outdated instructions regularly
9.2 For RAG Optimization
Query Formulation: ``` ❌ Poor: "Find login code" ✅ Better: "Find user authentication and session management logic" ✅ Best: "Find implementations of JWT token validation and refresh logic in API routes"
Rationale: More specific queries → better semantic matches ```
Search Strategy: ```python
Recommended search sequence
1. Broad semantic search (conceptual understanding)
semantic_search("authentication logic", max_results=20)
2. Filter with grep (validation)
grep_search("authenticate|login|jwt", isRegexp=True)
3. Targeted reads (deep inspection)
read_file("src/auth/AuthService.ts") ```
Token Efficiency:
- Request summaries instead of full files when possible
- Use read_file with offset and limit for large files
- Parallelize independent searches
- Cache frequently accessed context
9.3 For Agent Design
Single Responsibility:
❌ Bad: "UniversalAgent" with 50 tools doing everything
✅ Good: Specialized agents
- PlanAgent: Research + outline (read-only tools)
- ImplementAgent: Code changes (edit + terminal tools)
- ReviewAgent: Analysis + feedback (read + error tools)
Stopping Rules Clarity:
```
❌ Weak: "Don't do things outside your scope"
✅ Strong:
If you catch yourself writing code, STOP and present a plan instead. ```
Workflow Explicitness:
```
❌ Vague: "Gather context and respond"
✅ Explicit:
1. Context Gathering (MANDATORY)
- Run semantic_search for concepts
- Run grep_search for exact matches
- Read top 5 files
MANDATORY: Achieve 80% confidence or ask clarifying questions.
2. Synthesis
- Organize findings by theme
- Identify patterns and anti-patterns
3. User Feedback (MANDATORY)
Present findings and WAIT for feedback. DO NOT proceed to implementation. ```
9.4 For MCP Integration
Tool Selection: ```yaml
Principle: Least privilege
❌ Grant all tools by default ✅ Grant tools based on agent role:
PlanAgent: tools: ['semantic_search', 'github/get_issue'] # Read-only
ImplementAgent: tools: ['replace_string', 'run_terminal'] # Write access
DocAgent: tools: ['microsoft-docs/search', 'fetch_webpage'] # External docs ```
Authentication Security: ```json // ❌ DON'T: Hardcode tokens { "env": { "GITHUB_TOKEN": "ghp_abc123..." } }
// ✅ DO: Use environment variables { "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } }
// Set in shell: // export GITHUB_TOKEN=ghp_abc123... ```
Error Handling:
```markdown
If MCP tool unavailable:
- Log warning: "MCP server not configured"
- Fallback to built-in tools (e.g., fetch_webpage)
- Inform user of limitation
If MCP call fails:
- Retry once with exponential backoff
- If still fails, document error and continue
- Suggest manual alternative
Example: "Unable to fetch latest Azure docs via MCP. Providing answer based on training data (may be outdated). Consider checking [URL] manually." ```
10. Future Trends and Evolution
10.1 Emerging Patterns
1. Agentic Workflows (Multi-Agent Systems)
- Specialized agents coordinate via handoffs
- Context passed through .prompt.md files
- Workflow orchestration via "conductor" agents
2. Dynamic Tool Discovery - MCP servers dynamically register new tools - Agents query capabilities at runtime - User approvals cached per-session
3. Context Personalization
- Learn user preferences over time
- Store in "Copilot memories" (.editorconfig, CONTRIBUTING.md)
- Auto-generate project-specific instructions
4. Hybrid RAG + LLM Reasoning - RAG provides facts, LLM provides reasoning - Multi-hop reasoning across retrieved documents - Agentic frameworks (LangChain, Semantic Kernel)
10.2 MCP Ecosystem Growth
Current State: - ~50 community MCP servers (GitHub, Slack, databases) - Standardization in progress (v1.0 spec)
Near Future (2025-2026): - Enterprise MCP marketplaces - Pre-certified security-reviewed servers - Native OS integration (Windows File Explorer MCP) - Cross-platform agent frameworks (Microsoft Agent Framework)
10.3 Context Engineering Best Practices (Next Gen)
1. Context Tiering
Tier 1: Immediate (current file, selection) → Always included
Tier 2: Proximate (open files, recent history) → Included if space
Tier 3: Repository (semantic search) → Included on-demand
Tier 4: External (docs, issues via MCP) → Included with approval
2. Proactive Context Injection - Copilot predicts needed context before user asks - Pre-fetches related files based on intent detection - Suggests additional references ("Also consider: #file:...")
3. Context Explainability - Show context sources in UI ("Based on: .instructions.md + 3 files via RAG") - Allow users to exclude specific sources - Provide "why this context?" explanations
11. Conclusion
Key Findings Summary
- File Types Serve Distinct Roles:
.agent.md: Define agent behavior and tool authorization.instructions.md: Provide scoped, automatic context injection-
.prompt.md: Enable reusable, composable query templates -
Context Hierarchy is Strict:
- User explicit context (selections,
#file:) overrides all - Instructions override system defaults
-
RAG augments but doesn't override explicit guidance
-
RAG is Semantic Search + Context Augmentation:
- Vector embeddings capture code meaning
- Retrieval ranked by relevance, not keyword matching
-
Integrated into agent workflows via tools
-
MCP Bridges AI Agents and External Systems:
- Standardized protocol for tools, resources, prompts
- Dynamic discovery and invocation
-
Growing ecosystem of servers (GitHub, Azure, Dataverse)
-
Priority System Ensures Correctness:
- More specific context wins (file-level > global)
- Behavioral boundaries (stopping rules) protect integrity
- Conflicts resolved by source hierarchy
Practical Takeaways
For Developers:
- Start with global .instructions.md, specialize with targeted files
- Design agents with single responsibilities and clear handoffs
- Use RAG for discovery, explicit references for precision
- Integrate MCP for external data/tools
For Teams: - Version-control all context files for consistency - Document agent roles and handoff patterns - Review and prune instructions regularly - Share prompt templates for common workflows
For Organizations: - Establish MCP governance (approved servers, security reviews) - Train teams on context engineering best practices - Monitor token usage and optimize context efficiency - Build internal MCP servers for proprietary systems
12. References and Resources
Official Documentation
- Microsoft Learn: Copilot Context Overview
- Microsoft Learn: Custom Instructions
- Model Context Protocol Specification
- GitHub Copilot Documentation
MCP Resources
Code Samples
Community
Report Version: 1.0
Last Updated: November 26, 2025
Next Review: Q1 2026 (track MCP v1.0 release)