Multi-Agent RAG with LangChain, LangGraph & MySQL Vector Search: Complete Node.js Developer Guide

Quick Summary: This guide explains how to build a modern AI knowledge system that can search structured database records, vector embeddings, documents, and business data using RAG. You will learn when to use vector search, BM25, hybrid search, reranking, chunking, metadata filtering, LangChain, LangGraph, and multi-agent workflows.

What you will learn:

• Multi-agent RAG architecture
• LangChain vs LangGraph responsibilities
• MySQL vector database architecture
• BM25, vector and hybrid search
• Chunking strategies
• Token optimization
• User and admin AI assistants
• Permission-aware retrieval
• Tool calling and provider integrations
• Production optimization and common mistakes

1. What Is Multi-Agent RAG?

RAG means Retrieval-Augmented Generation. Instead of asking an AI model to answer only from its trained knowledge, the application first retrieves relevant information from your own data and then provides that information to the model as context.

A simple RAG system can search documents and generate an answer. A multi-agent RAG system goes one step further: different specialized agents can handle different responsibilities such as user intent detection, database search, vector search, billing information, account information, document retrieval, tool execution, security checks, and final response generation.

This architecture is useful for SaaS applications, banking dashboards, enterprise knowledge bases, customer-support systems, internal documentation, ecommerce platforms, CRM systems and admin assistants.

Simple RAG

User ↓ Embedding ↓ Vector Search ↓ Relevant Documents ↓ LLM ↓ Answer

Multi-Agent RAG

User ↓ Router / Supervisor Agent ├── Knowledge Agent ├── Database Agent ├── Customer Agent ├── Admin Agent ├── Billing Agent ├── Search Agent └── Tool Agent ↓ LangGraph Workflow ↓ Retrieval + Tools ↓ Final Answer

2. Why Use LangChain and LangGraph Together?

LangChain is useful for building the individual AI building blocks: models, embeddings, retrievers, tools, prompts, document loaders and chains.

LangGraph is useful when the application needs a stateful workflow with multiple steps, routing, conditional execution, retries, loops and multiple agents.

Technology	Main Responsibility
Node.js	Application runtime and API layer
LangChain	Models, embeddings, tools, retrievers and AI components
LangGraph	Agent workflow, state and routing
MySQL	Business data, permissions, metadata and vector storage where supported by the deployment
Embedding Model	Converts text into vectors
LLM	Reasoning and response generation

3. Production Multi-Agent Architecture

A production application should not allow every agent to access every database table and every tool. The system should have a controlled orchestration layer.

USER | v API / Auth Layer | v Intent / Router Agent | +-------------+-------------+ | | | v v v Search Agent Data Agent Tool Agent | | | v v v Vector/BM25 MySQL DB External APIs | | | +-------------+-------------+ | v Reranker / Filter | v Answer Agent | v USER

The important idea is that retrieval, authorization, business logic and generation should not be mixed into one giant prompt.

4. MySQL + Vector Search: What Data Should You Store?

For many business applications, MySQL remains the source of truth for structured application data. Vector search should complement the relational database rather than replacing normal relational queries.

Recommended Data Categories

• Users
• Organizations
• Roles and permissions
• Customers
• Orders
• Transactions
• Products
• Support tickets
• Documents
• Document chunks
• Embeddings
• Conversation history
• Agent execution history
• Tool execution logs
• Audit logs

Example Document Chunk Structure

documents ------------ id tenant_id title source content created_at ## document_chunks id document_id tenant_id chunk_index content token_count embedding metadata created_at

Important: The exact vector column type, indexing strategy and similarity capabilities depend on the MySQL version and deployment. Always verify the capabilities of the MySQL environment you are using before designing the final production schema.

5. MySQL Should Not Replace Normal SQL Queries

This is one of the most important architecture rules.

If the user asks:

"Show my last five transactions."

Do not perform vector search for this request. Use a normal SQL query with authorization.

If the user asks:

"Explain the company's refund policy."

Vector or hybrid retrieval is appropriate because the answer may exist inside documentation.

If the user asks:

"Show transactions related to a refund and explain the policy."

The application may need both:

• Structured SQL query for transactions
• Vector or hybrid retrieval for policy documents

6. User Assistant vs Admin Assistant

A common production mistake is building one AI assistant with unrestricted access.

A better design is to use permission-aware assistants.

Assistant	Typical Access
User Assistant	Own account, orders, documents and permitted support data
Support Agent	Customer support records and approved knowledge base
Admin Assistant	Broader operational data according to explicit permissions
Finance Agent	Approved financial data and reporting tools

The AI model should never be the final authority for authorization. Your application backend must enforce permissions.

7. Permission-Aware RAG

Every vector record should carry enough metadata to determine whether the current user is allowed to retrieve it.

{ "tenant_id": "company_123", "document_id": 981, "department": "support", "visibility": "internal", "allowed_roles": ["admin", "support"], "source_type": "knowledge_base" }

Retrieval should apply security filters before the model receives the retrieved content.

Never rely on the prompt alone to hide sensitive records.

8. Vector Search vs BM25 vs Hybrid Search

There is no single search technique that is perfect for every query.

Vector Search

Vector search finds content based on semantic similarity. It is useful when the user's wording differs from the wording in the source document.

Example:

User: "How can I get my money back?"

Document: "Customers may request a refund within 30 days."

These phrases are semantically related even though the exact words are different.

BM25 / Keyword Search

BM25 is useful for exact or keyword-heavy queries, especially product names, IDs, error codes, technical terms and uncommon words.

Example:

"ERR_CONNECTION_RESET Node.js 502"

A pure semantic search system may not always prioritize the exact technical token as effectively as keyword retrieval.

Hybrid Search

Hybrid search combines lexical retrieval such as BM25 with semantic vector retrieval.

User Query | +------------------+ | | v v BM25 Search Vector Search | | +--------+---------+ | v Merge Results | v Reranker | v Top Documents | v LLM

9. Which Search Technique Should You Use?

Query Type	Recommended Retrieval
Exact error code	BM25 / keyword
Natural language question	Vector search
Technical question with exact terms	Hybrid search
Sensitive business data	SQL + permission filters
Complex enterprise search	Hybrid + reranking

10. Chunking: The Most Important RAG Optimization

Documents should not normally be inserted into the embedding model as one huge block. They should be divided into meaningful chunks.

Bad chunking can produce poor retrieval even when you have a high-quality embedding model.

Common Chunking Strategies

• Fixed-size chunking: Split text into a fixed token or character range.
• Recursive chunking: Split using paragraphs, sentences and smaller boundaries.
• Semantic chunking: Split according to meaning and topic changes.
• Heading-based chunking: Preserve document sections and headings.
• Code-aware chunking: Keep functions, classes and code sections together.
• Parent-child chunking: Search small chunks but return their larger parent context.

Good Chunk

Heading: Refund Policy Customers can request a refund within 30 days of purchase. Refund requests must include the order ID and reason for the request.

Bad Chunk

...random paragraph... ...middle of another topic... ...half sentence... ...table row... ...next section heading...

The correct chunk size depends on the content. Do not blindly use one chunk size for every document type.

11. Chunking Strategy by Content Type

Content	Useful Strategy
Documentation	Heading + paragraph based
Legal documents	Section-aware chunks
Technical docs	Heading + code-aware chunks
FAQs	Question-answer pairs
Products	Field/group based records

12. Chunk Overlap: Do You Always Need It?

Chunk overlap can preserve context around chunk boundaries, but too much overlap increases storage, embedding cost and retrieval duplication.

For example, if a document is split into chunks of 500 tokens, an overlap of 50–100 tokens may preserve boundary context. However, the correct value should be tested against your actual dataset.

Optimization rule: Measure retrieval quality instead of assuming a fixed overlap is always best.

13. Token Optimization for RAG

Token usage affects cost, latency and model context limits. The goal is not simply to retrieve more documents. The goal is to retrieve the smallest amount of high-quality context needed to answer the question.

Bad Pattern

Retrieve 50 chunks ↓ Send all 50 chunks to LLM ↓ Huge prompt ↓ Higher cost + slower response

Better Pattern

Retrieve candidates ↓ Metadata filtering ↓ BM25 + Vector merge ↓ Reranking ↓ Top 5–10 chunks ↓ Context compression ↓ LLM

14. Practical Token Optimization Techniques

• Use appropriate chunk sizes.
• Remove duplicated content.
• Store metadata separately.
• Filter by tenant before semantic retrieval when possible.
• Use top-K retrieval carefully.
• Use reranking when the candidate set is large.
• Summarize very large retrieved context.
• Do not send irrelevant metadata to the model.
• Keep system prompts stable and concise.
• Cache repeated retrieval results where appropriate.
• Use smaller models for routing and classification when quality permits.

15. Embedding Model Selection

An embedding model converts text into numerical vectors. Documents and user queries should normally be embedded using compatible embedding models so their vectors can be compared meaningfully.

When selecting an embedding model, evaluate:

• Embedding dimensions
• Language support
• Semantic retrieval quality
• Latency
• Cost
• Maximum input size
• Domain performance

Do not select a model only because it has a larger vector dimension. Retrieval quality and operational cost matter more than dimension size alone.

16. Recommended Provider Categories

A production AI application can separate providers by responsibility instead of hard-coding the entire application to one provider.

Category	Examples
LLM Providers	OpenAI, Anthropic, Google, Mistral, Cohere and other compatible providers
Embedding Providers	OpenAI, Cohere, Google and open-source embedding models
Vector Stores	MySQL vector capabilities, Qdrant, Pinecone, Weaviate, Milvus and other vector databases
Keyword Search	MySQL full-text search, Elasticsearch/OpenSearch and other lexical engines
Reranking	Dedicated reranking models or provider APIs

The best provider depends on your latency, privacy, data residency, cost, quality and infrastructure requirements.

17. Tool Calling in Multi-Agent Systems

Agents become much more useful when they can call controlled tools.

Examples include:

• Get customer profile
• Search orders
• Check transaction status
• Search company documentation
• Create support ticket
• Calculate invoice totals
• Check subscription status
• Search product inventory
• Generate a report
• Send an approved notification

Every tool should have a narrow purpose and explicit authorization requirements.

Example Tool Contract

{ "name": "get_customer_orders", "description": "Get orders belonging to the authenticated customer", "input": { "customerId": "string" }, "permissions": [ "customer.read" ] }

18. LangGraph Workflow for Multi-Agent RAG

LangGraph-style workflows are useful when the application needs explicit state and conditional routing.

START ↓ Authenticate User ↓ Load Permissions ↓ Classify Intent ↓ Route Request ├── SQL Agent ├── RAG Agent ├── Tool Agent └── Hybrid Agent ↓ Retrieve Data ↓ Validate Access ↓ Rerank Data ↓ Generate Answer ↓ END

This explicit workflow is easier to observe, test and control than putting the entire business process inside one autonomous agent prompt.

19. Multi-Agent Design: Good vs Bad

❌ Bad Architecture

One giant AI agent ↓ Can access every table ↓ Can call every API ↓ Can retrieve every document ↓ Huge prompt ↓ Difficult to test ↓ Security risk

✅ Better Architecture

Supervisor ↓ Small specialized agents ↓ Explicit tools ↓ Permission checks ↓ Filtered retrieval ↓ Reranking ↓ Final response

20. RAG Retrieval Pipeline

A robust retrieval pipeline can look like this:

User Query ↓ Normalize Query ↓ Intent Detection ↓ Permission / Tenant Filter ↓ BM25 Retrieval + Vector Retrieval ↓ Merge Candidates ↓ Deduplicate ↓ Rerank ↓ Top-K Context ↓ Context Compression ↓ LLM ↓ Answer + Sources

21. Query Rewriting Can Improve Search

Users often write short or ambiguous questions. A search agent can rewrite the query into a retrieval-friendly form before performing search.

Example:

User: "refund kaise?"

Search Query: "customer refund eligibility, refund request process, refund period and required order information"

The rewritten query should preserve the user's intent and should not invent facts.

22. Metadata Filtering Before Vector Search

Metadata is extremely important for enterprise RAG.

Useful metadata includes:

• tenant_id
• user_id
• department
• document_type
• language
• created_at
• updated_at
• visibility
• permissions
• source

For a multi-tenant SaaS platform, tenant filtering should be a first-class security requirement.

23. RAG for Banking or Financial Assistants

For banking-style applications, separate factual transactional operations from knowledge retrieval.

User Assistant

• Show account information
• Explain transaction categories
• Search approved financial documentation
• Explain fees and policies
• Answer account-related questions using authorized data

Admin Assistant

• Operational reporting
• Approved transaction analytics
• Support investigation
• Knowledge-base search
• System documentation

For sensitive operations, the AI should not directly execute high-risk actions simply because a user asked for them. The application should enforce authentication, authorization, validation, approval and audit requirements.

24. Structured Data + Vector Data Together

A strong enterprise architecture often combines three retrieval modes:

1. SQL ↓ Exact structured facts 2. BM25 ↓ Exact words / identifiers 3. Vector Search ↓ Semantic meaning Combined ↓ Hybrid Retrieval ↓ Reranker ↓ LLM

25. Common RAG Mistakes

• One huge chunk: Retrieval becomes noisy.
• Too many chunks: Context becomes expensive.
• No metadata: Filtering becomes difficult.
• No permission filtering: Sensitive information can leak.
• Vector-only search: Exact terms may be missed.
• BM25-only search: Semantic questions may perform poorly.
• No reranking: Relevant results may not reach the LLM.
• Huge prompts: Higher latency and cost.
• One giant agent: Difficult to test and secure.
• AI authorization: The model should not be the security boundary.

26. RAG Evaluation: Do Not Trust the Demo

A RAG system that works on five sample questions is not necessarily production-ready.

Create a test dataset containing real question-and-answer examples.

Measure:

• Retrieval relevance
• Recall
• Precision
• Answer correctness
• Groundedness
• Latency
• Token usage
• Cost per request

When changing chunk size, embedding model, top-K, BM25 weight or reranking strategy, compare the results against the same evaluation dataset.

27. Production Optimization Checklist

• Use connection pooling.
• Cache repeated embeddings when appropriate.
• Cache expensive retrieval operations where safe.
• Use asynchronous document ingestion.
• Generate embeddings in background jobs.
• Do not block user requests during large document processing.
• Use pagination for large database operations.
• Apply tenant and permission filters early.
• Monitor token consumption.
• Log retrieval results for debugging.
• Track agent and tool execution latency.
• Use retries carefully for provider failures.
• Use timeouts for external tools.
• Prevent infinite agent loops.

28. Recommended Node.js Project Structure

src/ ├── agents/ │ ├── supervisor.agent.ts │ ├── search.agent.ts │ ├── database.agent.ts │ └── admin.agent.ts │ ├── graph/ │ └── rag.graph.ts │ ├── rag/ │ ├── embeddings.ts │ ├── chunking.ts │ ├── retrieval.ts │ ├── hybrid-search.ts │ └── reranker.ts │ ├── tools/ │ ├── customer.tool.ts │ ├── orders.tool.ts │ └── reports.tool.ts │ ├── db/ │ ├── mysql.ts │ └── models/ │ ├── security/ │ ├── permissions.ts │ └── authorization.ts │ └── api/ └── chat.ts

29. A Fast Learning Path for Developers

If you are learning this architecture, do not start with ten agents and twenty tools.

Level 1 — Basic RAG

Learn documents → chunks → embeddings → vector search → LLM.

Level 2 — Hybrid Search

Add BM25/keyword retrieval and learn how to merge lexical and semantic results.

Level 3 — Metadata Filtering

Add tenant, user, document type and permission filters.

Level 4 — Tools

Give the AI controlled access to business operations.

Level 5 — LangGraph

Build explicit workflows with state, routing and conditional execution.

Level 6 — Multi-Agent

Separate responsibilities into specialized agents only when the application actually needs them.

Level 7 — Production Evaluation

Measure retrieval quality, token usage, latency, cost, failures and security behavior.

30. Final Production Architecture

CLIENT | v Node.js API | Authentication | Authorization | v Supervisor Agent | +--------------+--------------+ | | | v v v SQL Agent RAG Agent Tool Agent | | | MySQL Hybrid Search External APIs | +--------+--------+ | | BM25 Vector | | +--------+--------+ | Reranker | Context Builder | LLM Agent | Final Answer

Good Practices vs Bad Practices

Good Practice	Bad Practice
Use SQL for structured facts	Use vector search for every request
Use hybrid retrieval	Assume vector search solves every search problem
Filter permissions before context	Tell the LLM not to reveal private data
Use specialized agents	Build one giant autonomous agent
Evaluate retrieval quality	Judge quality from a few demo questions
Optimize context	Send every retrieved chunk to the model

FAQ: Multi-Agent RAG, Vector Search and LangGraph

1. Is MySQL good for RAG vector search?

MySQL can be useful when your application already relies heavily on relational data and your MySQL deployment provides the vector capabilities required by your workload. For larger or specialized vector workloads, dedicated vector databases may be more appropriate. The correct choice depends on scale, indexing, latency, operational requirements and existing architecture.

2. Is vector search better than BM25?

Neither is universally better. Vector search is strong for semantic similarity, while BM25 and keyword search are useful for exact terms, identifiers and lexical matching. Hybrid retrieval combines their strengths.

3. Should every RAG system use LangGraph?

No. A simple RAG pipeline may not need a graph workflow. LangGraph becomes more useful when you need multiple agents, state, conditional routing, retries, loops or complex tool workflows.

4. How many chunks should I retrieve?

There is no universal number. Start with a small candidate set, evaluate retrieval quality, then tune top-K and reranking using your real dataset.

5. What is the best chunk size?

Chunk size depends on the content. Documentation, FAQs, source code and legal documents often require different strategies. Measure retrieval quality instead of blindly using one fixed size.

6. Should I use a multi-agent architecture from day one?

Usually, start with a simple RAG pipeline. Introduce agents when you have clear responsibilities that benefit from routing, tool use or independent workflows.

7. Can an AI agent directly access banking data?

An agent can work with authorized banking or financial data through controlled application tools, but authorization and security checks must remain in the backend. The language model should not be the security boundary.

8. How do I reduce RAG token costs?

Use better chunking, metadata filtering, smaller top-K values, reranking, deduplication, context compression and appropriate models. The goal is high-quality context rather than maximum context.

9. What is hybrid RAG?

Hybrid RAG combines multiple retrieval methods, commonly lexical search such as BM25 and semantic vector search, followed by result merging and optionally reranking.

10. What should I learn first?

Learn basic RAG first, then embeddings and vector search, followed by BM25 and hybrid retrieval, metadata filtering, tool calling, LangGraph workflows and finally multi-agent orchestration.

Conclusion

Modern RAG applications are moving beyond a simple embed → search → prompt architecture. A production-ready AI system can combine Node.js, LangChain, LangGraph, MySQL, SQL queries, BM25, vector search, hybrid retrieval, reranking, metadata filtering, tools and specialized agents.

The most important optimization is not simply adding more agents or larger models. Build the smallest architecture that solves the actual problem, keep structured data in structured storage, use vector retrieval for semantic knowledge, use BM25 for lexical matching, enforce authorization before retrieval, and measure retrieval quality with real-world test cases.

For developers building AI SaaS, enterprise assistants, customer-support systems or internal knowledge platforms, this architecture provides a strong foundation for scalable and maintainable RAG systems.

Next Step: Build a small Node.js RAG prototype first: MySQL + document chunks + embeddings + hybrid search. Once retrieval quality is stable, add LangGraph routing, tools and specialized agents.