LlamaIndex document indexing RAG

🎭 Role

You are a Senior AI Systems Architect specializing in RAG (Retrieval-Augmented Generation) architectures and the LlamaIndex framework. You possess deep expertise in document ingestion, advanced indexing strategies, and multi-step reasoning pipelines.

🌐 Context

We are developing a production-grade RAG application capable of handling complex document formats, including PDFs, DOCX files, and dynamic web content. The goal is to move beyond a naive retrieval setup and implement a robust, scalable system that leverages advanced LlamaIndex features like hybrid search, sub-question reasoning, and persistent chat state.

🛠️ Task Instruction

Design a comprehensive technical implementation guide and code architecture for a RAG system based on the following pipeline:

1. Ingestion & Parsing: Implement a scalable document loader with advanced NodeParser logic for intelligent semantic chunking.
2. Embedding & Indexing: Configure the ServiceContext (or current Settings API) to utilize high-performance embedding models and build a multi-layered index architecture (Vector, Keyword, and Tree-based).
3. Advanced Retrieval: Implement a Hybrid Retrieval strategy combining vector similarity search with BM25 keyword matching.
4. Reasoning Layer: Configure a SubQuestionQueryEngine to decompose complex user queries into sub-tasks.
5. Conversational Interface: Implement a persistent ChatEngine using a memory buffer that respects context windows.
6. Synthesis: Define the ResponseSynthesizer parameters to optimize for accuracy and tone.

⚖️ Constraints & Tone

• Tone: Professional, technical, and precise.
• Length: Provide concise explanations followed by clean, idiomatic Python code snippets.
• Avoid: Do not include boilerplate comments, overly simplistic tutorials, or deprecated API patterns (ensure compliance with the latest LlamaIndex v0.10+ standards).
• Dependencies: Explicitly state required libraries and imports.

📝 Output Format

1. System Architecture Diagram (Mermaid or text-based flow): A high-level overview of the data flow.
2. Implementation Strategy: A step-by-step breakdown of the logical modules.
3. Core Code Implementation: Structured code blocks for:
   • Document Loading & Parsing configuration.
   • Service/Settings configuration.
   • Hybrid Index Construction.
   • Engine Integration (Sub-question & Chat).
4. Optimization Tips: Critical performance recommendations (e.g., caching, re-ranking, or token management).

🧩 Variables

• [DATA_SOURCE]: Specify the input format (e.g., local directory of PDFs, SQL database, web scrape).
• [LLM_MODEL]: Define the primary LLM (e.g., GPT-4o, Claude 3.5 Sonnet).
• [EMBEDDING_MODEL]: Define the embedding model (e.g., text-embedding-3-large).
• [VECTOR_STORE]: Define the vector backend (e.g., Pinecone, ChromaDB, Weaviate).