Embedding

Overview

This module handles the embedding phase of the data pipeline. It reads previously generated JSON chunk files from the filesystem, reconstructs them into LangChain Document objects, and uploads them in batches to a Qdrant vector store for semantic search.

To run embedding phase:

python -m data.embedding.embedder

Functions

embedder

Reads chunk files for specified data sources, reconstructs them into document objects, and inserts them into the vector database in batches.

Logic and Data Flow:

1. Directory Mapping: Maps each source to its respective subdirectory inside {output_dir}/chunks/.

2. Document Reconstruction: Iterates through all .json files in the source directory, reads the content, and reconstructs LangChain Document objects using the page_content, metadata, and id.

3. Batch Insertion: To optimize memory and network usage, documents are pushed to the Qdrant vector store in batches of 5,000 using vector_store.add_documents().