Most AI-powered search today depends on cloud services, which creates two major challenges: data exposure and vendor lock-in. Sensitive information often leaves local environments, and organizations l… See more

Most AI-powered search today depends on cloud services, which creates two major challenges: data exposure and vendor lock-in. Sensitive information often leaves local environments, and organizations lose control over how their data is stored, processed, and queried. This workshop presents a practical alternative: building privacy-first vector search pipelines using containerized infrastructure and locally deployed Large Language Models (LLMs). We will explore multiple aspects of designing Retrieval-Augmented Generation (RAG) systems that are fully vector-database agnostic, allowing teams to choose or switch embedding stores without being tied to a single vendor. By running local LLMs such as Qwen3, GPT-OSS, etc. using tools like Ollama and OpenCode, teams can support large-context queries (100K+ tokens) while keeping all data entirely within their own environments. The session will cover local LLM setup, embedding workflows, indexing strategies, query orchestration, and secure inference patterns. We will also demonstrate how to ground responses using selective web retrieval for up-to-date context without compromising privacy, data sovereignty, or regulatory requirements. Attendees will leave with practical insights from real-world implementations, enabling them to design secure, offline-capable, privacy-first GenAI search platforms suitable for enterprise, regulated, and edge environments. Sobre Rudraksh Karpe: Rudraksh is a Forward Deployed Engineer (FDE) at Simplismart, where he builds solutions focused on high-performance AI inference. He previously worked as an AI Engineer at ZS Associates. He is a two-time Google Summer of Code participant with the openSUSE Project and is currently serving as a mentor for the GSoC 2026 term. He has presented internationally at conferences such as KubeCon EU, PyTorch Conference EU, OpenSearch Korea, PyCon US, PyCon JP, openSUSE Conference (Germany), Early Adopter Tech Summit (Florida), and openSUSE Asia Summit (Tokyo, Japan), focusing on GenAI, open-source, and cloud-native technologies. Sobre Hrittik Roy: Hrittik is a Platform Advocate at vCluster Labs and a CNCF Ambassador with expertise in cloud native technologies and open source communities. He has contributed extensively to developer advocacy, technical writing, and community engagement. Hrittik has been a keynote speaker at events such as the MCP Dev Summits and has delivered talks at Kubernetes Community Days, Open Source Summits, and more. He has also served as a Program Committee member for several KubeCons and CloudNativeCons. Show less