Back to writing
AI & LLMs·Nov 2025·8 min read

Beyond Palantir: What I Learned Building a RAG System with Python

I'd built a RAG system inside Palantir's platform, but it felt like a black box. So I rebuilt one independently using PostgreSQL, pgvector, BM25, FastAPI, and Redis, treating it as a learning journey, not a tutorial.

OK
Ofer Kulka
Staff Data & Analytics Engineer

Foundation and Discovery

At first, RAG systems seemed straightforward: ingest documents, split them, embed them, run search queries. But as I dug deeper and worked through materials like the "Advanced LLMs with Retrieval Augmented Generation" course, I came to realise how much complexity hides each step.

Document ingestion and parsing

Modern PDF parsers do not just extract text, they preserve structure and metadata such as reading order, page numbers, tables, formulas, code and even images. This can be brought in as additional context.

Chunking strategies

Two opposing forces: avoid chunks so broad that irrelevant information is captured or that exceed the model's context window, and chunks so small that they lack the context needed for a good response.

Embedding models are not created equal

Some models excel at text, some at images or tables, others handle multiple modalities. Choosing the right one for your use case matters.

Keyword search still matters

Especially for academic papers, where exact terminology ("sleep regression" vs. "sleep disruption") cannot be captured by semantic similarity alone.

LLMs can be tamed

Through prompts, context window engineering, and structured outputs. Structured outputs (e.g. data contracts with Instructor) act as guard rails, minimising hallucinations by setting an expected schema for the response.

I had a prototype running locally of Dave Ebbelaar's PostgreSQL + Pgvectorscale approach. It performed well but it was incomplete: no keyword search, no proper UI.

Streamlit UI for the locally deployed RAG system
Fig 01, Local deployment in Streamlit

Hybrid Search Breakthrough

Pure vector search was powerful but limited. The knowledge base was academic papers, where precision matters. It returned chunks that were contextually similar to my queries but missed where specific wording mattered, "sleep regression" is not the same as "sleep disruption" and the system couldn't reliably make that distinction.

BM25 (Best Matching 25) is an evolution of TF-IDF that measures how important a word is as a function of how often it appears in a document. It improves on TF-IDF in two key ways:

  • Word frequency bias, BM25 diminishes the returns each repeated mention contributes. The 11th mention matters less than the first 10.
  • Document length normalisation, long documents naturally contain more words and more keyword hits. BM25 penalises documents that are longer than average.

The challenge wasn't adding BM25, it was combining it with vector search meaningfully. My first attempt was a simple weighted average (70% vector, 30% BM25). The responses felt inconsistent.

That's when I discovered Reciprocal Rank Fusion (RRF). Unlike a weighted average, RRF focuses on rankings only. It merges the top results from each search engine into a single balanced list. Because it's model-agnostic, no score normalisation is required.

The result was a hybrid retrieval system that combined semantic search with the precision of keyword search. Responses became richer and more precise, suddenly the system returned answers closer to how a human would search and interpret documents. It became apparent that keyword search often dominates the ranking.

Hybrid retrieval search results panel showing vector + BM25 ranking via RRF
Fig 02, Hybrid search results: semantic + BM25 via RRF

The Architecture Shift: "Production-Ready"

With a functioning hybrid search in place, I started thinking bigger: what would a production-ready version look like? This is where I decided to experiment with AI-assisted code development, "Vibe Coding."

The "Vibe Coding" Experiment

My tool of choice, Claude Code.

Instead of using Claude just for writing code, I treated it as a technical conversation partner. I'd describe what I wanted and what I expected; Claude would suggest approaches, identify gaps, and help refine solutions.

The process was iterative and required overcoming a significant learning curve. A few hard-won lessons:

  • Ask Claude to be critical of your ideas and suggestions.
  • Be critical of Claude's suggestions or recommendations.
  • Do not expect Claude to understand the entire picture, have consistent check points to see what it understands.
  • Use Plan Mode to devise a blueprint first, iterate, agree, then execute. Otherwise the result might not be what you expect, or worse, Claude might overwrite your code.
  • Save your work at every step through version control.
  • Validate with comprehensive testing before deployment.

I was not yet familiar with Plan Mode, a built-in mode that prohibits any edits to the code. I'd ask Claude something or make a statement, then return to find a repository with multiple edits and errors. A hard lesson, and one that taught me to always plan first, validate, test often, and use version control.

While there was a significant productivity boost, Vibe Coding came with its own challenges: apps that wouldn't start, running code overwritten, recommendations that missed the mark. Agentic coding tools can accelerate development, but you need to stay engaged and critical. At the moment, the tool is only as effective as the human at the keyboard.

FastAPI: Separating Concerns

The first major architectural change was to separate the application into a frontend and a backend. The original RAG system was running behind a Streamlit app for interactivity. Instead of a monolith:

  • Pydantic schemas enforced data contracts and validation.
  • Dedicated endpoints for all operations.
  • Comprehensive error handling.
  • Streamlit became a lightweight UI client that called the backend service.

Redis: The Performance Game-Changer

I'd grown frustrated with query response times on re-runs and asked Claude for help. It suggested Redis, and the implementation delivered three wins:

  • Cost optimisation, mitigates redundant embedding API calls.
  • Latency reduction, responses dropped from seconds to milliseconds.
  • Better UX, repeated queries became near-instant.

The architecture now maintained clear separation of concerns. Pgvectorscale handled document embeddings and metadata; Redis managed query and response caching. Ultimately these additions improved maintainability, testing, error handling, stability, and performance.

Architecture

End-to-end shape: Streamlit on top, FastAPI in the middle, processors / search / services in the core, Pgvectorscale + Redis underneath, OpenAI for embeddings and LLM responses.

┌─────────────────────────────────────────────────────────┐
│                    Frontend Layer                       │
│  Streamlit UI, query input, results, status dashboard  │
└─────────────────────────┬───────────────────────────────┘
                          │ HTTP
┌─────────────────────────▼───────────────────────────────┐
│                     Backend Layer                       │
│  FastAPI, endpoints, request validation                │
└─────────────────────────┬───────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────┐
│                   Core Business Logic                   │
│  Processors      Search           Services              │
│  • Doc parsing   • Vector         • Cache               │
│  • PDF / chunk   • BM25           • LLM / synthesis     │
│                  • RRF hybrid                           │
└─────────────────────────┬───────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────┐
│                  Infrastructure Layer                   │
│  PGvectorscale (vectors, time-series, similarity)       │
│  Redis (query cache, session mgmt)                      │
└─────────────────────────┬───────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────┐
│                  External Services                      │
│  OpenAI, text-embedding-3-small, gpt-4o                │
└─────────────────────────────────────────────────────────┘

Key Learnings

PostgreSQL could double as a vector DB

With Timescale's solution you might not need a specialised store after all.

Indexing matters

Efficient ANN indexes like DiskANN keep searches fast and scalable.

Hybrid search is trickier than it looks

Fusing search engine results requires more than intuition, RRF won out over a naive weighted average.

Plan for resilience

Design for graceful failure, keep running even if Redis is down.

"Tame" the AI

Limit LLM calls. Let AI solve single components inside the application rather than figuring out the whole thing on its own, that's dangerous and expensive.

AI-assisted coding accelerates, doesn't replace

Treat AI as a partner, not a replacement.

The Verdict

Did I replicate Palantir's RAG system? No, that was never the goal. My aim was to understand how these systems really work, to learn to tame LLMs for better outputs, and to practice building agents with Pythonic frameworks. Along the way, I realised AI applications today should be treated as software first: solid engineering practices, calling on LLMs only when they add value, ideally for focused tasks where code alone cannot deliver consistent results.

Palantir's Foundry and Ontology reflect decades of work and polish. Platforms like that are autobahns, they get you to your destination quickly and smoothly. Building from scratch is the opposite: it's hiking a trail. It takes muscle, it's messy, but it rewards you with a deeper connection to the terrain. Once you've walked the trail, you come away with skills and insights that make you more capable wherever you go next.

Ofer Kulka

© 2026 Ofer Kulka · Data you can trust