Lenny RAG / EigenPM Explorer
Knowledge-graph RAG over 297 Lenny's Podcast transcripts, plus a 3D PCA map of those guests you can place yourself inside.
Problem
- How do you turn 297 hour-long conversations into coordinates where "close" means "thinks alike"?
- Entities cluster by topic, an AI cluster here, a B2B cluster there; I wanted a map of how people think.
Approach
- The retrieval half is built on top of traversaal-ai/lennyhub-rag, an open-source LightRAG + Qdrant pipeline.
- I built the guest space two ways from the extracted knowledge graph: from what people talk about (roughly 1,700 entities) and from how they connect ideas (the keywords on relationship edges).
- TF-IDF over 604 relationship keywords, each used at least 15 times across the corpus, reduced to 20 principal components with a Varimax rotation so the axes stay interpretable.
- A short quiz maps your answers into the same 20 dimensions and drops you on the map beside the guests you cluster with.
Outcome
- The relationship space explained meaningfully more variance, 25.9% vs. 14.8% in the top 20 components, and it separated guests by philosophy rather than subject.
- A 3D map of those 297 guests projected into a psychographic space, live as Latent Lenny.
- The retrieval half shipped without a formal eval suite.
Two halves, one corpus. The base is 297 transcripts of Lenny’s Podcast. Half one is a knowledge-graph RAG system that answers product questions over them with source attribution. Half two is EigenPM Explorer, a 3D map of those 297 guests projected into a psychographic space, live as Latent Lenny.
The retrieval half is built on top of traversaal-ai/lennyhub-rag, an open-source LightRAG + Qdrant pipeline. My additions on that side: an LLM-based cleaner that strips sponsor reads and boilerplate before indexing, and an ingestion path from Lenny’s official MCP server that added 645 more documents, 349 of them newsletter posts, with a refresh script that syncs only what changed. Re-indexing that larger corpus is still an open item.
The map is the part I would defend in a seminar. The question behind it: how do you turn 297 hour-long conversations into coordinates where “close” means “thinks alike”? I built the guest space two ways from the extracted knowledge graph: from what people talk about (roughly 1,700 entities) and from how they connect ideas (the keywords on relationship edges). Entities cluster by topic, an AI cluster here, a B2B cluster there; I wanted a map of how people think. The relationship space explained meaningfully more variance, 25.9% vs. 14.8% in the top 20 components, and it separated guests by philosophy rather than subject, so that is the space the map lives in.
The shipped model is small on purpose. TF-IDF over 604 relationship keywords, each used at least 15 times across the corpus, reduced to 20 principal components with a Varimax rotation so the axes stay interpretable. Twenty components hold about a quarter of the variance, which the app’s own limitations note calls substantial information loss; adding entity features back made the space worse, not better. The 3D view shows the first three components; distance and similarity use all 20.
Then a Three.js scene you can rotate, slice and click through. Hyperslice controls fix a hidden dimension at a value and fade everything far from it. A short quiz maps your answers into the same 20 dimensions and drops you on the map beside the guests you cluster with, plus a percentile for how far you sit from any one of them.
The honest caveat. The retrieval half shipped without a formal eval suite. Building one — retrieval quality and groundedness, measured rather than assumed — is the open to-do, and the write-up will land here when it does.