A now famous study from MIT<\/a> is stating how 95% of organizations fail to get ROI within the 6 months of their “AI projects”. Although we could argue about the relevancy of the study and what it actually measured, one of the key element to have a successful implementation is measurement.
An old BI principle is to know your KPI, what it really measures but also when it fails to measure. For example if you would use the speedometer on your dashboard’s car to measure the speed at which you are going, you’d be right as long as the wheels are touching the ground. So with that in mind, let’s see how we can create smart and reliable retrieval.
<\/p>\n\n\n\n
\n\n\n\n
From Hybrid to Adaptive<\/h2>\n\n\n\n
Hybrid search significantly improves retrieval quality by combining dense semantic vectors with sparse lexical signals. However, real-world queries vary:<\/p>\n\n\n\n
- \n
- Some are factual<\/strong>, asking for specific names, numbers, or entities.<\/li>\n\n\n\n
- Others are conceptual<\/strong>, exploring ideas, reasons, or relationships.<\/li>\n<\/ul>\n\n\n\n
A single static weighting between dense and sparse methods cannot perform optimally across all query types.<\/p>\n\n\n\n
Adaptive RAG<\/strong> introduces a lightweight classifier that analyzes each query to determine its type and dynamically adjusts the hybrid weights before searching.
For example:<\/p>\n\n\n\nQuery Type<\/th> Example<\/th> Dense Weight<\/th> Sparse Weight<\/th><\/tr><\/thead> Factual<\/td> \u201cWho founded PostgreSQL?\u201d<\/td> 0.3<\/td> 0.7<\/td><\/tr> Conceptual<\/td> \u201cHow does PostgreSQL handle concurrency?\u201d<\/td> 0.7<\/td> 0.3<\/td><\/tr> Exploratory<\/td> \u201cTell me about Postgres performance tuning\u201d<\/td> 0.5<\/td> 0.5<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n This dynamic weighting ensures that each search leverages the right signals:<\/p>\n\n\n\n
- \n
- Sparse when exact matching<\/strong> matters.<\/li>\n\n\n\n
- Dense when semantic similarity<\/strong> matters.<\/li>\n<\/ul>\n\n\n\n
Under the hood, our
AdaptiveSearchEngine<\/code> wraps dense and sparse retrieval modules. Before executing, it classifies the query, assigns weights, and fuses the results via a weighted Reciprocal Rank Fusion (RRF)<\/strong>, giving us the best of both worlds \u2014 adaptivity without complexity.<\/p>\n\n\n\n
\n\n\n\nConfidence-Driven Retrieval<\/h2>\n\n\n\n
Once we make retrieval adaptive, the next challenge is trust<\/strong>. How confident are we in the results we just returned?<\/p>\n\n\n\n
Confidence from Classification<\/h3>\n\n\n\n
Each query classification includes a confidence score<\/strong> (e.g., 0.92 \u201cfactual\u201d vs 0.58 \u201cconceptual\u201d).
When classification confidence is low, Adaptive RAG defaults to a balanced retrieval (dense 0.5, sparse 0.5) \u2014 avoiding extreme weighting that might miss relevant content.<\/p>\n\n\n\nConfidence from Retrieval<\/h3>\n\n\n\n
We also compute confidence based on retrieval statistics:<\/p>\n\n\n\n
- \n
- The similarity gap between the first and second ranked results (large gap = high confidence).<\/li>\n\n\n\n
- Average similarity score of the top-k results.<\/li>\n\n\n\n
- Ratio of sparse vs dense agreement (when both find the same document, confidence increases).<\/li>\n<\/ul>\n\n\n\n
These metrics are aggregated into a normalized confidence score<\/strong> between 0 and 1:<\/p>\n\n\n
\ndef compute_confidence(top_scores, overlap_ratio):\n sim_conf = min(1.0, sum(top_scores[:3]) \/ 3)\n overlap_conf = 0.3 + 0.7 * overlap_ratio\n return round((sim_conf + overlap_conf) \/ 2, 2)\n\n<\/pre><\/div>\n\n\n
If confidence < 0.5, the system triggers a fallback strategy<\/strong>:<\/p>\n\n\n\n
- \n
- Expands
top_k<\/code> results (e.g., from 10 \u2192 30).<\/li>\n\n\n\n- Broadens search to both dense and sparse equally.<\/li>\n\n\n\n
- Logs the event for later evaluation.<\/li>\n<\/ul>\n\n\n\n
The retrieval API now returns a structured response:<\/p>\n\n\n
\n{\n "query": "When was PostgreSQL 1.0 released?",\n "query_type": "factual",\n "confidence": 0.87,\n "precision@10": 0.8,\n "recall@10": 0.75\n}\n\n<\/pre><\/div>\n\n\nThis allows monitoring not just what<\/em> was retrieved, but how sure<\/em> the system is. Enabling alerting, adaptive reruns, or downstream LLM prompt adjustments (e.g., “Answer cautiously” when confidence < 0.6).<\/p>\n\n\n\n
\n\n\n\nEvaluating Quality with nDCG<\/h2>\n\n\n\n
Precision and recall are fundamental metrics for retrieval systems, but they don\u2019t consider the order<\/strong> of results. If a relevant document appears at rank 10 instead of rank 1, the user experience is still poor even if recall is high.<\/p>\n\n\n\n
That\u2019s why we now add nDCG@k (normalized Discounted Cumulative Gain)<\/strong> \u2014 a ranking-aware measure that rewards systems for ordering relevant results near the top.<\/p>\n\n\n\n
The idea:<\/p>\n\n\n\n
- \n
- DCG@k<\/strong> evaluates gain by position:<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/10\/image-5.png\")
<\/figure>\n<\/div>\n\n\n- \n
- nDCG@k<\/strong> normalizes this against the ideal order (IDCG):<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/10\/image-6.png\")
<\/figure>\n<\/div>\n\n\nA perfect ranking yields nDCG = 1.0. Poorly ordered but complete results may still have high recall, but lower nDCG.<\/p>\n\n\n\n
In practice, we calculate nDCG@10 for each query and average it over the dataset.
Our evaluation script (lab\/04_evaluate\/metrics.py<\/code>) integrates this directly:<\/p>\n\n\n\nfrom evaluation import ndcg_at_k\n\nscore = ndcg_at_k(actual=relevant_docs, predicted=retrieved_docs, k=10)\nprint(f"nDCG@10: {score:.3f}")\n\n<\/pre><\/div>\n\n\nResults on the Wikipedia dataset (25K articles)<\/h3>\n\n\n\n
Method<\/th> Precision@10<\/th> Recall@10<\/th> nDCG@10<\/th><\/tr><\/thead> Dense only<\/td> 0.61<\/td> 0.54<\/td> 0.63<\/td><\/tr> Hybrid fixed weights<\/td> 0.72<\/td> 0.68<\/td> 0.75<\/td><\/tr> Adaptive (dynamic)<\/strong><\/td> 0.78<\/strong><\/td> 0.74<\/strong><\/td> 0.82<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n These results confirm that adaptive weighting not only improves raw accuracy but also produces better-ranked results<\/strong>, giving users relevant documents earlier in the list.<\/p>\n\n\n\n
\n\n\n\nImplementation in our LAB<\/h2>\n\n\n\n
You can explore the implementation in the GitHub repository:<\/p>\n\n\n
\ngit clone https:\/\/github.com\/boutaga\/pgvector_RAG_search_lab\ncd pgvector_RAG_search_lab\n\n<\/pre><\/div>\n\n\n
Key components:<\/p>\n\n\n\n
- \n
lab\/04_search\/adaptive_search.py<\/code> \u2014 query classification, adaptive weights, confidence scoring.<\/li>\n\n\n\nlab\/04_evaluate\/metrics.py<\/code> \u2014 precision, recall, and nDCG evaluation.<\/li>\n\n\n\n- Streamlit UI (
streamlit run streamlit_demo.py<\/code>) \u2014 visualize retrieved chunks, scores, and confidence in real time.<\/li>\n<\/ul>\n\n\n\nExample usage:<\/p>\n\n\n
\npython lab\/04_search\/adaptive_search.py --query "Who invented SQL?"\n\n<\/pre><\/div>\n\n\n
Output:<\/p>\n\n\n
\nQuery type: factual (0.91 confidence)\nDense weight: 0.3 | Sparse weight: 0.7\nPrecision@10: 0.82 | Recall@10: 0.77 | nDCG@10: 0.84\n\n<\/pre><\/div>\n\n\n
This feedback loop closes the gap between research and production \u2014 making RAG not only smarter but measurable.<\/p>\n\n\n\n
What is \u201cRelevance\u201d?<\/h2>\n\n\n\n
When we talk about precision<\/strong>, recall<\/strong>, or nDCG<\/strong>, all three depend on one hidden thing:<\/p>\n\n\n\n
\n
a ground truth<\/em> of which documents are relevant for each query.<\/strong><\/p>\n<\/blockquote>\n\n\n\n
There are two main ways<\/strong> to establish that ground truth:<\/p>\n\n\n\n
Approach<\/th> Who decides relevance<\/th> Pros<\/th> Cons<\/th><\/tr><\/thead> Human labeling<\/strong><\/td> Experts mark which documents correctly answer each query<\/td> Most accurate; useful for benchmarks<\/td> Expensive and slow<\/td><\/tr> Automated or LLM-assisted labeling<\/strong><\/td> An LLM (or rules) judges if a retrieved doc contains the correct answer<\/td> Scalable and repeatable<\/td> Risk of bias \/ noise<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n In some business activity you are almost forced to use human labeling because the business technicalities are so deep that automating it is hard. Labeling can be slow and expensive for a business but I learned that it also is a way to introduce change management towards AI workflow by enabling key employees of the company to participate and build a solution with their expertise and without going through a harder project of asking to an external organization to create specific business logic into a software that was never made to handle it in the first place. As a DBA, I witnessed business logic move away from databases towards ORMs and application code and this time the business logic is going towards AI workflow. Starting this human labeling project my be the first step towards it and guarantees solid foundations.
Managers need to keep in mind that AI workflows are not just a technical solution, they are social-technical framework to allow organizational growth. You can’t just ship an AI chatbot into an app and expect 10x returns with minimal effort, this is a simplistic state of mind that already cost billions according the MIT study.
In a research setup (like yourpgvector_RAG_search_lab<\/code>), you can mix both<\/strong> approach:<\/p>\n\n\n\n- \n
- Start with a seed dataset<\/strong> of
(query, relevant_doc_ids)<\/code> pairs (e.g. small set labeled manually).<\/li>\n\n\n\n- Use the LLM to extend or validate<\/strong> relevance judgments automatically.<\/li>\n<\/ul>\n\n\n\n
For example:<\/p>\n\n\n
\nprompt = f"""\nQuery: {query}\nDocument: {doc_text[:2000]}\nIs this document relevant to answering the query? (yes\/no)\n"""\nllm_response = openai.ChatCompletion.create(...)\nlabel = llm_response['choices'][0]['message']['content'].strip().lower() == 'yes'\n\n<\/pre><\/div>\n\n\nThen you store that in a simple table or CSV:<\/p>\n\n\n\n
query_id<\/th> doc_id<\/th> relevant<\/th><\/tr><\/thead> 1<\/td> 101<\/td> true<\/td><\/tr> 1<\/td> 102<\/td> false<\/td><\/tr> 2<\/td> 104<\/td> true<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\nPrecision & Recall in Practice<\/h2>\n\n\n\n
Once you have that table of true relevances, you can compute:<\/p>\n\n\n\n
- \n
- Precision@k<\/strong> \u2192 \u201cOf the top k<\/em> documents I retrieved, how many were actually relevant?\u201d <\/li>\n\n\n\n
- Recall@k<\/strong> \u2192 \u201cOf all truly relevant documents, how many did I retrieve in my top k<\/em>?\u201d <\/li>\n<\/ul>\n\n\n\n
They\u2019re correlated but not the same:<\/p>\n\n\n\n
- \n
- High precision<\/strong> \u2192 few false positives.<\/li>\n\n\n\n
- High recall<\/strong> \u2192 few false negatives.<\/li>\n<\/ul>\n\n\n\n
For example:<\/p>\n\n\n\n
Query<\/th> Retrieved docs (top 5)<\/th> True relevant<\/th> Precision@5<\/th> Recall@5<\/th><\/tr><\/thead> \u201cWho founded PostgreSQL?\u201d<\/td> [d3, d7, d9, d1, d4]<\/td> [d1, d4]<\/td> 0.4<\/td> 1.0<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n You got both relevant docs (good recall = 1.0), but only 2 of the 5 retrieved were correct (precision = 0.4).<\/p>\n\n\n\n
\n\n\n\nWhy nDCG is Needed<\/h2>\n\n\n\n
Precision and recall only measure which<\/em> docs were retrieved, not where they appeared in the ranking<\/em>.<\/p>\n\n\n\n
nDCG@k<\/strong> adds ranking quality<\/em>:<\/p>\n\n\n\n
- \n
- Each relevant document gets a relevance grade<\/strong> (commonly 0, 1, 2 \u2014 irrelevant, relevant, highly relevant).<\/li>\n\n\n\n
- The higher it appears in the ranked list, the higher the gain.<\/li>\n<\/ul>\n\n\n\n
So if a highly relevant doc is ranked 1st, you get more credit than if it\u2019s ranked 10th.<\/p>\n\n\n\n
In your database<\/strong>, you can store relevance grades in a table like:<\/p>\n\n\n\n
query_id<\/th> doc_id<\/th> rel_grade<\/th><\/tr><\/thead> 1<\/td> 101<\/td> 2<\/td><\/tr> 1<\/td> 102<\/td> 1<\/td><\/tr> 1<\/td> 103<\/td> 0<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n Then your evaluator computes:<\/p>\n\n\n
\nimport math\n\ndef dcg_at_k(relevances, k):\n return sum((2**rel - 1) \/ math.log2(i+2) for i, rel in enumerate(relevances[:k]))\n\ndef ndcg_at_k(actual_relevances, k):\n ideal = sorted(actual_relevances, reverse=True)\n return dcg_at_k(actual_relevances, k) \/ dcg_at_k(ideal, k)\n\n<\/pre><\/div>\n\n\n
You do need to keep track of rank<\/strong> (the order in which docs were returned).
In PostgreSQL, you could log that like:<\/p>\n\n\n\nquery_id<\/th> doc_id<\/th> rank<\/th> score<\/th> rel_grade<\/th><\/tr><\/thead> 1<\/td> 101<\/td> 1<\/td> 0.92<\/td> 2<\/td><\/tr> 1<\/td> 102<\/td> 2<\/td> 0.87<\/td> 1<\/td><\/tr> 1<\/td> 103<\/td> 3<\/td> 0.54<\/td> 0<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n Then it\u2019s easy to run SQL to evaluate:<\/p>\n\n\n
\nSELECT query_id,\n SUM((POWER(2, rel_grade) - 1) \/ LOG(2, rank + 1)) AS dcg\nFROM eval_results\nWHERE rank <= 10\nGROUP BY query_id;\n\n<\/pre><\/div>\n\n\n
In a real system (like your Streamlit or API demo), you can:<\/p>\n\n\n\n
- \n
- Log each retrieval attempt<\/strong> (query, timestamp, ranking list, scores, confidence).<\/li>\n\n\n\n
- Periodically recompute metrics<\/strong> (precision, recall, nDCG) using a fixed ground-truth set.<\/li>\n<\/ul>\n\n\n\n
This lets you track if tuning (e.g., changing dense\/sparse weights) is improving performance.<\/p>\n\n\n\n
Structure of your evaluation log table could be:<\/p>\n\n\n\n
run_id<\/th> query_id<\/th> method<\/th> rank<\/th> doc_id<\/th> score<\/th> confidence<\/th> rel_grade<\/th><\/tr><\/thead> 2025-10-12_01<\/td> 1<\/td> adaptive_rrf<\/td> 1<\/td> 101<\/td> 0.92<\/td> 0.87<\/td> 2<\/td><\/tr> 2025-10-12_01<\/td> 1<\/td> adaptive_rrf<\/td> 2<\/td> 102<\/td> 0.85<\/td> 0.87<\/td> 1<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n From there, you can generate:<\/p>\n\n\n\n
- \n
- nDCG@10 trend over runs<\/strong> (e.g., in Prometheus or Streamlit chart)<\/li>\n\n\n\n
- Precision vs Confidence correlation<\/strong><\/li>\n\n\n\n
- Recall improvements per query type<\/strong><\/li>\n<\/ul>\n\n\n\n
\u26a0\ufe0f Note: While nDCG is a strong metric for ranking quality, it\u2019s not free from bias. Because it normalizes per query, easier questions (with few relevant documents) can inflate the average score. In our lab, we mitigate this by logging both raw DCG and nDCG, and by comparing results across query categories (factual vs conceptual vs exploratory). This helps ensure improvements reflect true retrieval quality rather than statistical artifacts.<\/em><\/p>\n\n\n\n
\n\n\n\nHuman + LLM Hybrid Evaluation (Practical Middle Ground)<\/h2>\n\n\n\n
For your PostgreSQL lab setup:<\/p>\n\n\n\n
- \n
- Label a small gold set<\/strong> manually (e.g., 20\u201350 queries \u00d7 3\u20135 relevant docs each).<\/li>\n\n\n\n
- For larger coverage, use the LLM as an auto-grader<\/strong>.
You can even use self-consistency<\/em>: ask the LLM to re-evaluate relevance twice and keep consistent labels only.<\/li>\n<\/ul>\n\n\n\nThis gives you a semi-automated evaluation dataset<\/strong>, good enough to monitor:<\/p>\n\n\n\n
- For larger coverage, use the LLM as an auto-grader<\/strong>.
- Precision vs Confidence correlation<\/strong><\/li>\n\n\n\n
- Periodically recompute metrics<\/strong> (precision, recall, nDCG) using a fixed ground-truth set.<\/li>\n<\/ul>\n\n\n\n
- The higher it appears in the ranked list, the higher the gain.<\/li>\n<\/ul>\n\n\n\n
- High recall<\/strong> \u2192 few false negatives.<\/li>\n<\/ul>\n\n\n\n
- Recall@k<\/strong> \u2192 \u201cOf all truly relevant documents, how many did I retrieve in my top k<\/em>?\u201d <\/li>\n<\/ul>\n\n\n\n
- Use the LLM to extend or validate<\/strong> relevance judgments automatically.<\/li>\n<\/ul>\n\n\n\n
- nDCG@k<\/strong> normalizes this against the ideal order (IDCG):<\/li>\n<\/ul>\n\n\n
- Expands
- Dense when semantic similarity<\/strong> matters.<\/li>\n<\/ul>\n\n\n\n
- Others are conceptual<\/strong>, exploring ideas, reasons, or relationships.<\/li>\n<\/ul>\n\n\n\n