RAGAS Evaluation: Measuring Faithfulness and Context Precision

A RAG system can fail in two separate places: retrieval brings back the wrong context, or generation ignores or embellishes the right context. A single “answer quality” score hides which one happened. RAGAS was designed to separate them, and that separation is the main reason to use it.

This article covers the evaluation loop only. For building the retrieval side, see semantic chunking and hybrid BM25-vector search.

The metrics, and what each one diagnoses

Names and inputs below follow the current RAGAS metric docs. “Reference” means a human-written correct answer for the question.

How faithfulness is computed

The judge model breaks the response into individual claims and checks each claim against the retrieved context. The score is supported claims divided by total claims, from 0 to 1. An answer with one supported claim and one unsupported claim scores 0.5. Faithfulness says nothing about whether the context itself was correct. A faithful answer built on a stale document is still wrong, which is why you need retrieval metrics alongside it.

How context precision is computed

Each retrieved chunk is judged relevant or not. The score is the mean of precision@k across the ranked list, so a relevant chunk in position one counts for more than the same chunk in position five. In the docs’ own two-chunk example, moving the irrelevant chunk to the top cuts the score from about 1.0 to about 0.5, even though the same chunks were retrieved. That is useful, because it means context precision responds to reranking changes, not just to retrieval changes.

The API changed: check which RAGAS you are reading about

Many RAGAS tutorials still rank well but use APIs that have since moved. As of October 2026, the latest release on PyPI is 0.4.3. The changes that matter are:

• Collections metrics are the recommended API. Import from ragas.metrics.collections and call await metric.ascore(...) (or .score(...)), which returns an object with .value. The older classes in ragas.metrics, used with SingleTurnSample and single_turn_ascore, are deprecated from 0.4 and the docs say they will be removed in 1.0.
• Field names. Early tutorials use question, answer, contexts and ground_truth. Current metrics take user_input, response, retrieved_contexts and reference. The old ground_truths parameter was removed in 0.3.9.
• Renamed classes. The collections class is AnswerRelevancy, while the legacy equivalent was ResponseRelevancy. Legacy context precision came in variants such as LLMContextPrecisionWithReference and LLMContextPrecisionWithoutReference. In collections these are ContextPrecision and ContextUtilization.
• Judge setup. llm_factory(model, client=...) wraps a provider client. The docs list direct support for OpenAI, Anthropic and Google, and LiteLLM for other providers.

Pin the version in your requirements file. An evaluation harness that silently upgrades is a moving ruler.

Offline: a golden set in CI

A golden set is 50–200 real questions, each with a reference answer written or approved by someone who knows the domain. Pull questions from search logs and support tickets rather than inventing them, and include the awkward ones: identifiers, multi-part questions, and questions the corpus cannot answer.

The harness runs every question through your actual pipeline, then scores the outputs. This sketch assumes a rag.answer() coroutine that returns the answer and the list of context strings it used. Model names come from environment variables so the judge is pinned explicitly.

import asyncio, json, os
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import (
    AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness,
)

from myapp import rag  # your pipeline: rag.answer(question) -> (answer, contexts)

client = AsyncOpenAI()
llm = llm_factory(os.environ["RAGAS_JUDGE_MODEL"], client=client)
embeddings = embedding_factory("openai", model=os.environ["RAGAS_EMBED_MODEL"], client=client)

faithfulness = Faithfulness(llm=llm)
relevancy = AnswerRelevancy(llm=llm, embeddings=embeddings)
precision = ContextPrecision(llm=llm)
recall = ContextRecall(llm=llm)

limit = asyncio.Semaphore(4)  # judge calls are rate-limited too

async def score(row: dict) -> dict:
    async with limit:
        answer, contexts = await rag.answer(row["question"])
        f, a, p, r = await asyncio.gather(
            faithfulness.ascore(user_input=row["question"], response=answer,
                                retrieved_contexts=contexts),
            relevancy.ascore(user_input=row["question"], response=answer),
            precision.ascore(user_input=row["question"], reference=row["reference"],
                             retrieved_contexts=contexts),
            recall.ascore(user_input=row["question"], reference=row["reference"],
                          retrieved_contexts=contexts),
        )
        return {"id": row["id"], "faithfulness": f.value, "answer_relevancy": a.value,
                "context_precision": p.value, "context_recall": r.value}

async def main() -> None:
    golden = [json.loads(line) for line in open("golden.jsonl")]
    results = await asyncio.gather(*(score(row) for row in golden))
    json.dump(results, open("ragas-results.json", "w"), indent=2)

asyncio.run(main())

The current docs also describe an experiments-first workflow (the @experiment decorator with a dataset backend) for tracking runs over time. The plain loop above does the same scoring with fewer moving parts, which makes it easier to drop into an existing CI job.

Gate on regressions, not on magic numbers

There is no universally “good” faithfulness score. It depends on the judge model, the prompts and how hard your questions are. Record a baseline from the current production configuration, then fail the build when a change drops a metric’s mean by more than a tolerance you choose. Also store per-question scores, because an average that holds steady can hide ten questions that collapsed.

import json, statistics, sys

TOLERANCE = 0.03  # your choice; tighten as the golden set grows
baseline = json.load(open("ragas-baseline.json"))      # {"faithfulness": 0.91, ...}
results = json.load(open("ragas-results.json"))

failed = False
for metric, base in baseline.items():
    values = [r[metric] for r in results if r[metric] is not None]
    current = statistics.mean(values)
    print(f"{metric}: {current:.3f} (baseline {base:.3f})")
    if current < base - TOLERANCE:
        failed = True

sys.exit(1 if failed else 0)

Online: reference-free metrics on sampled traffic

Production traffic has no reference answers, so only the reference-free metrics apply: Faithfulness, AnswerRelevancy and ContextUtilization. Every score is at least one extra LLM call, so sample rather than scoring every request. A fixed percentage plus every response that a user flagged is a sensible start.

Score asynchronously from logged traces, never in the request path. That requires logging the question, the exact context strings sent to the model and the response, together with the model and prompt versions. That logging discipline is the same one I describe for integrating OpenAI and Claude APIs into enterprise applications. Watch trends and sudden drops. A faithfulness dip that coincides with a content import or an embedding model change tells you exactly where to look.

Reading the scores together

Keep the judge honest

• Calibrate against humans. Hand-label 30–50 responses and check that the judge agrees with your labels before you trust its trend lines.
• Pin the judge model. Changing the judge changes every score. Re-baseline whenever you upgrade it.
• Expect noise. LLM judges are not perfectly repeatable. Compare means over the whole set, and re-run before acting on a small difference.
• Do not let the judge grade itself. Using the same model as both generator and judge invites shared blind spots. Use a different, ideally stronger, model for judging where budget allows.

Agent pipelines need more than RAG metrics. If your system calls tools, measure tool-call correctness separately. Strict tool schemas and error recovery loops covers the validation side.

Questions people actually ask

Can I evaluate a RAG pipeline without ground truth answers?

Partly. Faithfulness, answer relevancy and context utilisation need only the question, the retrieved contexts and the response, so they work on live traffic. Context recall and reference-based context precision need a reference answer, which is why a small human-approved golden set is still worth building.

What is the difference between faithfulness and answer correctness?

Faithfulness checks the answer against the retrieved context; correctness checks it against the true answer. An answer can be perfectly faithful to an outdated document and still be wrong.

Why do my RAGAS scores change between runs with no code change?

The metrics are computed by an LLM judge, which is not perfectly repeatable, and your pipeline’s own generation varies too. Pin both models, use a large enough question set, and compare averages against a tolerance rather than reacting to small moves.