Hello everyone!
In today's video we will learn about evaluators and haystack.
What are evaluators?
Haystack has tools needed to evaluate entire pipeline or individual components like retrievers, readers,
and generators.
We can use evaluation and its results to judge how well the system performs, compare the performance
of different models, and identify underperforming components.
End to end evaluation checks how the full pipeline is used and evaluates only the final outputs.
The pipeline is approached as a black box.
What are different types of evaluation?
Model based evaluation.
Model based evaluation uses less with prompt instructions or smaller fine tuned models to score aspects
of pipelines.
Output.
Statistical evaluation.
It requires no model and is thus a more lightweight way to score pipeline outputs.
Most statistical evaluators require ground truth label, such as documents relevant to the query or
the expected answer.
We will not cover statistical evaluation as part of this video.
We'll cover this in a more advanced tutorial in the Advanced Topics section.
Model based evaluation.
Model based evaluation and Hastac uses a language model to check the results of a pipeline.
This method is easy to use because it usually doesn't need labels for the outputs.
It's often used with retrieval augmented generative pipelines, but can work with any pipelines.
Large language model.
A common strategy for model based evaluation involves using a large language model, such as OpenAI's
ChatGPT GPT models as an evaluator model, often referred to as golden model, small and cross encoder
models.
These models can calculate semantic similarity.
This method of using small encoder models as evaluator is faster and cheaper to run, but is less flexible
in terms of what aspect you can evaluate.
What are different types of model based evaluation?
Faithfulness evaluator also known as low as a judge.
Faithfulness, also called groundedness, evaluates to what extent a generated answer is based on the
retrieved documents.
An L is used to extract statements from the answer and check the faithfulness of each statements separately.
If the answer is not based on the documents, the answer or the least part of it is called a hallucination.
Another type of model based evaluation is sass evaluator, also known as semantic answer similarity.
Semantic answer similarity uses transformer based cross encoder architecture to evaluate the semantic
similarity of two answers, rather than their lexical overlap.
Context relevance.
Evaluator context relevance refers to how relevant the retrieved documents are to the query, and is
used to judge that aspect.
It first extracts statement from documents and then checks how many of them are relevant for answering
the query.
There are other set of evaluators that haystack provides.
I have listed just three of them and will go through each of them in detail in this video, but you
can check out other evaluators that haystack has offered in their documentation.
Let's take a.