Tag:
eval
(9)Evaluating Long-Context Question & Answer Systems
Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
An LLM-as-Judge Won't Save The Product—Fixing Your Process Will
Applying the scientific method, building via eval-driven development, and monitoring AI output.
20 Apr 2025 · 5 min · eval llm engineering
AlignEval: Building an App to Make Evals Easy, Fun, and Automated
Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
Weights & Biases LLM-Evaluator Hackathon - Hackathon Judge
Being a human judge at the Weights & Biases LLM-as-a-Judge Hackathon
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
18 Aug 2024 · 49 min · llm eval production survey 🔥
Task-Specific LLM Evals that Do & Don't Work
Evals for classification, summarization, translation, copyright regurgitation, and toxicity.
Out-of-Domain Finetuning to Bootstrap Hallucination Detection
How to use open-source, permissive-use data and collect less labeled samples for our tasks.
05 Nov 2023 · 12 min · llm eval machinelearning python
Evaluation & Hallucination Detection for Abstractive Summaries
Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.
Counterfactual Evaluation for Recommendation Systems
Thinking about recsys as interventional vs. observational, and inverse propensity scoring.
10 Apr 2022 · 8 min · recsys eval machinelearning
Join 11,300+ readers getting updates on machine learning, RecSys, LLMs, and engineering.