Tag:
eval
(10)Product Evals in Three Simple Steps
Label some data, align LLM-evaluators, and run the eval harness with each change.
23 Nov 2025 Ā· 9 min Ā· eval engineering production
Evaluating Long-Context Question & Answer Systems
Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
An LLM-as-Judge Won't Save The ProductāFixing Your Process Will
Applying the scientific method, building via eval-driven development, and monitoring AI output.
20 Apr 2025 Ā· 5 min Ā· eval llm engineering
AlignEval: Building an App to Make Evals Easy, Fun, and Automated
Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
Weights & Biases LLM-Evaluator Hackathon - Hackathon Judge
Being a human judge at the Weights & Biases LLM-as-a-Judge Hackathon
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
18 Aug 2024 Ā· 49 min Ā· llm eval production survey š„
Task-Specific LLM Evals that Do & Don't Work
Evals for classification, summarization, translation, copyright regurgitation, and toxicity.
Out-of-Domain Finetuning to Bootstrap Hallucination Detection
How to use open-source, permissive-use data and collect less labeled samples for our tasks.
05 Nov 2023 Ā· 12 min Ā· llm eval machinelearning python
Evaluation & Hallucination Detection for Abstractive Summaries
Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.
Counterfactual Evaluation for Recommendation Systems
Thinking about recsys as interventional vs. observational, and inverse propensity scoring.
10 Apr 2022 Ā· 8 min Ā· recsys eval machinelearning
Join 11,800+ readers getting updates on machine learning, RecSys, LLMs, and engineering.