eugeneyan

Key themes in my work

New here? These are topics I write & speak about. Alternatively, browse tags or search.

Machine Learning Systems in Industry

Exploring ML systems in industry and how they're implemented.

• Lessons from A year of Building with LLMs: As seen on O’Reilly (Parts 1, 2, 3, book)
• Patterns for LLM Systems: Evals, RAG, fine-tuning, caching, guardrails, defensive UX.
• Improving RecSys & Search with LLMs: Model architecture, data generation, etc.
• System Design for RecSys & Search: Offline vs. online, retrieval vs. ranking.
• Real-time Retrieval: Examples from various companies and how to build an MVP.
• Patterns for Personalization: Via bandits, sequences, graphs, and user embeddings.
• Reinforcement Learning for Recsys: Long-term rewards and explore-exploit.
• Bandits for RecSys: Industry examples, warm-start, off-policy evaluation.
• Search Query Matching: Via lexical, graph, and representation learning methods.
• Push Notifications: What to send, what not to send, and how many to send.
• Content Moderation: Collecting labels, data augmentation, cascade pattern, etc.
• Bootstrapping Data Labels: With semi, active, and weakly supervised learning.
• Feature Stores: As a hierarchy of needs (e.g., access, serving, integrity, etc.)
• Data Discovery Platforms: How they help with find data and open source options.

Machine Learning Techniques

Surveys on machine learning methods.

• Prompting Fundamentals: Structured I/O, prefilling, n-shot prompting, CoT, etc.
• LLM Evals: Evals that actually work for classification, summarization, translation.
• LLM-Evaluators: Use cases, techniques, alignment, finetuning, and critiques.
• Q&A Evals: Metrics, how to build datasets and evaluators, and existing benchmarks.
• Out-of-Domain Finetuning: How to bootstrap on open data and label fewer samples.
• Synthetic Data: For pretraining, instruction-tuning, and preference-tuning.
• Intuition on Attention: Why Q, K, V vectors, multiple heads and layers, etc.
• Counterfactual Evaluation & IPS: Aka the observational vs. interventional problem
• Measuring and Mitigating Position Bias: Accounting for bias in ordered results.
• Primer on Language Models: From RNN to Word2Vec to Transformer to BERT to T5.
• Primer on Text-to-Image: Diffusion, conditioning, guidance, and latent space.

Machine Learning & Engineering

Practices at the intersection of ML and engineering.

• ML Design Patterns: Patterns in code & systems such as factory, decorator, proxy, etc.
• ML Design Patterns for Systems: HITL, hard mining, reframe, cascade, flywheel, etc.
• Python Patterns: Lesser seen patterns such as mixins, relative imports, etc.
• ML Testing: Implementation, expected learned behavior, and evaluation metrics.
• Pipeline Testing: Managing brittle tests in data & machine learning pipelines.
• Model Testing: Why you shouldn’t mock machine learning models in unit tests.
• Challenges with ML in Production and A Practical Guide to Overcome them.
• Python Project Setup: With a workflow of checks that run with each  git push.

Weekend Prototypes

I don't get to hack as much as I want, but when I do, they're a ton of fun.

• News Agents: Automating my daily news flash via agentic workflows.
• AI Reading Club: Prototyping an AI-powered reading experience.
• AlignEval: Building an App to Make Evals Easy, Fun (and Automated? 🤞)
• AI Coach: Talk to Tara, an AI Coach, anytime at +1 (206) 558 8782.
• Obsidian Copilot: A Prototype Assistant for Writing and Reflecting.
• LLM UXs: Interacting with LLMs with Minimal Chat.
• RaspberryLLM: Dr. Seuss headlines, HackerNews trolls, etc.
• LLMs to Research, Reflect, and Plan: LLMs, document retrieval, and Discord.
• ApplyingML.com: Ghost knowledge of applying machine learning.
• RecSys with Graphs & NLP: Node2Vec, Gensim word2vec, PyTorch word2vec.
• Baseline RecSys in PyTorch: The humble matrix factorization.
• SortMySkills: Sorting 50 skills to identify your passion and strengths.
• Image Search: Using image embeddings and cosine similarity.
• Image Classification: Transfer learning via Keras, Theano, and AWS.
• Title Classification: Data acquisition, data prep, building a frontend.

Mechanisms for Business, Product, and Tech Teams

Processes and tools for effective projects and teams.

• How to Interview ML/AI Engineers: How to run phone screens, loops, and debriefs.
• Mechanisms for ML Projects: Pilot & copilot, literature reviews, and more.
• Mechanisms for technical teams: Weekly debriefs, learning sessions, WBRs, etc.
• Weekly 15-5s: Increase visibility and earn trust with this 15-minute habit.
• Scrum for Data Science: Parts of Scrum that fit the data science process well.

Learning & Career

Practices that worked well for me and general advice.

• How to Run A Weekly Paper Club: Benefits, and how to read and facilitate papers.
• Onboarding Effectively: Mindset, 100-day plan, and balancing learning and doing.
• Data Science Red Flags: What to look out for when interviewing with a new team.
• Influencing without Authority: Socratic method, earning trust, finding advocates, etc.
• Comparing Data/ML Roles: Data/Applied/Research scientist, and ML engineer.
• Georgia Tech OMSCS FAQ: A guide to applying for and completing OMSCS.

Writing Tips

Especially in the context of a career in tech and data.

• FAQ on my Writing Process: Why I write, who I write for, how I write, and more.
• Paradoxical Rules of Writing: There are many rules and also no rules at all.
• Writing Why, What, How: A framework to make your writing more effective.
• Design Docs for ML Systems: Applying Why-What-How to ML design docs.
• Writing Lessons You Don’t Learn at School: And my writing syllabus.
• Writing Online for Non-Writers: First, write for yourself. And other advice.
• Writing for Tech Careers: Why write, my writing process, and writing at work.

Featured Talks

Talks that've received the most positive feedback and engagement.

• AIE World’s Fair 2024 Keynote: What We Learned from a Year of Building with LLMs.
• Netflix PRS 2024: LLMs-based recommendation experiences—Challenges & Lessons.
• AIE Summit 2023 Keynote: Lego blocks of LLM systems (evals, RAG, guardrails, etc).
• RecSys 2022 Keynote: Is the juice worth the squeeze, for real-time recommendations?
• System Design for RecSys & Search: Candidate retrieval and ranking in industry.
• OLX Product & Tech Keynote: How Asia’s tech giants scale & the SuperApp strategy.

Ideas & Opinions

Random ideas and unnecessarily strong opinions.

• In Defence of Simplicity: Pushing back on the cult of complexity.
• The 85% Rule: When pushing harder != better outcomes.
• Beginner’s Mind: A mindset for continued learning and growth.
• The First Rule of Machine Learning: Start without machine learning.
• End-to-End Data Science: Why it works and examples from Stitchfix and Netflix.
• Machine Learning & Life Lessons: Seven things ML taught me about life.
• My Seven Habits: The seven habits that improved my last decade.

Summaries & Notes

Summaries and permanent notes, tidied up for public consumption.

• LLM Reading List: Fundamental papers and their one-sentence summaries.
• RecSys 2022: 3 fav papers, 17 summaries, and many lessons.
• RecSys 2021: Simple models, tech stacks, and ops practices.
• RecSys 2020: Bias, sequential models, bandits, robust offline evaluation.

Other resources

That are mostly scattered across the internet.

• applied-llms.org: Practical lessons from a year of building with LLMs.
• applied-ml: Papers and tech blogs on real-world machine learning in industry.
• applyingml.com: Papers, guides, and interviews on how to apply ML effectively.
• open-llms: Open LLMs available for commercial use.
• ml-design-docs: Template of design docs for machine learning systems.
• testing-ml: Examples of implementation & behavioral tests for ML code.
• python-collab-template: Template with tests, type checks, linting, etc.
• papermill-mlflow: Experimentation workflow for machine learning.
• 1-on-1s: Questions to ask during 1-on-1s, from my time as a manager.

Join 11,200+ readers getting updates on machine learning, RecSys, LLMs, and engineering.