# Eugene Yan

> I work to bridge the field and the frontier, and help build safe, reliable AI systems that scale. I write and speak about AI-powered products, recommendation systems, and engineering, and share prototypes here.

Every page on this site has a markdown version at the same URL with `.md` appended. For example, https://eugeneyan.com/writing/llm-patterns is also available at https://eugeneyan.com/writing/llm-patterns.md. Requests that send `Accept: text/markdown` to a page URL receive the markdown version.

## Writing

- [Patterns for Building Cybersecurity Evals](https://eugeneyan.com/writing/cybersecurity-evals.md): A sandboxed target, inputs that influence task difficulty, tools, and a grader.
- [Using LLMs to Secure Source Code](https://eugeneyan.com/writing/secure-source-code.md): Build a threat model, discover vulnerabilities, verify, triage, and patch.
- [How to Work and Compound with AI](https://eugeneyan.com/writing/working-with-ai.md): Context as infra, taste as config, verification for autonomy, scale via delegation, closing the loop.
- [2025 Year in Review](https://eugeneyan.com/writing/2025-review.md): An eventful year of progress in health and career, while making time for travel and reflection.
- [Product Evals in Three Simple Steps](https://eugeneyan.com/writing/product-evals.md): Label some data, align LLM-evaluators, and run the eval harness with each change.
- [Advice for New Principal Tech ICs (i.e., Notes to Myself)](https://eugeneyan.com/writing/principal.md): Based on what I've learned from role models and mentors in Amazon
- [Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs](https://eugeneyan.com/writing/semantic-ids.md): An LLM that can converse in English & item IDs, and make recommendations w/o retrieval or tools.
- [Evaluating Long-Context Question & Answer Systems](https://eugeneyan.com/writing/qa-evals.md): Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
- [Exceptional Leadership: Some Qualities, Behaviors, and Styles](https://eugeneyan.com/writing/leadership.md): What makes a good leader? What do good leaders do? And commando, soldier, and police leadership.
- [Building News Agents for Daily News Recaps with MCP, Q, and tmux](https://eugeneyan.com/writing/news-agents.md): Learning to automate simple agentic workflows with Amazon Q CLI, Anthropic MCP, and tmux.
- [An LLM-as-Judge Won't Save The Product—Fixing Your Process Will](https://eugeneyan.com/writing/eval-process.md): Applying the scientific method, building via eval-driven development, and monitoring AI output.
- [Frequently Asked Questions about My Writing Process](https://eugeneyan.com/writing/writing-faq.md): How I started, why I write, who I write for, how I write, and more.
- [Improving Recommendation Systems & Search in the Age of LLMs](https://eugeneyan.com/writing/recsys-llm.md): Model architectures, data generation, training paradigms, and unified frameworks inspired by LLMs.
- [Building AI Reading Club: Features & Behind the Scenes](https://eugeneyan.com/writing/aireadingclub.md): Exploring how an AI-powered reading experience could look like.
- [2024 Year in Review](https://eugeneyan.com/writing/2024-review.md): A peaceful year of steady progress on my craft and health.
- [Seemingly Paradoxical Rules of Writing](https://eugeneyan.com/writing/paradox.md): With regard to writing, there are many rules and also no rules at all.
- [How to Run a Weekly Paper Club (and Build a Learning Community)](https://eugeneyan.com/writing/paper-club.md): Benefits of running a weekly paper club, how to start one, and how to read and facilitate papers.
- [My Minimal MacBook Pro Setup Guide](https://eugeneyan.com/writing/mac-setup.md): Setting up my new MacBook Pro from scratch
- [39 Lessons on Building ML Systems, Scaling, Execution, and More](https://eugeneyan.com/writing/conf-lessons.md): ML systems, production & scaling, execution & collaboration, building for users, conference etiquette.
- [AlignEval: Building an App to Make Evals Easy, Fun, and Automated](https://eugeneyan.com/writing/aligneval.md): Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
- [Building the Same App Using Various Web Frameworks](https://eugeneyan.com/writing/web-frameworks.md): FastAPI, FastHTML, Next.js, SvelteKit, and thoughts on how coding assistants influence builders' choices.
- [Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)](https://eugeneyan.com/writing/llm-evaluators.md): Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
- [How to Interview and Hire ML/AI Engineers](https://eugeneyan.com/writing/how-to-interview.md): What to interview for, how to structure the phone screen, interview loop, and debrief, and a few tips.
- [Prompting Fundamentals and How to Apply them Effectively](https://eugeneyan.com/writing/prompting.md): Structured input/output, prefilling, n-shots prompting, chain-of-thought, reducing hallucinations, etc.
- [What We've Learned From A Year of Building with LLMs](https://eugeneyan.com/writing/llm-lessons.md): From the tactical nuts & bolts to the operational day-to-day to the long-term business strategy.
- [Building an AI Coach to Help Tame My Monkey Mind](https://eugeneyan.com/writing/ai-coach.md): Building an AI coach with speech-to-text, text-to-speech, an LLM, and a virtual number.
- [Task-Specific LLM Evals that Do & Don't Work](https://eugeneyan.com/writing/evals.md): Evals for classification, summarization, translation, copyright regurgitation, and toxicity.
- [Don't Mock Machine Learning Models In Unit Tests](https://eugeneyan.com/writing/unit-testing-ml.md): How unit testing machine learning code differs from typical software practices
- [How to Generate and Use Synthetic Data for Finetuning](https://eugeneyan.com/writing/synthetic.md): Overcoming the bottleneck of human annotations in instruction-tuning, preference-tuning, and pretraining.
- [Language Modeling Reading List (to Start Your Paper Club)](https://eugeneyan.com/writing/llm-reading-list.md): Some fundamental papers and a one-sentence summary for each; start your own paper club!
- [2023 Year in Review](https://eugeneyan.com/writing/2023-review.md): An expanded charter, lots of writing and speaking, and finally learning to snowboard.
- [Push Notifications: What to Push, What Not to Push, and How Often](https://eugeneyan.com/writing/push.md): Sending helpful & engaging pushes, filtering annoying pushes, and finding the frequency sweet spot.
- [Out-of-Domain Finetuning to Bootstrap Hallucination Detection](https://eugeneyan.com/writing/finetuning.md): How to use open-source, permissive-use data and collect less labeled samples for our tasks.
- [Reflections on AI Engineer Summit 2023](https://eugeneyan.com/writing/aieng-reflections.md): The biggest deployment challenges, backward compatibility, multi-modality, and SF work ethic.
- [Evaluation & Hallucination Detection for Abstractive Summaries](https://eugeneyan.com/writing/abstractive.md): Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.
- [How to Match LLM Patterns to Problems](https://eugeneyan.com/writing/llm-problems.md): Distinguishing problems with external vs. internal LLMs, and data vs non-data patterns
- [Patterns for Building LLM-based Systems & Products](https://eugeneyan.com/writing/llm-patterns.md): Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.
- [Obsidian-Copilot: An Assistant for Writing & Reflecting](https://eugeneyan.com/writing/obsidian-copilot.md): Writing drafts via retrieval-augmented generation. Also reflecting on the week's journal entries.
- [Some Intuition on Attention and the Transformer](https://eugeneyan.com/writing/attention.md): What's the big deal, intuition on query-key-value vectors, multiple heads, multiple layers, and more.
- [Open-LLMs - A list of LLMs for Commercial Use](https://eugeneyan.com/writing/open-llms.md): It started with a question that had no clear answer, and led to eight PRs from the community.
- [Interacting with LLMs with Minimal Chat](https://eugeneyan.com/writing/llm-ux.md): Should chat be the main UX for LLMs? I don't think so and believe we can do better.
- [More Design Patterns For Machine Learning Systems](https://eugeneyan.com/writing/more-patterns.md): 9 patterns including HITL, hard mining, reframing, cascade, data flywheel, business rules layer, and more.
- [Raspberry-LLM - Making My Raspberry Pico a Little Smarter](https://eugeneyan.com/writing/raspberry-llm.md): Generating Dr. Seuss headlines, fake WSJ quotes, HackerNews troll comments, and more.
- [Experimenting with LLMs to Research, Reflect, and Plan](https://eugeneyan.com/writing/llm-experiments.md): Also, shortcomings in document retrieval and how to overcome them with search & recsys techniques.
- [LLM-powered Biographies](https://eugeneyan.com/writing/llm-bio.md): Asking LLMs to generate biographies to get a sense of how they memorize and regurgitate.
- [How to Write Data Labeling/Annotation Guidelines](https://eugeneyan.com/writing/labeling-guidelines.md): Writing good instructions to achieve high precision and throughput.
- [Content Moderation & Fraud Detection - Patterns in Industry](https://eugeneyan.com/writing/content-moderation.md): Collecting ground truth, data augmentation, cascading heuristics and models, and more.
- [Mechanisms for Effective Technical Teams](https://eugeneyan.com/writing/mechanisms-for-teams.md): End of week debrief, weekly business review, monthly learning sessions, and quarter review.
- [Mechanisms for Effective Machine Learning Projects](https://eugeneyan.com/writing/mechanisms-for-projects.md): Pilot & copilot, literature review, methodology review, and timeboxing.
- [Goodbye Roam Research, Hello Obsidian](https://eugeneyan.com/writing/roam-to-obsidian.md): How to migrate and sync notes & images across devices
- [What To Do If Dependency Teams Can’t Help](https://eugeneyan.com/writing/getting-help.md): Seeking first to understand, earning trust, and preparing for away team work.
- [2022 in Review & 2023 Goals](https://eugeneyan.com/writing/2022-in-review.md): Travelled, wrote, and learned a lot, L5 -> L6, gave a keynote at RecSyS, and started a meetup.
- [Autoencoders and Diffusers: A Brief Comparison](https://eugeneyan.com/writing/autoencoders-vs-diffusers.md): A quick overview of variational and denoising autoencoders and comparing them to diffusers.
- [Text-to-Image: Diffusion, Text Conditioning, Guidance, Latent Space](https://eugeneyan.com/writing/text-to-image.md): The fundamentals of text-to-image generation, relevant papers, and experimenting with DDPM.
- [RecSys 2022: Recap, Favorite Papers, and Lessons](https://eugeneyan.com/writing/recsys2022.md): My three favorite papers, 17 paper summaries, and ML and non-ML lessons.
- [Writing Robust Tests for Data & Machine Learning Pipelines](https://eugeneyan.com/writing/testing-pipelines.md): Or why I should write fewer integration tests.
- [Simplicity is An Advantage but Sadly Complexity Sells Better](https://eugeneyan.com/writing/simplicity.md): Pushing back on the cult of complexity.
- [Uncommon Uses of Python in Commonly Used Libraries](https://eugeneyan.com/writing/uncommon-python.md): Some off-the-beaten uses of Python learned from reading libraries.
- [Why You Should Write Weekly 15-5s](https://eugeneyan.com/writing/15-5.md): 15 minutes a week to document your work, increase visibility, and earn trust.
- [Design Patterns in Machine Learning Code and Systems](https://eugeneyan.com/writing/design-patterns.md): Understanding and spotting patterns to use code and components as intended.
- [What I Wish I Knew About Onboarding Effectively](https://eugeneyan.com/writing/onboarding.md): Mindset, 100-day plan, and balancing learning and taking action to earn trust.
- [Bandits for Recommender Systems](https://eugeneyan.com/writing/bandits.md): Industry examples, exploration strategies, warm-starting, off-policy evaluation, and more.
- [How to Measure and Mitigate Position Bias](https://eugeneyan.com/writing/position-bias.md): Introducing randomness and/or learning from inherent randomness to mitigate position bias.
- [Counterfactual Evaluation for Recommendation Systems](https://eugeneyan.com/writing/counterfactual-evaluation.md): Thinking about recsys as interventional vs. observational, and inverse propensity scoring.
- [Traversing High-Level Intent and Low-Level Requirements](https://eugeneyan.com/writing/intent-vs-requirements.md): How they differ and why they work better in different situations.
- [Data Science Project Quick-Start](https://eugeneyan.com/writing/project-quick-start.md): Hard-won lessons on how to start data science projects effectively.
- [Mailbag: How to Define a Data Team's Vision and Roadmap](https://eugeneyan.com/writing/becoming-a-data-leader.md): I'm heading into a team lead role and would like to define the vision and roadmap.
- [Red Flags to Look Out for When Joining a Data Team](https://eugeneyan.com/writing/red-flags.md): What to consider for in terms of data, roadmap, role, manager, tooling, etc.
- [How to Keep Learning about Machine Learning](https://eugeneyan.com/writing/how-to-keep-learning.md): Beyond getting that starting role, how does one continue growing in the field?
- [2021 Year in Review](https://eugeneyan.com/writing/2021-year-in-review.md): Met most of my goals, adopted a puppy, and built ApplyingML.com.
- [Informal Mentors Grew into ApplyingML.com!](https://eugeneyan.com/writing/applyingml.md): More than two dozen interviews with ML Practitioners sharing their stories and advice
- [5 Lessons I Learned from Writing Online (Guest post by Susan Shu)](https://eugeneyan.com/writing/what-i-learned-from-writing-online-susan-shu.md): Susan shares 5 lessons she gained from writing online in public over the past year.
- [What I Learned from Writing Online - For Fellow Non-Writers](https://eugeneyan.com/writing/what-i-learned-from-writing-online.md): Write before you're ready, write for yourself, quantity over quality, and a few other lessons.
- [RecSys 2021 - Papers and Talks to Chew on](https://eugeneyan.com/writing/recsys2021.md): Simple baselines, ideas, tech stacks, and packages to try.
- [The First Rule of Machine Learning: Start without Machine Learning](https://eugeneyan.com/writing/first-rule-of-ml.md): Why this is the first rule, some baseline heuristics, and when to move on to machine learning.
- [Reinforcement Learning for Recommendations and Search](https://eugeneyan.com/writing/reinforcement-learning-for-recsys-and-search.md): Focusing on long-term rewards, exploration, and frequently updated item.
- [Bootstrapping Labels via ___ Supervision & Human-In-The-Loop](https://eugeneyan.com/writing/bootstrapping-data-labels.md): How to generate labels from scratch with semi, active, and weakly supervised learning.
- [Mailbag: How to Bootstrap Labels for Relevant Docs in Search](https://eugeneyan.com/writing/mailbag-bootstrap-relevant-docs.md): Building semantic search; how to calculate recall when relevant documents are unknown.
- [Influencing without Authority for Data Scientists](https://eugeneyan.com/writing/influencing-without-authority.md): Show them the data, the Socratic method, earning trust, and more.
- [System Design for Recommendations and Search](https://eugeneyan.com/writing/system-design-for-discovery.md): Breaking it into offline vs. online environments, and candidate retrieval vs. ranking steps.
- [Patterns for Personalization in Recommendations and Search](https://eugeneyan.com/writing/patterns-for-personalization.md): A whirlwind tour of bandits, embedding+MLP, sequences, graph, and user embeddings.
- [The Metagame of Applying Machine Learning](https://eugeneyan.com/writing/machine-learning-metagame.md): How to go from knowing machine learning to applying it at work to drive impact.
- [Search: Query Matching via Lexical, Graph, and Embedding Methods](https://eugeneyan.com/writing/search-query-matching.md): An overview and comparison of the various approaches, with examples from industry search systems.
- [My Impostor Syndrome Stories (Guest Post by Susan Shu)](https://eugeneyan.com/writing/imposter-syndrome-susan.md): Even high achieving individuals experience impostor syndrome; here's how Susan learned to manage it.
- [How to Live with Chronic Imposter Syndrome](https://eugeneyan.com/writing/imposter-syndrome.md): More education, achievements, and awards don't shoo away imposter syndrome. Here's what might help.
- [Planning Your Career: Values and Superpowers](https://eugeneyan.com/writing/values-and-superpowers.md): What do you deeply care about? What do you excel at? Build a career out of that.
- [Choosing Problems in Data Science and Machine Learning](https://eugeneyan.com/writing/how-to-choose-problems.md): Short vs. long-term gain, incremental vs. disruptive innovation, and resume-driven development.
- [Seven Habits that Shaped My Last Decade](https://eugeneyan.com/writing/seven-habits-that-shaped-my-decade.md): I wish I started sooner. All have improved my life and several have compounding effects.
- [How to Write Design Docs for Machine Learning Systems](https://eugeneyan.com/writing/ml-design-docs.md): Pointers to think through your methodology and implementation, and the review process.
- [How to Write Better with The Why, What, How Framework](https://eugeneyan.com/writing/writing-docs-why-what-how.md): Three documents I write (one-pager, design doc, after-action review) and how I structure them.
- [Feature Stores: A Hierarchy of Needs](https://eugeneyan.com/writing/feature-stores.md): Access, serving, integrity, convenience, autopilot; use what you need.
- [How to Win a Data Hackathon (Hacklytics 2021)](https://eugeneyan.com/writing/how-to-win-data-hackathon.md): What the top teams did to win the 36-hour data hackathon. No, not machine learning.
- [Growing and Running Your Data Science Team](https://eugeneyan.com/writing/data-science-teams.md): What I learned about hiring and training, and fostering innovation, discipline, and camaraderie.
- [You Don't Really Need Another MOOC](https://eugeneyan.com/writing/you-dont-need-another-mooc.md): Stop procrastinating, go off the happy path, learn just-in-time, and get your hands dirty.
- [Mailbag: How to Get Experienced DS Resume Noticed by Recruiters?](https://eugeneyan.com/writing/mailbag-resume-for-experienced-ds.md): How to increase the chances of getting called up by recruiters?
- [Real-time Machine Learning For Recommendations](https://eugeneyan.com/writing/real-time-recommendations.md): Why real-time? How have China & US companies built them? How to design & build an MVP?
- [2021 Roadmap: Sharing, Helping, and Living More](https://eugeneyan.com/writing/2021-roadmap.md): A public roadmap to track and share my progress; nothing mission or work-related.
- [2020 Retrospective: New Country, New Role, New Habit](https://eugeneyan.com/writing/retrospective-2020.md): Wrapping up 2020 with writing and site statistics, graphs, and a word cloud.
- [Catch the Flying Daggers](https://eugeneyan.com/writing/flying-dagger.md): A short story on flying daggers and life's challenges.
- [How I’m Reflecting on 2020 and Planning for 2021](https://eugeneyan.com/writing/how-i-reflect-and-plan.md): Time to clear the cache, evaluate existing processes, and start new threads.
- [Alexey Grigorev on His Career, Data Science, and Writing](https://eugeneyan.com/writing/informal-mentors-alexey-grigorev.md): How he switched from engineering to data science, what "senior" means, and how writing helps.
- [Mailbag: What's the Architecture for your Blog?](https://eugeneyan.com/writing/mailbag-blog-architecture.md): How did you set up your site and what's an easy way to replicate it?
- [What Machine Learning Can Teach Us About Life - 7 Lessons](https://eugeneyan.com/writing/life-lessons-from-machine-learning.md): Data cleaning, transfer learning, overfitting, ensembling, and more.
- [How to Prevent or Deal with a Data Science Role or Title Mismatch](https://eugeneyan.com/writing/role-title-mismatch.md): Interview questions you should ask and how to evolve your job scope.
- [Applied / Research Scientist, ML Engineer: What’s the Difference?](https://eugeneyan.com/writing/data-science-roles.md): A personal take on their deliverables and skills, and what it means for the industry and your team.
- [Chip Huyen on Her Career, Writing, and Machine Learning](https://eugeneyan.com/writing/informal-mentors-chip-huyen.md): Setbacks she faced, overcoming them, and how writing changed her life.
- [Data Discovery Platforms and Their Open Source Solutions](https://eugeneyan.com/writing/data-discovery-platforms.md): What questions do they answer? How do they compare? What open-source solutions are available?
- [Why I switched from Netlify back to GitHub Pages](https://eugeneyan.com/writing/netlify-back-to-github-pages.md): DNS server snafus led to email & security issues. Also, limited free build minutes monthly.
- [Why Have a Data Science Portfolio and What It Shows](https://eugeneyan.com/writing/data-science-portfolio-how-why-what.md): Not 'How to build a data science portfolio', but 'Whys' and 'Whats'.
- [How to Install Google Scalable Nearest Neighbors (ScaNN) on Mac](https://eugeneyan.com/writing/how-to-install-scann-on-mac.md): Step-by-step walkthrough on the environment, compilers, and installation for ScaNN.
- [How Prototyping Can Help You to Get Buy-In](https://eugeneyan.com/writing/prototyping-to-get-buy-in.md): Building prototypes helped get buy-in when roadmaps & design docs failed.
- [Is Writing as Important as Coding?](https://eugeneyan.com/writing/writing-and-coding.md): As careers grow, how does the balance between writing & coding change? Hear from 4 tech leaders.
- [RecSys 2020: Takeaways and Notable Papers](https://eugeneyan.com/writing/recsys2020.md): Emphasis on bias, more sequential models & bandits, robust offline evaluation, and recsys in the wild.
- [Appreciating the Present](https://eugeneyan.com/writing/present.md): What if the alternative was nothingness?
- [Routines and Tools to Optimize My Day (Guest Post by Susan Shu)](https://eugeneyan.com/writing/favorite-productivity-coffee-routines-habits.md): For years I've refined my routines and found tools to manage my time. Here I share it with readers.
- [How to Accomplish More with Less - Useful Tools & Routines](https://eugeneyan.com/writing/how-to-accomplish-more-with-less.md): My tools for organization and creation, autopilot routines, and Maker's schedule
- [Migrating Site Comments to Utterances](https://eugeneyan.com/writing/migrating-to-utterances.md): A step-by-step of how to migrate from json comments to Utterances.
- [How to Test Machine Learning Code and Systems](https://eugeneyan.com/writing/testing-ml.md): Checking for correct implementation, expected learned behaviour, and satisfactory performance.
- [Mailbag: Parsing Fields from PDFs—When to Use Machine Learning?](https://eugeneyan.com/writing/mailbag-pdf-fields.md): Should I switch from a regex-based to ML-based solution on my application?
- [How Reading Papers Helps You Be a More Effective Data Scientist](https://eugeneyan.com/writing/why-read-papers.md): Why read papers, what papers to read, and how to read them.
- [Mailbag: I'm Now a Senior DS—How should I Approach this?](https://eugeneyan.com/writing/mailbag-senior-ds.md): Becoming a senior after three years and dealing with imposter syndrome.
- [Embrace Beginner's Mind; Avoid The Wrong Way To Be An Expert](https://eugeneyan.com/writing/beginners-mind.md): How not to become an expert beginner and to progress through beginner, intermediate, and so on.
- [NLP for Supervised Learning - A Brief Survey](https://eugeneyan.com/writing/nlp-supervised-learning-survey.md): Examining the broad strokes of NLP progress and comparing between models
- [Unpopular Opinion: Data Scientists Should be More End-to-End](https://eugeneyan.com/writing/end-to-end-data-science.md): Why (and why not) be more end-to-end, how to, and Stitch Fix and Netflix's experience
- [Adding a Checkbox & Download Button to a FastAPI-HTML app](https://eugeneyan.com/writing/fastapi-html-checkbox-download.md): Updating our FastAPI app to let users select options and download results.
- [What I Did Not Learn About Writing In School](https://eugeneyan.com/writing/what-i-did-not-learn-about-writing-in-school.md): Surprising lessons I picked up from the best books, essays, and videos on writing non-fiction.
- [Georgia Tech's OMSCS FAQ (based on my experience)](https://eugeneyan.com/writing/georgia-tech-omscs-faq.md): Why OMSCS? How can I get accepted? How much time needed? Did it help your career? And more...
- [How to Set Up a HTML App with FastAPI, Jinja, Forms & Templates](https://eugeneyan.com/writing/how-to-set-up-html-app-with-fastapi-jinja-forms-templates.md): I couldn't find any guides on serving HTML with FastAPI, thus I wrote this to plug the hole on the internet.
- [Why You Need to Follow Up After Your Data Science Project](https://eugeneyan.com/writing/why-you-need-to-follow-up-after-your-data-science-project.md): Ever revisit a project & replicate the results the first time round? Me neither. Thus I adopted these habits.
- [What I Do During A Data Science Project To Deliver Success](https://eugeneyan.com/writing/what-i-do-during-a-data-science-project-to-ensure-success.md): It's not enough to have a good strategy and plan. Execution is just as important.
- [How to Update a GitHub Profile README Automatically](https://eugeneyan.com/writing/how-to-update-github-profile-readme-automatically.md): I wanted to add my recent writing to my GitHub Profile README but was too lazy to do manual updates.
- [The 85% Rule: When Giving It Your 100% Gets You Less than 85%](https://eugeneyan.com/writing/when-giving-your-100-gets-you-less-than-85.md): I thought giving it my all led to maximum outcomes; then I learnt about the 85% rule.
- [My Notes From Spark+AI Summit 2020 (Application-Specific Talks)](https://eugeneyan.com/writing/notes-from-sparkai-summit-application-specific.md): Part II of the previous write-up, this time on applications and frameworks of Spark in production
- [My Notes From Spark+AI Summit 2020 (Application-Agnostic Talks)](https://eugeneyan.com/writing/notes-from-sparkai-summit-application-agnostic.md): Sharing my notes & practical knowledge from the conference for people who don't have the time.
- [How to Set Up a Python Project For Automation and Collaboration](https://eugeneyan.com/writing/setting-up-python-project-for-automation-and-collaboration.md): After this article, we'll have a workflow of tests and checks that run automatically with each git push.
- [Mailbag: Qns on the Intersection of Data Science and Business](https://eugeneyan.com/writing/mailbag-ds-requirements.md): Does DS have business requirements? When does it make sense to split DS and DE??
- [Why Are My Airflow Jobs Running “One Day Late”?](https://eugeneyan.com/writing/why-airflow-jobs-one-day-late.md): A curious discussion made me realize my expert blind spot. And no, Airflow is not late.
- [What I Do Before a Data Science Project to Ensure Success](https://eugeneyan.com/writing/what-i-do-before-a-data-science-project-to-ensure-success.md): Haste makes waste. Diving into a data science problem may not be the fastest route to getting it done.
- [What I Love about Scrum for Data Science](https://eugeneyan.com/writing/what-i-love-about-scrum-for-data-science.md): Initially, I didn't like it. But over time, it grew on me. Here's why.
- [How to Apply Crocker's Law for Feedback and Growth](https://eugeneyan.com/writing/how-to-apply-crockers-law-for-feedback-and-growth.md): Crocker's Law, cognitive dissonance, and how to receive (uncomfortable) feedback better.
- [A Practical Guide to Maintaining Machine Learning in Production](https://eugeneyan.com/writing/practical-guide-to-maintaining-machine-learning.md): Can maintaining machine learning in production be easier? I go through some practical tips.
- [6 Little-Known Challenges After Deploying Machine Learning](https://eugeneyan.com/writing/challenges-after-deploying-machine-learning.md): I thought deploying machine learning was hard. Then I had to maintain multiple systems in prod.
- [How to Write: Advice from David Perell and Sahil Lavingia](https://eugeneyan.com/writing/how-to-write-david-x-sahil.md): An expansion of my Twitter thread that went viral.
- [A Hackathon Where the Dinkiest Idea Won. Why?](https://eugeneyan.com/writing/evaluating-ideas-at-a-hackathon.md): What I Learnt about evaluating ideas from first-hand participation in a hackathon.
- [Serendipity: Accuracy’s Unpopular Best Friend in Recommenders](https://eugeneyan.com/writing/serendipity-and-accuracy-in-recommender-systems.md): What I learned about measuring diversity, novelty, surprise, and serendipity from 10+ papers.
- [How to Give a Kick-Ass Data Science Talk](https://eugeneyan.com/writing/how-to-give-a-kick-ass-data-science-talk.md): Why you should give a talk and some tips from five years of speaking and hosting meet-ups.
- [Commando, Soldier, Police and Your Career Choices](https://eugeneyan.com/writing/commando-soldier-police-and-your-career.md): Should I join a start-up? Which offer should I accept? A simple metaphor to guide your decisions.
- [Stop Taking Regular Notes; Use a Zettelkasten Instead](https://eugeneyan.com/writing/note-taking-zettelkasten.md): Using a Zettelkasten helps you make connections between notes, improving learning and memory.
- [Writing is Learning: How I Learned an Easier Way to Write](https://eugeneyan.com/writing/reading-note-taking-writing.md): Writing begins before actually writing; it's a cycle of reading -> note-taking -> writing.
- [Simpler Experimentation with Jupyter, Papermill, and MLflow](https://eugeneyan.com/writing/experimentation-workflow-with-jupyter-papermill-mlflow.md): Automate your experimentation workflow to minimize effort and iterate faster.
- [My Journey from Psych Grad to Leading Data Science at Lazada](https://eugeneyan.com/writing/psych-grad-to-data-science-lead.md): How hard work, many failures, and a bit of luck got me into the field and up the ladder.
- [Beating the Baseline Recommender with Graph & NLP in Pytorch](https://eugeneyan.com/writing/recommender-systems-graph-and-nlp-pytorch.md): Beating the baseline using Graph & NLP techniques on PyTorch, AUC improvement of ~21% (Part 2 of 2).
- [Building a Strong Baseline Recommender in PyTorch, on a Laptop](https://eugeneyan.com/writing/recommender-systems-baseline-pytorch.md): Building a baseline recsys based on data scraped off Amazon. Warning - Lots of charts! (Part 1 of 2).
- [OMSCS CS6200 (Introduction to OS) Review and Tips](https://eugeneyan.com/writing/omscs-cs6200-introduction-to-operating-systems.md): OMSCS CS6200 (Introduction to OS) - Moving data from one process to another, multi-threaded.
- [OMSCS CS6750 (Human Computer Interaction) Review and Tips](https://eugeneyan.com/writing/omscs-cs6750-human-computer-interaction.md): OMSCS CS6750 (Human Computer Interaction) - You are not your user! Or how to build great products.
- [Goodbye Wordpress, Hello Jekyll!](https://eugeneyan.com/writing/goodbye-wordpress-hello-jekyll.md): Moving off wordpress and hosting for free on GitHub. And gaining full customization!
- [OMSCS CS6440 (Intro to Health Informatics) Review and Tips](https://eugeneyan.com/writing/omscs-cs6440-intro-to-health-informatics.md): OMSCS CS6440 (Intro to Health Informatics) - A primer on key tech and standards in healthtech.
- [OMSCS CS7646 (Machine Learning for Trading) Review and Tips](https://eugeneyan.com/writing/omscs-cs7646-machine-learning-for-trading.md): OMSCS CS7646 (Machine Learning for Trading) - Don't sell your house to trade algorithmically.
- [What does a Data Scientist really do?](https://eugeneyan.com/writing/what-does-a-data-scientist-really-do.md): No, you don't need a PhD or 10+ years of experience.
- [Data Science and Agile (Frameworks for Effectiveness)](https://eugeneyan.com/writing/data-science-and-agile-frameworks-for-effectiveness.md): Taking the best from agile and modifying it to fit the data science process (Part 2 of 2).
- [Data Science and Agile (What Works, and What Doesn't)](https://eugeneyan.com/writing/data-science-and-agile-what-works-and-what-doesnt.md): A deeper look into the strengths and weaknesses of Agile in Data Science projects (Part 1 of 2).
- [OMSCS CS6601 (Artificial Intelligence) Review and Tips](https://eugeneyan.com/writing/omscs-cs6601-artificial-intelligence.md): OMSCS CS6601 (Artificial Intelligence) - First, start with the simplest solution, and then add intelligence.
- [OMSCS CS6460 (Education Technology) Review and Tips](https://eugeneyan.com/writing/omscs-cs6460-education-technology.md): OMSCS CS6460 (Education Technology) - How to scale education widely through technology.
- [OMSCS CS7642 (Reinforcement Learning) Review and Tips](https://eugeneyan.com/writing/omscs-cs7642-reinforcement-learning.md): OMSCS CS7642 (Reinforcement Learning) - Landing rockets (fun!) via deep Q-Learning (and its variants).
- [Building a Strong Data Science Team Culture](https://eugeneyan.com/writing/building-a-strong-data-science-team-culture.md): Culture >> Hierarchy, Process, Bureaucracy.
- [OMSCS CS7641 (Machine Learning) Review and Tips](https://eugeneyan.com/writing/omscs-cs7641-machine-learning.md): OMSCS CS7641 (Machine Learning) - Revisiting the fundamentals and learning new techniques.
- [My first 100 days as Data Science Lead](https://eugeneyan.com/writing/my-first-100-days-as-data-science-lead.md): How being a Lead / Manager is different from being an individual contributor.
- [OMSCS CS6300 (Software Development Process) Review and Tips](https://eugeneyan.com/writing/omscs-cs6300-software-development-process.md): OMSCS CS6300 (Software Development Process) - Java and collaboratively developing an Android app.
- [How to get started in Data Science](https://eugeneyan.com/writing/how-to-get-started-in-data-science.md): Tools and skills to pick up, and how to practice them.
- [OMSCS CS6476 (Computer Vision) Review and Tips](https://eugeneyan.com/writing/omscs-cs6476-computer-vision.md): OMSCS CS6476 Computer Vision - Performing computer vision tasks with ONLY numpy.
- [One way to help a data science team innovate successfully](https://eugeneyan.com/writing/one-way-to-help-a-data-science-team-succeed.md): If things are not failing, you're not innovating enough. - Elon Musk
- [Product Categorization API Part 3: Creating an API](https://eugeneyan.com/writing/product-categorization-api-part-3-creating-an-api.md): Or how to put machine learning models into production.
- [Image search is now live!](https://eugeneyan.com/writing/image-search-is-now-live.md): A web app to find similar products based on image.
- [Product Classification API Part 2: Data Preparation](https://eugeneyan.com/writing/product-categorization-api-part-2-data-preparation.md): Cleaning up text and messing with ascii (urgh!)
- [Image classification API is now live!](https://eugeneyan.com/writing/image-categorization-is-now-live.md): A simple web app to classify fashion images into Amazon categories.
- [I'm going back to school](https://eugeneyan.com/writing/im-going-back-to-school.md): Got accepted into Georgia Tech's Computer Science Masters!
- [SortMySkills is now live!](https://eugeneyan.com/writing/sortmyskills-is-now-live.md): A card sorting game to discover youl passion by identifying skills you like and dislike.
- [Product Classification API Part 1: Data Acquisition](https://eugeneyan.com/writing/product-categorization-api-part-1-data-acquisition-and-formatting.md): Parsing json and formatting product titles and categories.
- [Thoughts on Functional Programming in Scala Course (Coursera)](https://eugeneyan.com/writing/thoughts-on-functional-programming-in-scala-course-coursera.md): Learning Scala from Martin Odersky, father of Scala.
- [First post!](https://eugeneyan.com/writing/first-post.md): Time to start writing.
- [DataKind Singapore’s Latest Project Accelerator](https://eugeneyan.com/writing/datakind-sg-project-accelerator.md): Guest post of how DataKind SG worked with NGOs to frame their problems and suggests solutions

## Speaking

- [AI Engineer 2026 - Using LLMs to Secure Source Code](https://eugeneyan.com/speaking/aie-2026.md): Why now, six steps to an agentic harness for security, and the org bottlenecks that follow.
- [AI Engineer 2025 - Improving RecSys & Search with LLM techniques](https://eugeneyan.com/speaking/aie-2025.md): Recsys & search are converging with LLMs via semantic IDs, data augmentation, and unified foundation models.
- [NVIDIA GTC 2025 - Building LLM-Powered Applications](https://eugeneyan.com/speaking/nvidia-gtc-2025.md): Chip Huyen and I share what we've learned, best practices, and insights at NVIDIA GTC 2025.
- [Weights & Biases LLM-Evaluator Hackathon - Hackathon Judge](https://eugeneyan.com/speaking/hackathon-judge.md): Being a human judge at the Weights & Biases LLM-as-a-Judge Hackathon
- [AI Engineer 2024 Keynote - What We Learned from a Year of LLMs](https://eugeneyan.com/speaking/aie-2024.md): Special double-feature closing keynote from the 6 authors of the hit O'Reilly article on Applied LLMs.
- [Netflix PRS 2024 - Applying LLMs to Recommendation Experiences](https://eugeneyan.com/speaking/netflix-prs.md): Challenges and lessons from deploying LLM experiences: evals, scalability, guardrails.
- [AI Engineer 2023 Keynote - Building Blocks for LLM Systems](https://eugeneyan.com/speaking/ai-eng-summit.md): Evals, retrieval-augmented generation, guardrails, and collecting feedback; all that good stuff.
- [RecSys 2022 Keynote - Is the Juice Worth the Squeeze?](https://eugeneyan.com/speaking/recsys2022-keynote.md): Invited keynote at the Workshop for Online Recommender Systems and User Modeling (ORSUM)
- [The Data Scientist Show - Building end-to-end ML systems](https://eugeneyan.com/speaking/data-science-show-eugene-yan.md): Daliana and I had a 2hr chat on all things data science and machine learning.
- [MLOps Community - System Design for RecSys & Search](https://eugeneyan.com/speaking/mlops-community-recsys.md): An overview of system design, candidate retrieval, and ranking, with industry examples.
- [Amazon Science - Eugene Yan and the Art of Writing about Science](https://eugeneyan.com/speaking/amazon-science-feature.md): Why the Amazon applied scientist takes the time to break down his work for readers.
- [SF Big Analytics - System Design for RecSys & Search](https://eugeneyan.com/speaking/sf-data-meetup-recsys.md): Why real-time RecSys? What does the system design look like in industry? How to build an MVP?
- [Towards Data Science - Author Spotlight with Eugene Yan](https://eugeneyan.com/speaking/tds-author-spotlight.md): My favourite project, how I write weekly and how you can start, and content I would like to see more of.
- [Bukalapak - Fireside Chat with the Data Science team](https://eugeneyan.com/speaking/bukalapak-fireside.md): We discussed about how to build and run data teams and engage better with business.
- [TalkPython - What ML can Teach Us About Life](https://eugeneyan.com/speaking/talk-python.md): Mike and I take a philosophical detour on Talk Python and discuss life lessons from machine learning.
- [DataTalksClub - Building an ML System; Behind the Scenes](https://eugeneyan.com/speaking/machine-learning-in-healthcare.md): Design and architecture, tech stack, methodology, results, and lessons learned.
- [DataTalksClub - The Importance Of Writing In A Tech Career](https://eugeneyan.com/speaking/writing-for-a-tech-career.md): Why did I start writing? What's my writing process? What's the writing culture at Amazon like?
- [CareerFair - Day-to-day as an Applied Scientist at Amazon](https://eugeneyan.com/speaking/data-scientist-day-to-day.md): What's an average day like? What's great about the role? How's working in Amazon?
- [Datacast Podcast - Effective Data Science with Eugene Yan](https://eugeneyan.com/speaking/datacast-eugeneyan.md): My chat with James Le about my experience, leadership, agile, ML in production, writing, and more.
- [DataScience SG Meetup - RecSys, Beyond the Baseline](https://eugeneyan.com/speaking/recommender-systems-beyond-the-baseline-talk.md): Comparing baselines (matrix factorization) against novel approaches using graphs & NLP.
- [DataScience SG x ODSC Meetup - Applying ML to Healthcare](https://eugeneyan.com/speaking/machine-learning-in-production-for-healthcare-talk.md): In-depth sharing on how to put machine learning systems into production.
- [OLX Prod Tech 2019 Keynote - Asia's Tech Giants & SuperApps](https://eugeneyan.com/speaking/olx-group-keynote-asia-tech-giants-talk.md): Keynote on how Asia's tech giants scale and their SuperApp strategy.
- [DATAx - A Production ML system for SEA's Biggest Hospital Group](https://eugeneyan.com/speaking/machine-learning-largest-hospital-group-talk.md): How we built an ML system to predict hospitalization costs at admission; sharing at DATAx Conference.
- [DataScience SG Meetup - Panel On the Different Roles in Data](https://eugeneyan.com/speaking/different-roles-in-data-talk.md): What's the difference between a data scientist, data engineer, and ML engineer? A panel at Google.
- [GovTech Conference - Data Science and Agile—Can or Not?](https://eugeneyan.com/speaking/data-science-and-agile-can-or-not-talk.md): Yes, Agile can be adopted by data science teams. Moderating a panel at GovTech STACK.
- [Big Data & Analytics Summit - Data Science Challenges @ Lazada](https://eugeneyan.com/speaking/data-science-challenges-impact-lazada-talk.md): Technical challenges easy compared to business and people issues. Sharing at the BDA Summit.
- [INSEAD Lunchtime Talks - How Lazada uses Data](https://eugeneyan.com/speaking/insead-how-lazada-uses-data-talk.md): And my idiosyncratic journey to VP of Data Science at Lazada (Alibaba). A Lunchtime chat at INSEAD.
- [SMU - What is Data Analytics and How do I get into it?](https://eugeneyan.com/speaking/sharing-at-singapore-management-university-on-data-analytics-talk.md): What is data science, how to pick it up, and how to enter the field? A discussion with SMU undergrads.
- [Tech in Asia - My Journey in Data Science and Advice for others](https://eugeneyan.com/speaking/my-sharing-with-tech-in-asia-talk.md): Sharing about why data science, data science myths, a typical day, and more with TIA.
- [SMU Masters in IT - How to get started in Data Science](https://eugeneyan.com/speaking/how-to-get-started-in-data-science-talk.md): Tools and skills to pick up and how to practice them. An Invited Talk with Masters in IT candidates.
- [Strata x Hadoop 2016 - How Lazada Ranks Products](https://eugeneyan.com/speaking/shared-about-my-work-in-lazada-at-strata-hadoop-singapore-2016-talk.md): How Lazada ranks products to improve customer experience and conversion at Strata 2016.
- [DataScience SG Meetup - How we got top 3% in Kaggle](https://eugeneyan.com/speaking/dssg-kaggle-top-3-percent-talk.md): Sharing about my first data science competition at DataScience SG.

## Pages

- [About](https://eugeneyan.com/about.md): Eugene Yan works to bridge the field and the frontier, and helps build safe, reliable AI systems that scale. He also writes and speaks about AI-powered systems and products, and prototypes for fun.
- [Angel Investments](https://eugeneyan.com/investments.md): Some angel investments I've made.
- [How I Can Help](https://eugeneyan.com/how-i-can-help.md): Guidelines on what I cannot, and can, help with
- [Newsletter](https://eugeneyan.com/subscribe.md): Get new posts in your inbox as they are published. No spam, and you can unsubscribe at any time.
- [Prototyping](https://eugeneyan.com/prototyping.md): I prototype to learn and have fun.
- [README](https://eugeneyan.com/readme.md): A user manual to operating Eugene Yan. A work in progress, just like me.
- [Search](https://eugeneyan.com/search.md): Search this site.
- [Start Here](https://eugeneyan.com/start-here.md): Key themes I write about: LLM systems and evals, recommendation systems, ML in production, and how to work, write, and grow in tech.

## Optional

- [RSS feed](https://eugeneyan.com/rss/)