# Writing

> I write to learn and share what I learned.

- URL: https://eugeneyan.com/writing
- Index of the whole site: https://eugeneyan.com/llms.txt

- [Patterns for Building Cybersecurity Evals](https://eugeneyan.com/writing/cybersecurity-evals.md) (2026-06-21): A sandboxed target, inputs that influence task difficulty, tools, and a grader.
- [Using LLMs to Secure Source Code](https://eugeneyan.com/writing/secure-source-code.md) (2026-05-27): Build a threat model, discover vulnerabilities, verify, triage, and patch.
- [How to Work and Compound with AI](https://eugeneyan.com/writing/working-with-ai.md) (2026-05-03): Context as infra, taste as config, verify for autonomy, scale via delegation, close the loop.
- [2025 Year in Review](https://eugeneyan.com/writing/2025-review.md) (2025-12-14): An eventful year of progress in health and career, while making time for travel and reflection.
- [Product Evals in Three Simple Steps](https://eugeneyan.com/writing/product-evals.md) (2025-11-23): Label some data, align LLM-evaluators, and run the eval harness with each change.
- [Advice for New Principal Tech ICs (i.e., Notes to Myself)](https://eugeneyan.com/writing/principal.md) (2025-10-19): Based on what I've learned from role models and mentors in Amazon
- [Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs](https://eugeneyan.com/writing/semantic-ids.md) (2025-09-14): An LLM that can converse in English & item IDs, and make recommendations w/o retrieval or tools.
- [Evaluating Long-Context Question & Answer Systems](https://eugeneyan.com/writing/qa-evals.md) (2025-06-22): Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
- [Exceptional Leadership: Some Qualities, Behaviors, and Styles](https://eugeneyan.com/writing/leadership.md) (2025-05-18): What makes a good leader? What do good leaders do? And commando, soldier, and police leadership.
- [Building News Agents for Daily News Recaps with MCP, Q, and tmux](https://eugeneyan.com/writing/news-agents.md) (2025-05-04): Learning to automate simple agentic workflows with Amazon Q CLI, Anthropic MCP, and tmux.
- [An LLM-as-Judge Won't Save The Product—Fixing Your Process Will](https://eugeneyan.com/writing/eval-process.md) (2025-04-20): Applying the scientific method, building via eval-driven development, and monitoring AI output.
- [Frequently Asked Questions about My Writing Process](https://eugeneyan.com/writing/writing-faq.md) (2025-03-30): How I started, why I write, who I write for, how I write, and more.
- [Improving Recommendation Systems & Search in the Age of LLMs](https://eugeneyan.com/writing/recsys-llm.md) (2025-03-16): Model architectures, data generation, training paradigms, and unified frameworks inspired by LLMs.
- [Building AI Reading Club: Features & Behind the Scenes](https://eugeneyan.com/writing/aireadingclub.md) (2025-01-12): Exploring how an AI-powered reading experience could look like.
- [2024 Year in Review](https://eugeneyan.com/writing/2024-review.md) (2024-12-22): A peaceful year of steady progress on my craft and health.
- [A Spark of the Anti-AI Backlash (on Bluesky)](https://eugeneyan.com/writing/anti.md) (2024-12-08): How the sharing of 1M Bluesky posts uncovered the strong anti-AI sentiment on Bluesky.
- [Seemingly Paradoxical Rules of Writing](https://eugeneyan.com/writing/paradox.md) (2024-12-01): With regard to writing, there are many rules and also no rules at all.
- [How to Run a Weekly Paper Club (and Build a Learning Community)](https://eugeneyan.com/writing/paper-club.md) (2024-11-24): Benefits of running a weekly paper club, how to start one, and how to read and facilitate papers.
- [My Minimal MacBook Pro Setup Guide](https://eugeneyan.com/writing/mac-setup.md) (2024-11-17): Setting up my new MacBook Pro from scratch
- [39 Lessons on Building ML Systems, Scaling, Execution, and More](https://eugeneyan.com/writing/conf-lessons.md) (2024-11-03): ML systems, production & scaling, execution & collaboration, building for users, conference etiquette.
- [AlignEval: Building an App to Make Evals Easy, Fun, and Automated](https://eugeneyan.com/writing/aligneval.md) (2024-10-27): Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
- [Building the Same App Using Various Web Frameworks](https://eugeneyan.com/writing/web-frameworks.md) (2024-09-08): FastAPI, FastHTML, Next.js, SvelteKit, and thoughts on how coding assistants influence builders' choices.
- [Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)](https://eugeneyan.com/writing/llm-evaluators.md) (2024-08-18): Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
- [How to Interview and Hire ML/AI Engineers](https://eugeneyan.com/writing/how-to-interview.md) (2024-07-07): What to interview for, how to structure the phone screen, interview loop, and debrief, and a few tips.
- [Prompting Fundamentals and How to Apply them Effectively](https://eugeneyan.com/writing/prompting.md) (2024-05-26): Structured input/output, prefilling, n-shots prompting, chain-of-thought, reducing hallucinations, etc.
- [What We've Learned From A Year of Building with LLMs](https://eugeneyan.com/writing/llm-lessons.md) (2024-05-12): From the tactical nuts & bolts to the operational day-to-day to the long-term business strategy.
- [Building an AI Coach to Help Tame My Monkey Mind](https://eugeneyan.com/writing/ai-coach.md) (2024-04-07): Building an AI coach with speech-to-text, text-to-speech, an LLM, and a virtual number.
- [Task-Specific LLM Evals that Do & Don't Work](https://eugeneyan.com/writing/evals.md) (2024-03-31): Evals for classification, summarization, translation, copyright regurgitation, and toxicity.
- [Don't Mock Machine Learning Models In Unit Tests](https://eugeneyan.com/writing/unit-testing-ml.md) (2024-02-25): How unit testing machine learning code differs from typical software practices
- [How to Generate and Use Synthetic Data for Finetuning](https://eugeneyan.com/writing/synthetic.md) (2024-02-11): Overcoming the bottleneck of human annotations in instruction-tuning, preference-tuning, and pretraining.
- [Language Modeling Reading List (to Start Your Paper Club)](https://eugeneyan.com/writing/llm-reading-list.md) (2024-01-07): Some fundamental papers and a one-sentence summary for each; start your own paper club!
- [2023 Year in Review](https://eugeneyan.com/writing/2023-review.md) (2023-12-31): An expanded charter, lots of writing and speaking, and finally learning to snowboard.
- [Push Notifications: What to Push, What Not to Push, and How Often](https://eugeneyan.com/writing/push.md) (2023-12-24): Sending helpful & engaging pushes, filtering annoying pushes, and finding the frequency sweet spot.
- [Out-of-Domain Finetuning to Bootstrap Hallucination Detection](https://eugeneyan.com/writing/finetuning.md) (2023-11-05): How to use open-source, permissive-use data and collect less labeled samples for our tasks.
- [Reflections on AI Engineer Summit 2023](https://eugeneyan.com/writing/aieng-reflections.md) (2023-10-15): The biggest deployment challenges, backward compatibility, multi-modality, and SF work ethic.
- [Evaluation & Hallucination Detection for Abstractive Summaries](https://eugeneyan.com/writing/abstractive.md) (2023-09-03): Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.
- [How to Match LLM Patterns to Problems](https://eugeneyan.com/writing/llm-problems.md) (2023-08-13): Distinguishing problems with external vs. internal LLMs, and data vs non-data patterns
- [Patterns for Building LLM-based Systems & Products](https://eugeneyan.com/writing/llm-patterns.md) (2023-07-30): Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.
- [Obsidian-Copilot: An Assistant for Writing & Reflecting](https://eugeneyan.com/writing/obsidian-copilot.md) (2023-06-11): Writing drafts via retrieval-augmented generation. Also reflecting on the week's journal entries.
- [Some Intuition on Attention and the Transformer](https://eugeneyan.com/writing/attention.md) (2023-05-21): What's the big deal, intuition on query-key-value vectors, multiple heads, multiple layers, and more.
- [Open-LLMs - A list of LLMs for Commercial Use](https://eugeneyan.com/writing/open-llms.md) (2023-05-07): It started with a question that had no clear answer, and led to eight PRs from the community.
- [Interacting with LLMs with Minimal Chat](https://eugeneyan.com/writing/llm-ux.md) (2023-04-30): Should chat be the main UX for LLMs? I don't think so and believe we can do better.
- [More Design Patterns For Machine Learning Systems](https://eugeneyan.com/writing/more-patterns.md) (2023-04-23): 9 patterns including HITL, hard mining, reframing, cascade, data flywheel, business rules layer, and more.
- [Raspberry-LLM - Making My Raspberry Pico a Little Smarter](https://eugeneyan.com/writing/raspberry-llm.md) (2023-04-16): Generating Dr. Seuss headlines, fake WSJ quotes, HackerNews troll comments, and more.
- [Experimenting with LLMs to Research, Reflect, and Plan](https://eugeneyan.com/writing/llm-experiments.md) (2023-04-09): Also, shortcomings in document retrieval and how to overcome them with search & recsys techniques.
- [LLM-powered Biographies](https://eugeneyan.com/writing/llm-bio.md) (2023-03-19): Asking LLMs to generate biographies to get a sense of how they memorize and regurgitate.
- [How to Write Data Labeling/Annotation Guidelines](https://eugeneyan.com/writing/labeling-guidelines.md) (2023-03-12): Writing good instructions to achieve high precision and throughput.
- [Content Moderation & Fraud Detection - Patterns in Industry](https://eugeneyan.com/writing/content-moderation.md) (2023-02-26): Collecting ground truth, data augmentation, cascading heuristics and models, and more.
- [Mechanisms for Effective Technical Teams](https://eugeneyan.com/writing/mechanisms-for-teams.md) (2023-02-05): End of week debrief, weekly business review, monthly learning sessions, and quarter review.
- [Mechanisms for Effective Machine Learning Projects](https://eugeneyan.com/writing/mechanisms-for-projects.md) (2023-01-22): Pilot & copilot, literature review, methodology review, and timeboxing.
- [Goodbye Roam Research, Hello Obsidian](https://eugeneyan.com/writing/roam-to-obsidian.md) (2023-01-15): How to migrate and sync notes & images across devices
- [What To Do If Dependency Teams Can’t Help](https://eugeneyan.com/writing/getting-help.md) (2023-01-08): Seeking first to understand, earning trust, and preparing for away team work.
- [2022 in Review & 2023 Goals](https://eugeneyan.com/writing/2022-in-review.md) (2022-12-24): Travelled, wrote, and learned a lot, L5 -> L6, gave a keynote at RecSyS, and started a meetup.
- [Autoencoders and Diffusers: A Brief Comparison](https://eugeneyan.com/writing/autoencoders-vs-diffusers.md) (2022-12-11): A quick overview of variational and denoising autoencoders and comparing them to diffusers.
- [Text-to-Image: Diffusion, Text Conditioning, Guidance, Latent Space](https://eugeneyan.com/writing/text-to-image.md) (2022-11-27): The fundamentals of text-to-image generation, relevant papers, and experimenting with DDPM.
- [RecSys 2022: Recap, Favorite Papers, and Lessons](https://eugeneyan.com/writing/recsys2022.md) (2022-10-02): My three favorite papers, 17 paper summaries, and ML and non-ML lessons.
- [Writing Robust Tests for Data & Machine Learning Pipelines](https://eugeneyan.com/writing/testing-pipelines.md) (2022-09-04): Or why I should write fewer integration tests.
- [Simplicity is An Advantage but Sadly Complexity Sells Better](https://eugeneyan.com/writing/simplicity.md) (2022-08-14): Pushing back on the cult of complexity.
- [Uncommon Uses of Python in Commonly Used Libraries](https://eugeneyan.com/writing/uncommon-python.md) (2022-07-31): Some off-the-beaten uses of Python learned from reading libraries.
- [Why You Should Write Weekly 15-5s](https://eugeneyan.com/writing/15-5.md) (2022-06-26): 15 minutes a week to document your work, increase visibility, and earn trust.
- [Design Patterns in Machine Learning Code and Systems](https://eugeneyan.com/writing/design-patterns.md) (2022-06-12): Understanding and spotting patterns to use code and components as intended.
- [What I Wish I Knew About Onboarding Effectively](https://eugeneyan.com/writing/onboarding.md) (2022-05-22): Mindset, 100-day plan, and balancing learning and taking action to earn trust.
- [Bandits for Recommender Systems](https://eugeneyan.com/writing/bandits.md) (2022-05-08): Industry examples, exploration strategies, warm-starting, off-policy evaluation, and more.
- [How to Measure and Mitigate Position Bias](https://eugeneyan.com/writing/position-bias.md) (2022-04-17): Introducing randomness and/or learning from inherent randomness to mitigate position bias.
- [Counterfactual Evaluation for Recommendation Systems](https://eugeneyan.com/writing/counterfactual-evaluation.md) (2022-04-10): Thinking about recsys as interventional vs. observational, and inverse propensity scoring.
- [Traversing High-Level Intent and Low-Level Requirements](https://eugeneyan.com/writing/intent-vs-requirements.md) (2022-03-20): How they differ and why they work better in different situations.
- [Data Science Project Quick-Start](https://eugeneyan.com/writing/project-quick-start.md) (2022-03-06): Hard-won lessons on how to start data science projects effectively.
- [Mailbag: How to Define a Data Team's Vision and Roadmap](https://eugeneyan.com/writing/becoming-a-data-leader.md) (2022-02-18): I'm heading into a team lead role and would like to define the vision and roadmap.
- [Red Flags to Look Out for When Joining a Data Team](https://eugeneyan.com/writing/red-flags.md) (2022-02-13): What to consider for in terms of data, roadmap, role, manager, tooling, etc.
- [How to Keep Learning about Machine Learning](https://eugeneyan.com/writing/how-to-keep-learning.md) (2022-01-19): Beyond getting that starting role, how does one continue growing in the field?
- [2021 Year in Review](https://eugeneyan.com/writing/2021-year-in-review.md) (2021-11-28): Met most of my goals, adopted a puppy, and built ApplyingML.com.
- [Informal Mentors Grew into ApplyingML.com!](https://eugeneyan.com/writing/applyingml.md) (2021-11-25): More than two dozen interviews with ML Practitioners sharing their stories and advice
- [5 Lessons I Learned from Writing Online (Guest post by Susan Shu)](https://eugeneyan.com/writing/what-i-learned-from-writing-online-susan-shu.md) (2021-11-07): Susan shares 5 lessons she gained from writing online in public over the past year.
- [What I Learned from Writing Online - For Fellow Non-Writers](https://eugeneyan.com/writing/what-i-learned-from-writing-online.md) (2021-10-17): Write before you're ready, write for yourself, quantity over quality, and a few other lessons.
- [RecSys 2021 - Papers and Talks to Chew on](https://eugeneyan.com/writing/recsys2021.md) (2021-10-03): Simple baselines, ideas, tech stacks, and packages to try.
- [The First Rule of Machine Learning: Start without Machine Learning](https://eugeneyan.com/writing/first-rule-of-ml.md) (2021-09-19): Why this is the first rule, some baseline heuristics, and when to move on to machine learning.
- [Reinforcement Learning for Recommendations and Search](https://eugeneyan.com/writing/reinforcement-learning-for-recsys-and-search.md) (2021-09-05): Focusing on long-term rewards, exploration, and frequently updated item.
- [Bootstrapping Labels via ___ Supervision & Human-In-The-Loop](https://eugeneyan.com/writing/bootstrapping-data-labels.md) (2021-08-01): How to generate labels from scratch with semi, active, and weakly supervised learning.
- [Mailbag: How to Bootstrap Labels for Relevant Docs in Search](https://eugeneyan.com/writing/mailbag-bootstrap-relevant-docs.md) (2021-07-20): Building semantic search; how to calculate recall when relevant documents are unknown.
- [Influencing without Authority for Data Scientists](https://eugeneyan.com/writing/influencing-without-authority.md) (2021-07-04): Show them the data, the Socratic method, earning trust, and more.
- [System Design for Recommendations and Search](https://eugeneyan.com/writing/system-design-for-discovery.md) (2021-06-27): Breaking it into offline vs. online environments, and candidate retrieval vs. ranking steps.
- [Patterns for Personalization in Recommendations and Search](https://eugeneyan.com/writing/patterns-for-personalization.md) (2021-06-13): A whirlwind tour of bandits, embedding+MLP, sequences, graph, and user embeddings.
- [The Metagame of Applying Machine Learning](https://eugeneyan.com/writing/machine-learning-metagame.md) (2021-05-02): How to go from knowing machine learning to applying it at work to drive impact.
- [Search: Query Matching via Lexical, Graph, and Embedding Methods](https://eugeneyan.com/writing/search-query-matching.md) (2021-04-25): An overview and comparison of the various approaches, with examples from industry search systems.
- [My Impostor Syndrome Stories (Guest Post by Susan Shu)](https://eugeneyan.com/writing/imposter-syndrome-susan.md) (2021-04-18): Even high achieving individuals experience impostor syndrome; here's how Susan learned to manage it.
- [How to Live with Chronic Imposter Syndrome](https://eugeneyan.com/writing/imposter-syndrome.md) (2021-04-11): More education, achievements, and awards don't shoo away imposter syndrome. Here's what might help.
- [Planning Your Career: Values and Superpowers](https://eugeneyan.com/writing/values-and-superpowers.md) (2021-04-04): What do you deeply care about? What do you excel at? Build a career out of that.
- [Choosing Problems in Data Science and Machine Learning](https://eugeneyan.com/writing/how-to-choose-problems.md) (2021-03-21): Short vs. long-term gain, incremental vs. disruptive innovation, and resume-driven development.
- [Seven Habits that Shaped My Last Decade](https://eugeneyan.com/writing/seven-habits-that-shaped-my-decade.md) (2021-03-14): I wish I started sooner. All have improved my life and several have compounding effects.
- [How to Write Design Docs for Machine Learning Systems](https://eugeneyan.com/writing/ml-design-docs.md) (2021-03-07): Pointers to think through your methodology and implementation, and the review process.
- [How to Write Better with The Why, What, How Framework](https://eugeneyan.com/writing/writing-docs-why-what-how.md) (2021-02-28): Three documents I write (one-pager, design doc, after-action review) and how I structure them.
- [Feature Stores: A Hierarchy of Needs](https://eugeneyan.com/writing/feature-stores.md) (2021-02-21): Access, serving, integrity, convenience, autopilot; use what you need.
- [How to Win a Data Hackathon (Hacklytics 2021)](https://eugeneyan.com/writing/how-to-win-data-hackathon.md) (2021-02-14): What the top teams did to win the 36-hour data hackathon. No, not machine learning.
- [Growing and Running Your Data Science Team](https://eugeneyan.com/writing/data-science-teams.md) (2021-01-31): What I learned about hiring and training, and fostering innovation, discipline, and camaraderie.
- [You Don't Really Need Another MOOC](https://eugeneyan.com/writing/you-dont-need-another-mooc.md) (2021-01-24): Stop procrastinating, go off the happy path, learn just-in-time, and get your hands dirty.
- [Mailbag: How to Get Experienced DS Resume Noticed by Recruiters?](https://eugeneyan.com/writing/mailbag-resume-for-experienced-ds.md) (2021-01-16): How to increase the chances of getting called up by recruiters?
- [Real-time Machine Learning For Recommendations](https://eugeneyan.com/writing/real-time-recommendations.md) (2021-01-10): Why real-time? How have China & US companies built them? How to design & build an MVP?
- [2021 Roadmap: Sharing, Helping, and Living More](https://eugeneyan.com/writing/2021-roadmap.md) (2021-01-03): A public roadmap to track and share my progress; nothing mission or work-related.
- [2020 Retrospective: New Country, New Role, New Habit](https://eugeneyan.com/writing/retrospective-2020.md) (2020-12-20): Wrapping up 2020 with writing and site statistics, graphs, and a word cloud.
- [Catch the Flying Daggers](https://eugeneyan.com/writing/flying-dagger.md) (2020-12-11): A short story on flying daggers and life's challenges.
- [How I’m Reflecting on 2020 and Planning for 2021](https://eugeneyan.com/writing/how-i-reflect-and-plan.md) (2020-12-06): Time to clear the cache, evaluate existing processes, and start new threads.
- [Alexey Grigorev on His Career, Data Science, and Writing](https://eugeneyan.com/writing/informal-mentors-alexey-grigorev.md) (2020-11-29): How he switched from engineering to data science, what "senior" means, and how writing helps.
- [Mailbag: What's the Architecture for your Blog?](https://eugeneyan.com/writing/mailbag-blog-architecture.md) (2020-11-24): How did you set up your site and what's an easy way to replicate it?
- [What Machine Learning Can Teach Us About Life - 7 Lessons](https://eugeneyan.com/writing/life-lessons-from-machine-learning.md) (2020-11-22): Data cleaning, transfer learning, overfitting, ensembling, and more.
- [How to Prevent or Deal with a Data Science Role or Title Mismatch](https://eugeneyan.com/writing/role-title-mismatch.md) (2020-11-15): Interview questions you should ask and how to evolve your job scope.
- [Applied / Research Scientist, ML Engineer: What’s the Difference?](https://eugeneyan.com/writing/data-science-roles.md) (2020-11-08): A personal take on their deliverables and skills, and what it means for the industry and your team.
- [Chip Huyen on Her Career, Writing, and Machine Learning](https://eugeneyan.com/writing/informal-mentors-chip-huyen.md) (2020-11-01): Setbacks she faced, overcoming them, and how writing changed her life.
- [Data Discovery Platforms and Their Open Source Solutions](https://eugeneyan.com/writing/data-discovery-platforms.md) (2020-10-25): What questions do they answer? How do they compare? What open-source solutions are available?
- [Why I switched from Netlify back to GitHub Pages](https://eugeneyan.com/writing/netlify-back-to-github-pages.md) (2020-10-21): DNS server snafus led to email & security issues. Also, limited free build minutes monthly.
- [Why Have a Data Science Portfolio and What It Shows](https://eugeneyan.com/writing/data-science-portfolio-how-why-what.md) (2020-10-18): Not 'How to build a data science portfolio', but 'Whys' and 'Whats'.
- [How to Install Google Scalable Nearest Neighbors (ScaNN) on Mac](https://eugeneyan.com/writing/how-to-install-scann-on-mac.md) (2020-10-14): Step-by-step walkthrough on the environment, compilers, and installation for ScaNN.
- [How Prototyping Can Help You to Get Buy-In](https://eugeneyan.com/writing/prototyping-to-get-buy-in.md) (2020-10-11): Building prototypes helped get buy-in when roadmaps & design docs failed.
- [Is Writing as Important as Coding?](https://eugeneyan.com/writing/writing-and-coding.md) (2020-10-04): As careers grow, how does the balance between writing & coding change? Hear from 4 tech leaders.
- [RecSys 2020: Takeaways and Notable Papers](https://eugeneyan.com/writing/recsys2020.md) (2020-09-27): Emphasis on bias, more sequential models & bandits, robust offline evaluation, and recsys in the wild.
- [Appreciating the Present](https://eugeneyan.com/writing/present.md) (2020-09-26): What if the alternative was nothingness?
- [Routines and Tools to Optimize My Day (Guest Post by Susan Shu)](https://eugeneyan.com/writing/favorite-productivity-coffee-routines-habits.md) (2020-09-20): For years I've refined my routines and found tools to manage my time. Here I share it with readers.
- [How to Accomplish More with Less - Useful Tools & Routines](https://eugeneyan.com/writing/how-to-accomplish-more-with-less.md) (2020-09-13): My tools for organization and creation, autopilot routines, and Maker's schedule
- [Migrating Site Comments to Utterances](https://eugeneyan.com/writing/migrating-to-utterances.md) (2020-09-07): A step-by-step of how to migrate from json comments to Utterances.
- [How to Test Machine Learning Code and Systems](https://eugeneyan.com/writing/testing-ml.md) (2020-09-06): Checking for correct implementation, expected learned behaviour, and satisfactory performance.
- [Mailbag: Parsing Fields from PDFs—When to Use Machine Learning?](https://eugeneyan.com/writing/mailbag-pdf-fields.md) (2020-09-04): Should I switch from a regex-based to ML-based solution on my application?
- [How Reading Papers Helps You Be a More Effective Data Scientist](https://eugeneyan.com/writing/why-read-papers.md) (2020-08-30): Why read papers, what papers to read, and how to read them.
- [Mailbag: I'm Now a Senior DS—How should I Approach this?](https://eugeneyan.com/writing/mailbag-senior-ds.md) (2020-08-27): Becoming a senior after three years and dealing with imposter syndrome.
- [Embrace Beginner's Mind; Avoid The Wrong Way To Be An Expert](https://eugeneyan.com/writing/beginners-mind.md) (2020-08-23): How not to become an expert beginner and to progress through beginner, intermediate, and so on.
- [NLP for Supervised Learning - A Brief Survey](https://eugeneyan.com/writing/nlp-supervised-learning-survey.md) (2020-08-16): Examining the broad strokes of NLP progress and comparing between models
- [Unpopular Opinion: Data Scientists Should be More End-to-End](https://eugeneyan.com/writing/end-to-end-data-science.md) (2020-08-09): Why (and why not) be more end-to-end, how to, and Stitch Fix and Netflix's experience
- [Adding a Checkbox & Download Button to a FastAPI-HTML app](https://eugeneyan.com/writing/fastapi-html-checkbox-download.md) (2020-08-05): Updating our FastAPI app to let users select options and download results.
- [What I Did Not Learn About Writing In School](https://eugeneyan.com/writing/what-i-did-not-learn-about-writing-in-school.md) (2020-08-02): Surprising lessons I picked up from the best books, essays, and videos on writing non-fiction.
- [Georgia Tech's OMSCS FAQ (based on my experience)](https://eugeneyan.com/writing/georgia-tech-omscs-faq.md) (2020-07-26): Why OMSCS? How can I get accepted? How much time needed? Did it help your career? And more...
- [How to Set Up a HTML App with FastAPI, Jinja, Forms & Templates](https://eugeneyan.com/writing/how-to-set-up-html-app-with-fastapi-jinja-forms-templates.md) (2020-07-23): I couldn't find any guides on serving HTML with FastAPI, thus I wrote this to plug the hole on the internet.
- [Why You Need to Follow Up After Your Data Science Project](https://eugeneyan.com/writing/why-you-need-to-follow-up-after-your-data-science-project.md) (2020-07-19): Ever revisit a project & replicate the results the first time round? Me neither. Thus I adopted these habits.
- [What I Do During A Data Science Project To Deliver Success](https://eugeneyan.com/writing/what-i-do-during-a-data-science-project-to-ensure-success.md) (2020-07-12): It's not enough to have a good strategy and plan. Execution is just as important.
- [How to Update a GitHub Profile README Automatically](https://eugeneyan.com/writing/how-to-update-github-profile-readme-automatically.md) (2020-07-11): I wanted to add my recent writing to my GitHub Profile README but was too lazy to do manual updates.
- [The 85% Rule: When Giving It Your 100% Gets You Less than 85%](https://eugeneyan.com/writing/when-giving-your-100-gets-you-less-than-85.md) (2020-07-09): I thought giving it my all led to maximum outcomes; then I learnt about the 85% rule.
- [My Notes From Spark+AI Summit 2020 (Application-Specific Talks)](https://eugeneyan.com/writing/notes-from-sparkai-summit-application-specific.md) (2020-07-05): Part II of the previous write-up, this time on applications and frameworks of Spark in production
- [My Notes From Spark+AI Summit 2020 (Application-Agnostic Talks)](https://eugeneyan.com/writing/notes-from-sparkai-summit-application-agnostic.md) (2020-06-28): Sharing my notes & practical knowledge from the conference for people who don't have the time.
- [How to Set Up a Python Project For Automation and Collaboration](https://eugeneyan.com/writing/setting-up-python-project-for-automation-and-collaboration.md) (2020-06-21): After this article, we'll have a workflow of tests and checks that run automatically with each git push.
- [Mailbag: Qns on the Intersection of Data Science and Business](https://eugeneyan.com/writing/mailbag-ds-requirements.md) (2020-06-21): Does DS have business requirements? When does it make sense to split DS and DE??
- [Why Are My Airflow Jobs Running “One Day Late”?](https://eugeneyan.com/writing/why-airflow-jobs-one-day-late.md) (2020-06-17): A curious discussion made me realize my expert blind spot. And no, Airflow is not late.
- [What I Do Before a Data Science Project to Ensure Success](https://eugeneyan.com/writing/what-i-do-before-a-data-science-project-to-ensure-success.md) (2020-06-15): Haste makes waste. Diving into a data science problem may not be the fastest route to getting it done.
- [What I Love about Scrum for Data Science](https://eugeneyan.com/writing/what-i-love-about-scrum-for-data-science.md) (2020-06-07): Initially, I didn't like it. But over time, it grew on me. Here's why.
- [How to Apply Crocker's Law for Feedback and Growth](https://eugeneyan.com/writing/how-to-apply-crockers-law-for-feedback-and-growth.md) (2020-05-31): Crocker's Law, cognitive dissonance, and how to receive (uncomfortable) feedback better.
- [A Practical Guide to Maintaining Machine Learning in Production](https://eugeneyan.com/writing/practical-guide-to-maintaining-machine-learning.md) (2020-05-25): Can maintaining machine learning in production be easier? I go through some practical tips.
- [6 Little-Known Challenges After Deploying Machine Learning](https://eugeneyan.com/writing/challenges-after-deploying-machine-learning.md) (2020-05-18): I thought deploying machine learning was hard. Then I had to maintain multiple systems in prod.
- [How to Write: Advice from David Perell and Sahil Lavingia](https://eugeneyan.com/writing/how-to-write-david-x-sahil.md) (2020-05-09): An expansion of my Twitter thread that went viral.
- [A Hackathon Where the Dinkiest Idea Won. Why?](https://eugeneyan.com/writing/evaluating-ideas-at-a-hackathon.md) (2020-05-03): What I Learnt about evaluating ideas from first-hand participation in a hackathon.
- [Serendipity: Accuracy’s Unpopular Best Friend in Recommenders](https://eugeneyan.com/writing/serendipity-and-accuracy-in-recommender-systems.md) (2020-04-26): What I learned about measuring diversity, novelty, surprise, and serendipity from 10+ papers.
- [How to Give a Kick-Ass Data Science Talk](https://eugeneyan.com/writing/how-to-give-a-kick-ass-data-science-talk.md) (2020-04-18): Why you should give a talk and some tips from five years of speaking and hosting meet-ups.
- [Commando, Soldier, Police and Your Career Choices](https://eugeneyan.com/writing/commando-soldier-police-and-your-career.md) (2020-04-12): Should I join a start-up? Which offer should I accept? A simple metaphor to guide your decisions.
- [Stop Taking Regular Notes; Use a Zettelkasten Instead](https://eugeneyan.com/writing/note-taking-zettelkasten.md) (2020-04-05): Using a Zettelkasten helps you make connections between notes, improving learning and memory.
- [Writing is Learning: How I Learned an Easier Way to Write](https://eugeneyan.com/writing/reading-note-taking-writing.md) (2020-03-28): Writing begins before actually writing; it's a cycle of reading -> note-taking -> writing.
- [Simpler Experimentation with Jupyter, Papermill, and MLflow](https://eugeneyan.com/writing/experimentation-workflow-with-jupyter-papermill-mlflow.md) (2020-03-15): Automate your experimentation workflow to minimize effort and iterate faster.
- [My Journey from Psych Grad to Leading Data Science at Lazada](https://eugeneyan.com/writing/psych-grad-to-data-science-lead.md) (2020-02-27): How hard work, many failures, and a bit of luck got me into the field and up the ladder.
- [Beating the Baseline Recommender with Graph & NLP in Pytorch](https://eugeneyan.com/writing/recommender-systems-graph-and-nlp-pytorch.md) (2020-01-13): Beating the baseline using Graph & NLP techniques on PyTorch, AUC improvement of ~21% (Part 2 of 2).
- [Building a Strong Baseline Recommender in PyTorch, on a Laptop](https://eugeneyan.com/writing/recommender-systems-baseline-pytorch.md) (2020-01-06): Building a baseline recsys based on data scraped off Amazon. Warning - Lots of charts! (Part 1 of 2).
- [OMSCS CS6200 (Introduction to OS) Review and Tips](https://eugeneyan.com/writing/omscs-cs6200-introduction-to-operating-systems.md) (2019-12-15): OMSCS CS6200 (Introduction to OS) - Moving data from one process to another, multi-threaded.
- [OMSCS CS6750 (Human Computer Interaction) Review and Tips](https://eugeneyan.com/writing/omscs-cs6750-human-computer-interaction.md) (2019-09-03): OMSCS CS6750 (Human Computer Interaction) - You are not your user! Or how to build great products.
- [Goodbye Wordpress, Hello Jekyll!](https://eugeneyan.com/writing/goodbye-wordpress-hello-jekyll.md) (2019-08-25): Moving off wordpress and hosting for free on GitHub. And gaining full customization!
- [OMSCS CS6440 (Intro to Health Informatics) Review and Tips](https://eugeneyan.com/writing/omscs-cs6440-intro-to-health-informatics.md) (2019-08-04): OMSCS CS6440 (Intro to Health Informatics) - A primer on key tech and standards in healthtech.
- [OMSCS CS7646 (Machine Learning for Trading) Review and Tips](https://eugeneyan.com/writing/omscs-cs7646-machine-learning-for-trading.md) (2019-05-11): OMSCS CS7646 (Machine Learning for Trading) - Don't sell your house to trade algorithmically.
- [What does a Data Scientist really do?](https://eugeneyan.com/writing/what-does-a-data-scientist-really-do.md) (2019-04-30): No, you don't need a PhD or 10+ years of experience.
- [Data Science and Agile (Frameworks for Effectiveness)](https://eugeneyan.com/writing/data-science-and-agile-frameworks-for-effectiveness.md) (2019-02-02): Taking the best from agile and modifying it to fit the data science process (Part 2 of 2).
- [Data Science and Agile (What Works, and What Doesn't)](https://eugeneyan.com/writing/data-science-and-agile-what-works-and-what-doesnt.md) (2019-01-26): A deeper look into the strengths and weaknesses of Agile in Data Science projects (Part 1 of 2).
- [OMSCS CS6601 (Artificial Intelligence) Review and Tips](https://eugeneyan.com/writing/omscs-cs6601-artificial-intelligence.md) (2018-12-20): OMSCS CS6601 (Artificial Intelligence) - First, start with the simplest solution, and then add intelligence.
- [OMSCS CS6460 (Education Technology) Review and Tips](https://eugeneyan.com/writing/omscs-cs6460-education-technology.md) (2018-08-25): OMSCS CS6460 (Education Technology) - How to scale education widely through technology.
- [OMSCS CS7642 (Reinforcement Learning) Review and Tips](https://eugeneyan.com/writing/omscs-cs7642-reinforcement-learning.md) (2018-07-30): OMSCS CS7642 (Reinforcement Learning) - Landing rockets (fun!) via deep Q-Learning (and its variants).
- [Building a Strong Data Science Team Culture](https://eugeneyan.com/writing/building-a-strong-data-science-team-culture.md) (2018-05-12): Culture >> Hierarchy, Process, Bureaucracy.
- [OMSCS CS7641 (Machine Learning) Review and Tips](https://eugeneyan.com/writing/omscs-cs7641-machine-learning.md) (2017-12-27): OMSCS CS7641 (Machine Learning) - Revisiting the fundamentals and learning new techniques.
- [My first 100 days as Data Science Lead](https://eugeneyan.com/writing/my-first-100-days-as-data-science-lead.md) (2017-09-25): How being a Lead / Manager is different from being an individual contributor.
- [OMSCS CS6300 (Software Development Process) Review and Tips](https://eugeneyan.com/writing/omscs-cs6300-software-development-process.md) (2017-08-13): OMSCS CS6300 (Software Development Process) - Java and collaboratively developing an Android app.
- [How to get started in Data Science](https://eugeneyan.com/writing/how-to-get-started-in-data-science.md) (2017-06-25): Tools and skills to pick up, and how to practice them.
- [OMSCS CS6476 (Computer Vision) Review and Tips](https://eugeneyan.com/writing/omscs-cs6476-computer-vision.md) (2017-05-15): OMSCS CS6476 Computer Vision - Performing computer vision tasks with ONLY numpy.
- [One way to help a data science team innovate successfully](https://eugeneyan.com/writing/one-way-to-help-a-data-science-team-succeed.md) (2017-02-19): If things are not failing, you're not innovating enough. - Elon Musk
- [Product Categorization API Part 3: Creating an API](https://eugeneyan.com/writing/product-categorization-api-part-3-creating-an-api.md) (2017-02-13): Or how to put machine learning models into production.
- [Image search is now live!](https://eugeneyan.com/writing/image-search-is-now-live.md) (2017-01-14): A web app to find similar products based on image.
- [Product Classification API Part 2: Data Preparation](https://eugeneyan.com/writing/product-categorization-api-part-2-data-preparation.md) (2016-12-11): Cleaning up text and messing with ascii (urgh!)
- [Image classification API is now live!](https://eugeneyan.com/writing/image-categorization-is-now-live.md) (2016-11-27): A simple web app to classify fashion images into Amazon categories.
- [I'm going back to school](https://eugeneyan.com/writing/im-going-back-to-school.md) (2016-11-02): Got accepted into Georgia Tech's Computer Science Masters!
- [SortMySkills is now live!](https://eugeneyan.com/writing/sortmyskills-is-now-live.md) (2016-10-23): A card sorting game to discover youl passion by identifying skills you like and dislike.
- [Product Classification API Part 1: Data Acquisition](https://eugeneyan.com/writing/product-categorization-api-part-1-data-acquisition-and-formatting.md) (2016-10-11): Parsing json and formatting product titles and categories.
- [Thoughts on Functional Programming in Scala Course (Coursera)](https://eugeneyan.com/writing/thoughts-on-functional-programming-in-scala-course-coursera.md) (2016-07-31): Learning Scala from Martin Odersky, father of Scala.
- [First post!](https://eugeneyan.com/writing/first-post.md) (2016-07-06): Time to start writing.
- [DataKind Singapore’s Latest Project Accelerator](https://eugeneyan.com/writing/datakind-sg-project-accelerator.md) (2015-09-17): Guest post of how DataKind SG worked with NGOs to frame their problems and suggests solutions
