Code Generation & Synthetic Data With Loubna Ben Allal #51 Artwork

AI Stories

Artificial Intelligence, Machine Learning, Data Science and Deep Learning are completely changing the world we live in today. Companies around the world start to make sensible use of big data to influence business decisions and create our future. From video recommendations to autonomous driving, from stock prediction to weather forecasting, the AI revolution is everywhere. The AI stories podcast brings together some of the best Data Scientists, Machine Learning Engineers, Business leaders and researchers that are at the front of this revolution. They are here to talk about their career, how they arrive where they are, give advice and share their vision. They explain how they make use of AI in their daily routine, how they use algorithms to solve business problems and make the world a better place. They are here to share their stories: their AI stories. Hosted by Neil Leiser, Data Scientist at Iwoca. Follow Neil to learn more about career, Data Science, AI and Machine Learning. Linkedin: https://www.linkedin.com/in/leiserneil/ Twitter: https://twitter.com/LeiserNeil

All Episodes

AI Stories

Code Generation & Synthetic Data With Loubna Ben Allal #51

November 07, 2024 • Neil Leiser • Season 4 • Episode 2

Our guest today is Loubna Ben Allal, Machine Learning Engineer at Hugging Face 🤗 .

In our conversation, Loubna first explains how she built two impressive code generation models: StarCoder and StarCoder2. We dig into the importance of data when training large models and what can be done on the data side to improve LLMs performance.

We then dive into synthetic data generation and discuss the pros and cons. Loubna explains how she built Cosmopedia, a dataset fully synthetic generated using Mixtral 8x7B.

Loubna also shares career mistakes, advice and her take on the future of developers and code generation.

If you enjoyed the episode, please leave a 5 star review and subscribe to the AI Stories Youtube channel.

Cosmopedia Dataset: https://huggingface.co/blog/cosmopedia

StarCoder blog post: https://huggingface.co/blog/starcoder

Follow Loubna on LinkedIn: https://www.linkedin.com/in/loubna-ben-allal-238690152/

Follow Neil on LinkedIn: https://www.linkedin.com/in/leiserneil/

---

(00:00) - Intro

(02:00) - How Loubna Got Into Data & AI

(03:57) - Internship at Hugging Face

(06:21) - Building A Code Generation Model: StarCoder

(12:14) - Data Filtering Techniques for LLMs

(18:44) - Training StarCoder

(21:35) - Will GenAI Replace Developers?

(25:44) - Synthetic Data Generation & Building Cosmopedia

(35:44) - Evaluating a 1B Params Model Trained on Synthetic Data

(43:43) - Challenges faced & Career Advice