Inside The Max Inference Stack A Technical Deep Dive

Summary

me: X: x.com/calebfoundry LinkedIn: linkedin.com/in/calebeom/ TikTok: ... A single token of KV cache on Mistral 7B costs 131 KB. Multiply that by 16000 tokens of context and 80 concurrent users and the ... What happens when an AI agent starts working, and how do we measure its performance? In this episode of Why does a 70B language model crawl at 8 tokens per second on one setup, then feel instant on another? The difference is ... Today we have Philip Kiely from Baseten on the show. Baseten is a Series B startup focused on providing infrastructure for AI ... Why are your expensive GPUs sitting idle while your text generation maxes out? In this complete guide to LLM Open-source LLMs are great for conversational applications, but they can be difficult to scale in production and deliver latency ... Inferact CEO and co-founder Simon Mo joins Lightspeed partners Bucky Moore and James Alcorn to break down why Most agentic LLM workflows are surprisingly inefficient. In this Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and ... Zoom link: us02web.zoom.us/j/82308186562 Talk Introductions and Meetup Updates by Chris Fregly and Antje Barth ...