Lessons Learned Building LLM Training/Inference Platforms

There are two main obstacles that make LLMOps harder than classical MLOps: data gravity (the models and datasets are big) and GPUs. How do I find ‘affordable’ GPUs? Neoclouds are cheaper, but they are of mixed quality. What about minimizing GPU idle time? You can treat your GPUs like a job queue, but how do you share between training and serving workloads? How do you make a queue ‘ergonomic’ for researchers to use? How do you make runs reproducible and avoid dependency hell? uv is great, but it makes you ‘solve’ dependencies by hand when it comes to CUDA. I do not have all the answers, but I will share many things I learned through working on these problems.