A Uniquely Interactive Experience

Abstract:
In recent years, Machine Learning models and architectures have become increasingly complex. This growing complexity makes it more difficult to deliver high quality in terms of model performance, robustness and explainability. The introduction of automated evaluations of the trustworthiness of a model is one solution to guarantee that your clients can rely on your model’s predictions. This talk will guide you through the different AI quality pillars, their importance and how to evaluate them.

What you’ll Learn:
– Which are the three AI quality pillars: performance, robustness & explainability
– Why is it important to evaluate the quality and trustworthiness of your ML model
– How can you integrate this evaluation seamlessly into your MLOps workflow

What is unique about this, which can’t be found online?
It gives exclusive insights into a novel quality assessment of ML models – aidkit is the first tool of its kind.

Technical Level: 4

Why is this takeaway(s) important?
Explainable AI and robustness are topics that have caught the attention of ML researchers in recent years and there are few online resources. Being a core contributor to aidkit, I have the expertise needed to explain the quality pillars and give exclusive insights on how to introduce these essential checks in an MLOps workflow.

María is a Machine Learning Research Engineer focused on AI robustness and explainability. She holds two degrees in Mathematics and Physics and is currently working at neurocat GmbH where she contributes to the core of aidkit, an AI quality and trustworthiness evaluation tool. She is also the founder of “NLP en ES 🤗” – the Hugging Face community of Spanish-speaking NLP professionals – and part of the Core Team of Women in AI & Robotics.

Abstract:
There is generally a gap between data scientists who create machine learning models and existing devops tools and processes to put those models into production. This can lead to models not being properly validated and isolated from appropriate business logic and associated monitoring for their proper use. Many models remain as research outputs and do not make it to production.

In this talk we will show how using Tempo (an open source Python SDK) data scientists can easily prepare and test locally their machine learning models and then deploy them either directly or as part of CI/CD and Gitops processes. Data scientists can be involved in the inference logic needed to call their models correctly and easily add other required inference components for their models such as outlier detectors and model explainers as part of their work.

We will illustrate deploying simple models to more advanced outlier detection, model explanation and multi-model ensembles with custom business logic. We will show models locally tested and then pushed to production on Kubernetes with Seldon or Kubeflow’s KFServing.

Data scientists will learn how they can get their work to production faster while ensuring their model is used correctly when it reaches production.

What you’ll Learn:
They will learn about different types of security vulnerabilities when it comes to machine learning and how to handle such issues.

What is unique about this, which can’t be found online?
Machine learning security is not a topic covered often/enough. So, this talk is unique in nature due to the subject matter alone.

Technical Level: 5

Why is this takeaway(s) important?
The end of a machine learning cycle involves taking a model into production where it is used on real world data. Up until this point, most practitioners have considered accuracy, bias, potential data drift, etc. However, security concerns are most likely not a big factor or may even be overlooked. This talk will cover common attacks and how to be prepare for such attacks, which is important for the following takeaway: A model that has been hacked exposes your organization and customers to many harmful outcomes, which should be avoided if at all possible.

Clive is CTO of Seldon. Seldon helps enterprises put machine learning into production. Clive developed Seldon’s open source Kubernetes based machine learning deployment platform Seldon Core. He is also a core contributor to the Kubeflow and KFServing projects.

Abstract:
As one of Canada’s largest grocery store chains, our team runs several ML systems on production. We wanted to take one such system and integrate it with several MLOps tools including MLflow, Seldon Core, and Feast on Google Cloud Platform. The goal is to reduce overall development time of a model from EDA to deployment, as well as enable end to end tracking of these ML pipelines.

In this talk, we will share our experience on setting up a recommender system using the tools mentioned. We will also talk about our overall platform architecture and how the MLOps tools fit into the end-to-end ML pipeline.

What you’ll Learn:
How to put a ML system on production using some of the most commonly used MLOps tools and how MLOps fit into the end to end ML pipeline.

What is unique about this, which can’t be found online?
This talk focuses on a specific e-commerce machine learning problem that we are solving here at Loblaw Digital using open source MLOps tools

Technical Level: 7

Why is this takeaway(s) important?
There are several MLOps tools out there but it is important to evaluate how they integrate together in production and what could be the challenges that arise when we go from EDA to Model Serving.

Mefta Sadat is a Senior ML Engineer at Loblaw Digital. He has been here for three years building the Data Engineering and Machine Learning platform. He focuses on productionizing ML services, tools and data pipelines. Previously Mefta worked at a Toronto based Video Streaming Company and designed and built the recommendation system for the Zoneify App from scratch. He received his MSc in Computer Science from Ryerson University focusing on research to mitigate risk in Software Engineering using ML.

Abstract:
Data Scientist has become the “sexiest” job title in the 21st century. However, we see significant challenges, especially in traditional industries, regarding model adoption and deploying ML to production. This is due to ML being mostly in academia for the last 70 years and not a lot of thought that went into operationalization of ML. A lot of Data Scientists are traditionally more focused on scientific approach (scripting and testing models) without considering operational aspects of it. We need to realize there must be more engineering rigour in developing ML models vs just calling model.fit. It’s also not as simple as taking everything from software development. When software development initially came from academia there were no proper development frameworks and early adopters borrowed traditional engineering processes (waterfall!). It took a long time to invent agile. I believe we have a similar situation with ML where people think we can just apply software development practices to ML development, whereas not everything is directly applicable. But we can take the best practices and build upon them. In this talk we will take a look at those challenges and explore potential solutions with switching model development into hands of ML Developers/Engineers that are focused on developing machine learning software systems (from scratch – data source) keeping in mind that it will be deployed on production.

What you’ll Learn:
Best practices of ML model development. We need to realize there must be more engineering rigour in developing ML models for them to be used in industry.

Technical Level: 3

Marcin holds a PhD in Software Engineering and Intelligent Systems from the University of Alberta, Canada, and an MSc in Applied Computing Science from AGH University of Science and Technology, Poland. He has published over 30 scientific, peer-reviewed publications and has over a decade of practical academic and industry experience with Software Engineering, Machine Learning, Big Data, and Predictive Analytics. Marcin has a passion for building and growing high performing Data Science teams, and for managing and coordinating Data Science, ML, and Big Data products. Currently, he is leading Data Science initiatives at AltaML as VP of Data Science. Marcin is also highly involved in Edmonton’s Data Science community, has co-founded the Edmonton DS meetup, and is teaching Data Science at North Alberta Institute of Technology.

Abstract:
How can a feature store help with MLOps?
In this talk, we introduce how a Feature Store for Machine Learning (ML) can decompose end–to-end ML pipelines into
(1) a feature pipeline that takes raw data from data warehouses, a data lake, and operational data stores and transforms it into features that are cached in the feature store, and
(2) model training/validation/deployment pipelines. Both pipelines have different requirements, often favor different technologies (Spark for feature pipelines, Python for model training), can run at different cadences, and are even managed by different teams (data engineering, and data scientists or ML engineers). The benefits of the Feature Store for ML architecture will be elucidated throughout the talk.

What you’ll Learn:
The state-of-the-art in structure ML pipelines with a Feature Store

What is unique about this, which can’t be found online?
We have experience running ML Pipelines in production with large companies and can share some of our recent experiences in when to use Spark or Python for feature engineering. When to use Deequ vs Great Expectations. Which open-source feature store is best for you?

Technical Level: 4

Jim Dowling is CEO of Logical Clocks and an Associate Professor at KTH Royal Institute of Technology. He is lead architect of the open-source Hopsworks platform, a horizontally scalable data platform for machine learning that includes the industry’s first Feature Store.

Abstract:
In this talk, I will present two recently developed tools for ML optimization: Skyline and RL-Scope. Skyline: Interactive in-editor computational performance profiling, visualization, and debugging for PyTorch deep neural networks. RL-Scope is a cross-stack profiler for deep reinforcement learning workloads.

What you’ll Learn:
The availability of tools for efficient optimization for DNN and RL models

Technical Level: 5

Gennady Pekhimenko is an Assistant Professor at the University of Toronto, CS department and (by courtesy) ECE department, where he is leading the EcoSystem (Efficient Computing Systems) group. Gennady is also a Faculty Member at Vector Institute and a CIFAR AI chair. Before joining Univ. of Toronto, he spent a year in 2017 at Microsoft Research in Redmond in Systems Research group. He got his PhD from the Computer Science Department at Carnegie Mellon University in 2016. Gennady is a recipient of Amazon Machine Learning Research Award, Facebook Faculty Research Award, Connaught New Researcher Award, NVIDIA Graduate, Microsoft Research, Qualcomm Innovation, and NSERC CGS-D Fellowships. His research interests are in the areas of systems, computer architecture, and applied machine learning.

Abstract:
In this talk we’ll share Ecolab’s journey with MLOps, and how this global hygiene technologies provider built a cutting-edge, cloud data science architecture which enables them to deploy new AI services every 30-90 days, to address risks before they occur.
Building scalable AI applications that generate value in real business environments require not just advanced technologies, but also better processes for data science, engineering and DevOps teams to collaborate effectively.

What you’ll Learn:
In this session we’ll touch upon the technology – and also the processes that need to be put in place to break the silos and efficiently implement MLOps across the enterprise

Yaron Haviv is a serial entrepreneur who has been applying his deep technological experience in data, cloud, AI and networking to leading startups and enterprise companies since the late 1990s. As the co-founder and CTO of Iguazio, Yaron drives the strategy for the company’s data science platform and leads the shift towards real-time AI. He also initiated and built Nuclio, a leading open source serverless platform with over 3,400 Github stars and MLRun, Iguazio’s open source MLOps orchestration framework.

Prior to co-founding Iguazio in 2014, Yaron was the Vice President of Datacenter Solutions at Mellanox (now NVIDIA), where he led technology innovation, software development and solution integrations. He was also the CTO and Vice President of R&D at Voltaire, a high-performance computing, IO and networking company which floated on the NYSE in 2007. Yaron is an active contributor to the CNCF Working Group and was one of the foundation’s first members. He presents at major industry events and writes tech content for leading publications including TheNewStack, Hackernoon, DZone, Towards Data Science and more.

Abstract:
The number of components and level of sophistication in end-to-end ML pipelines can vary from problem to problem but there’s one common element that is the key to make the whole system great and useful: your training data. The more time you spend developing the training dataset in your ML pipeline, the more positive results you’ll get. In this talk we’ll present the use case of a text classification pipeline we developed from scratch to integrate with one of our products. We’ll show details of how we designed an appropriate classification taxonomy, a consistent annotated training dataset, and how the end-to-end pipeline was pieced together to deploy a BERT-based model in a low latency real-time text classification system.

What you’ll Learn:
Audiences will learn strategies to develop a training dataset, particularly a nlp ready dataset, from scratch and how to connect it with the rest of the ML pipeline to train, monitor and deploy models for production use.

What is unique about this, which can’t be found online?
Most resources found online provide generic guides to develop and deploy language models using custom dataset. We intend fo give a very specific example that can be used in other contexts and domains.

Technical Level: 4

Why is this takeaway(s) important?
Recently, the availability of great pre-trained ML models ready to be fine-tuned with custom datasets has moved the focus of model selection to data development and curation. This makes end-to-end ML pipelines center around making sure your data is in its best possible shape and less on making sure you’re using the most optimal version of a chosen model. The use case I present in the context of NLP will highlight this data centricity as the key to achieving great model performance.

Diego Castaneda is a Data Scientist with a background in computational astrophysics. I currently support the needs of the messaging data team at Shopify.

Jennifer Bader is a Content Strategist. She works at Shopify, an e-commerce platform that makes it easier for people to start, run, and grow a business.

She joined Shopify to lead content strategy for Kit, Shopify’s AI-powered virtual employee. She now works on the Messaging team, which gives entrepreneurs tools to connect with their customers and collaborate with their team.

Jennifer has worked in UX content design and communications for the last 20 years. Before Shopify, she worked for OpenTable, IDEO, and a technology commercialization institute.

She’s passionate about the intersection of data science and content design. And using conversational interfaces to humanize e-commerce.

Abstract:
Hyperparameter optimization is a ubiquitous task, mostly treated as an expensive black-box optimization problem with high resource consumption. This talk will introduce MSR technologies to perform low-cost and effective hyperparameter optimization. We build FLAML: a fast and lightweight AutoML library to automatically find accurate machine learning models at low cost. It significantly outperforms top-ranked AutoML libraries on a large open source AutoML benchmark under equal, or sometimes orders of magnitude smaller budget constraints.

What you’ll Learn:
FLAML is a lightweight Python library that finds accurate machine learning models automatically, efficiently and economically. It frees users from selecting learners and hyperparameters for each learner. It is fast and economical. The simple and lightweight design makes it easy to extend, such as adding customized learners or metrics. FLAML is powered by a new, cost-effective hyperparameter optimization and learner selection method invented by Microsoft Research. It significantly outperforms top-ranked AutoML libraries on a large open source AutoML benchmark under equal, or sometimes orders of magnitude smaller budget constraints.

Technical Level: 2

Chi Wang is a principal researcher in Microsoft Research @ Redmond. His research is centered around the interplay between theories and systems related to data science and data platforms, including AutoML, ML for systems, and scalable data science. He received AMiner Most Influential Scholar Award in 2018 and 2016, and SIGKDD Data Science/Data Mining PhD Dissertation Award in 2015.

Dr. Qingyun Wu is currently a postdoc researcher at Microsoft Research, the New York City lab. Her research interests include AutoML and reinforcement learning. Qingyun will be joining Penn State University as an assistant professor in the fall of 2021.

Abstract:
Building and operating ML applications requires different infrastructure from traditional software, which has led to the development of “ML platforms” specifically designed to build and manage ML applications. In this talk, I’ll discuss some of the common challenges in productionizing ML based on experience building MLflow, an open source ML platform started at Databricks. MLflow is now the most widely used open source project in this area, with over 4 million downloads per month and integrations with dozens of other products. I’ll also highlight some interesting problems users face at scale, such as the need for versioning and reproducibility on petabyte-scale datasets, the impact of privacy and interpretability regulations on ML, and “hands-free” ML use cases that can train thousands of models without direct tuning from the ML developer. I’ll show how ongoing work in MLflow and the open source ecosystem around it (Delta Lake, PyTorch, PyCaret, Apache Spark, etc) is helping to tackle these problems.

What you’ll Learn:
Challenges running ML in production, based on our experience at thousands of enterprise customers, and features available in MLflow and its open source integrations.

Technical Level: 5

Matei Zaharia is a Cofounder and Chief Technologist at Databricks and an Assistant Professor of Computer Science at Stanford, where he does research on machine learning and distributed systems. Matei started the widely used Apache Spark and MLflow open source projects, and has contributed to other data and AI infrastructure projects such as Delta Lake and Koalas. Matei’s work was recognized through the 2014 ACM Doctoral Dissertation Award for the best PhD dissertation in computer science and the US Presidential Early Career Award for Scientists and Engineers (PECASE).

Abstract:
Released last year, the O’Reilly book Machine Learning Design Patterns captures best practices and solutions to recurring problems in machine learning. The authors, three Google engineers, catalog proven methods to help data scientists tackle common problems throughout the ML process. These design patterns codify the experience of hundreds of experts into straightforward, approachable advice. In this talk, Sara will dive into three patterns from the book focused on MLOps.

What you’ll Learn:
After this talk, attendees will have a better understanding of how to approach MLOps for their own ML tasks.

Technical Level: 5

Sara is a Senior Developer Advocate for Google Cloud, focusing on machine learning. She inspires developers and data scientists to integrate ML into their applications through demos, online content, and events. Sara also co-authored the O’Reilly book Machine Learning Design Patterns. Before Google, she was a Developer Advocate on the Firebase team. When she’s not writing code, she can be found on a spin bike or eating frosting.

Abstract:
For any organization whose core product or business depends on ML models (think Slack search, Twitter feed ranking, or Tesla Autopilot), ensuring that production ML models are performing with high efficacy is crucial. In fact, according to the McKinsey report on model risk, defective models have led to revenue losses of hundreds of millions of dollars in the financial sector alone. However, in spite of the significant harms of defective models, tools to detect and remedy model performance issues for production ML models are missing.

In this talk, we discuss why model monitoring matters, what do we mean by model monitoring, and considerations when setting up a model monitoring system.

What you’ll Learn:
Why model monitoring matters, what do we mean by model monitoring, and considerations when setting up a model monitoring system.

What is unique about this, which can’t be found online?
Model Monitoring is an emerging field with lots of confusion around what it entails. This talk will simplify the concepts and make them accessible to the audience.

Technical Level: 4

Why is this takeaway(s) important?
AI & ML models are being used in a variety of applications but there is inadequate monitoring in place to ensure that models work as expected. This session will raise awareness for model monitoring, what it entails and what the audience can do today to get started.

Manasi Vartak is the founder and CEO of Verta, an MIT spinoff building an open-core MLOps platform for the full ML lifecycle. Verta grew out of Manasi’s Ph.D. work at MIT on ModelDB, the first open-source model management system deployed at Fortune 500 companies. The Verta MLOps platform enables data scientists and ML engineers to robustly take trained ML models through the end-to-end MLOps cycle including versioning, packaging, release, operations, and monitoring. Previously, Manasi worked on feed ranking at Twitter and dynamic ad-targeting at Google. Manasi has spoken at several top research as well as industrial conferences such as the Strata O’Reilly Conference, SIGMOD, VLDB, Spark Summit, and AnacondaCON, and has authored a course on model management.

Abstract:
What is it like to build an ML platform during the pandemic, with a new team, at a new company?

DoorDash’s mission is to grow and empower local economies. As DoorDash’s business grows, it is essential to establish a centralized ML platform to accelerate the ML development process and to power the numerous ML use cases.

This presentation will detail the DoorDash ML platform journey during the pandemic, which includes the way we establish a close collaboration and relationship with the Data Science community, how we intentionally set the guardrails in the early days to enable us to make progress, the principled approach of building out the ML platform while meeting the needs of the Data Science community, and finally the technology stack and architecture that powers billions of predictions per day and supports a diverse set of ML use cases.

What you’ll Learn:
Building an ML platform during the pandemic is possible, but it requires a systematic approach and creative ideas.

Technical Level: 3

Hien Luu is a Sr. Engineering Manager at DoorDash, leading the Machine Learning Platform team. He is particularly passionate about the intersection between Big Data and Artificial Intelligence. He is the author of the Beginning Apache Spark 2 book. Teaching is one his passions and he is currently teaching Apache Spark course at UCSC Silicon Valley Extension school. He has given presentations at various conferences like QCon SF, QCon London, Hadoop Summit, JavaOne, ArchSummit and Lucene/Solr Revolution.

Dawn Lu is a senior data scientist at DoorDash, a technology company that empowers merchants to grow their businesses by offering on-demand delivery. She focuses primarily on developing and deploying machine learning models that power DoorDash’s logistics engine, including food ready time predictions, supply & demand forecasting, as well as pay algorithms. Before joining the technology industry, she worked as a strategy consultant — and during this time she discovered her passion for using quantitative approaches to better understand and predict consumer behavior. She holds a Bachelor’s degree in Economics from Yale University.

Abstract:
The recent paradigm shift of using data and algorithms to produce code (in the shape of models), called “Software 2.0”, has seen an acceleration of investment into various tools and platforms across the entire industry. Startups, cloud providers, internal platform at large companies: everyone needs an MLOps platform.

What you’ll Learn:
What it means to build an MLOps platform for AV.

Nico is a Sr Director of AI Infrastructure at NVIDIA. He is currently managing the team delivering the MLOps platform that enables the development of Autonomous Vehicles software for NVIDIA partners. Previously, Nico was managing the Cortex team at Twitter, responsible for delivering the deep learning platform for Twitter.

Abstract:
Graph neural networks (GNNs) research has surged to become one of the hottest topics in machine learning in the last years. GNNs have seen a series of recent successes in problems from the fields of biology, chemistry, social science, physics, and many others. So far, GNN models have been primarily developed for static graphs that do not change over time. However, many interesting real-world graphs are dynamic and evolving in time, with prominent examples including social networks, financial transactions, and recommender systems. In many cases, it is the dynamic behavior of such systems that conveys important insights, otherwise lost if one considers only a static graph. This talk will discuss Temporal Graph Networks, a recent and general approach for machine learning over dynamic graphs.

What you’ll Learn:
What dynamic graphs are, that they are ubiquitous in applications, and how to learn on dynamic graphs

Technical Level: 5

Emanuele is currently a Machine Learning Researcher at Twitter, as well as a PhD student at Imperial College. He obtained his Bachelor degree from Imperial College and his Masters Degree from Cambridge University. His research focuses on Graph Neural Networks, in particular developing new models that can handle dynamic graphs, i.e. graphs that change over time.

Abstract:
In this talk we will examine the Mayflower project — its goals; its breakthroughs; its technology — and highlight it as an example of how AI is being brought to the edge.

What you’ll Learn:
Edge Computing

Technical Level: 5

Rob High is the Vice President and Chief Technology Officer for the IBM Edge Computing portfolio. High has architecture and technical strategy responsibility for offerings that will enable a new generation of edge computing — where work is placed close to where data is produced and actions are performed. Prior to this assignment he was the CTO for IBM Watson, responsible for setting the technical strategy and vision for IBM’s leadership in AI computing. High is recognized as a global technical leader, author and evangelist, and was named an IBM Fellow in 2008 for his pioneering work in the field of SOA and for his leadership on WebSphere.

Abstract:
As more businesses adopt AI, upcoming AI regulations are forcing companies to build responsible and ethical AI. Previously, many practitioners thought validating a model with Explainable AI before deployment was enough. Not anymore. With more AI solutions in place, firms are putting governance at the forefront by enforcing it throughout the AI lifecycle to ensure responsible and ethical AI development. In this session, we introduce a Model Performance Management framework, its advantages in the machine learning life cycle and providing a continuous feedback loop to establish a responsible AI practice.

What you’ll Learn:
Using Model Performance Management to improve models more efficiently and how AI explainability enables MPM at the core.

Krishna is the co-founder and CEO of Fiddler, a Machine Learning Monitoring company that empowers companies to efficiently monitor and troubleshoot ML models with Explainable AI. Prior to founding Fiddler, Gade led the team that built Facebook’s explainability feature ‘Why am I seeing this?’. He’s an entrepreneur with a technical background with experience creating scalable platforms and expertise in converting data into intelligence. Having held senior engineering leadership roles at Facebook, Pinterest, Twitter and Microsoft, he’s seen the effects that bias has on AI and machine learning decision making processes, and with Fiddler, his goal is to enable enterprises across the globe solve this problem.

Abstract:
Data owners often desire to publish a synthetic data instance that ensures the privacy of individual records, but is similarly useful as the original sensitive data. Existing differentially private data synthesis methods usually fail in preserving one of the most fundamental data properties: the underlying correlations and dependencies among tuples and attributes (i.e., the structure of the data), compromising the utility of the output to downstream tasks.

With probabilistic database models and show how to: (i) privately learn a probabilistic model based on the original private data (how data was generated); and (ii) sample a new synthetic private instance based on the learned model. We leverage the HoloClean framework to learn the underlying data distribution, and we show that learning this model privately is technically challenging.

In this talk I briefly describe the HoloClean framework for structured data prediction, and Kamino a system that learns HoloClean models privately and use them to synthesize “useful” private data instances.

What you’ll Learn:
They will learn about different types of security vulnerabilities when it comes to machine learning and how to handle such issues.

What is unique about this, which can’t be found online?
Machine learning security is not a topic covered often/enough. So, this talk is unique in nature due to the subject matter alone.

Technical Level: 5

Why is this takeaway(s) important?
The end of a machine learning cycle involves taking a model into production where it is used on real world data. Up until this point, most practitioners have considered accuracy, bias, potential data drift, etc. However, security concerns are most likely not a big factor or may even be overlooked.

This talk will cover common attacks and how to be prepare for such attacks, which is important for the following takeaway: A model that has been hacked exposes your organization and customers to many harmful outcomes, which should be avoided if at all possible.

Ihab Ilyas is a professor in the Cheriton School of Computer Science and the NSERC-Thomson Reuters Research Chair on data quality at the University of Waterloo. His main research focuses on the areas of big data and database systems, with special interest in data quality and integration, managing uncertain data, machine learning for data curation, and information extraction.

Ihab is a co-founder of Tamr, a startup focusing on large-scale data integration, and he is also the co-founder of inductiv (acquired by Apple), a Waterloo-based startup on using AI for structured data cleaning. He is a recipient of the Ontario Early Researcher Award, a Cheriton Faculty Fellowship, an NSERC Discovery Accelerator Award, and a Google Faculty Award, and he is an ACM Fellow.

Abstract:
At Avast we complete over 17 million phishing detections a day, providing crucial online protection for this type of attacks. There are a lot of challenges that data scientists frequently face, such as inconsistent environments, transitioning from research to production, access management, model deployment.

In this talk Joao Da Silva will present Avast MLOps maturity journey, tooling and cultural shift that has increased velocity, improved collaboration and brought structure for productising machine learning whilst integrating model tracking, storage, cross-system orchestration and E2E model deployments for complete and modern machine learning pipelines

Joao Da Silva will deliver a presentation that anyone in any organisation can relate to and take notes on a real journey with regards to MLOps tooling, mindset and adoption in a large organisation.

What you’ll Learn:
Audience will learn about a real journey in improving MLOps mindset and maturity levels in a organisation whilst proving information on the tooling and strategies created for it to become a reality.

What is unique about this, which can’t be found online?
It’s the true MLOps strategy, Architecture and platform of Avast which came to fruition after initial adopting of principles and MATS stack as the first step into productising anti-virus and phishing protection research

Technical Level: 4

Why is this takeaway(s) important?
As part of our journey we identified that Avast needed to shift it’s mindset and culture around ML delivery, it’s also true from my experience from our local that many organisations are still reluctant to adopt a MLOps strategy. This talk will present that with a MLOps maturity level based strategy, minor adjustments and small investment MLOps mindset can increase their output and ML quality greatly

Public speaker in spark-summit, data-summit and local ML Meetups as well as Meetup co-organizer in Prague. I have been involved in the Data engineering field for a while and I am a FP lover. I am currently Architecting MLOps platform and strategy for Avast so we can provide a safe online environment to the wider population.

Abstract:
Advances in machine learning and big data are disrupting every industry. However, even when companies deploy to production, they face significant challenges in staying there, with performance degrading significantly from offline benchmarks over time, a phenomenon known as performance drift. Models deployed over an extended period of time often experience performance drift due to changing data distribution.

In this talk, we discuss approaches to mitigate the effects of performance drift, illustrating our methods on a sample prediction task. Leveraging my experience at a startup deploying and monitoring production-grade ML pipelines for predictive maintenance, we also address several aspects of machine learning often overlooked in academia, such as the incorporation of non-technical collaborators and the integration of machine learning in an agile framework.

What you’ll Learn:
This talk will consist of:
– Using Python and open-source datasets to demonstrate an example of training and validating a model in an offline setting that subsequently experiences performance degradation when it is deployed.
– Using Prometheus, Grafana, and mltrace to show how one can build tools to monitor production pipelines and enable teams of different stakeholders to quickly identify performance degradation using the right metrics.

This talk will be a slideshow presentation accompanied by a Python notebook demo. It is aimed towards engineers that deploy and debug models in production, but may be of broader interest for people building machine learning-based products, and requires a familiarity with machine learning basics (train/test sets, decision trees).

Technical Level: 5

Shreya is a computer scientist living in San Francisco interested in making machine learning work in the “real world.” She is an incoming PhD student at UC Berkeley. Previously, she was the first ML engineer at Viaduct, did ML research at Google Brain, and completed her BS and MS in computer science at Stanford.

Abstract:
There are several known attacks against ML models that can lead to altered, harmful model outcomes or exposure of sensitive training data. Unfortunately, traditional model assessment measures don’t tell us much about whether a model is secure. In addition to other debugging steps, it may be prudent to add some or all of the known ML attacks into any white-hat hacking exercises or red-team audits your organization is already conducting. This talk will go over common machine learning security attacks and the remediation steps an organization can take to deter these pitfalls.

What you’ll Learn:
They will learn about different types of security vulnerabilities when it comes to machine learning and how to handle such issues.

What is unique about this, which can’t be found online?
Machine learning security is not a topic covered often/enough. So, this talk is unique in nature due to the subject matter alone.

Technical Level: 5

Why is this takeaway(s) important?
The end of a machine learning cycle involves taking a model into production where it is used on real world data. Up until this point, most practitioners have considered accuracy, bias, potential data drift, etc. However, security concerns are most likely not a big factor or may even be overlooked. This talk will cover common attacks and how to be prepare for such attacks, which is important for the following takeaway: A model that has been hacked exposes your organization and customers to many harmful outcomes, which should be avoided if at all possible.

Navdeep Gill is a Lead Data Scientist, Team Lead at H2O.ai where he leads efforts around Responsible AI and previously focused on GPU accelerated machine learning, automated machine learning, and the core H2O-3 platform.

Prior to joining H2O.ai, Navdeep worked at Cisco focusing on data science and software development. Before that Navdeep was a researcher/analyst in several neuroscience labs at the following institutions: California State University, East Bay, University of California, San Francisco, and Smith Kettlewell Eye Research Institute.

Navdeep graduated from California State University, East Bay with a M.S. in Computational Statistics, a B.S. in Statistics, and a B.A. in Psychology with a minor in Mathematics.

Michelle is a Data Scientist and Product Manager of the AI App Store at H2O.ai. Her background is in pure math and computer science and she is passionate about applying these skills to answer real world questions. When not coding or thinking of analytics, Michelle can be found hanging out with her dog or playing rocking out with the band.

Abstract:
The day an ML application is deployed to production and begins facing the real world invariably begins in triumph and ends in frustration for the model builder. The joy of seeing accurate predictions is quickly overshadowed by a myriad of operational challenges that arise, from debugging to troubleshooting to monitoring. In DevOps, analogous software operations have long been refined into an artform. Sophisticated tools enable engineers to quickly identify and resolve issues, continuously improving software stability and robustness. By contrast, in the ML world, operations are still done with Jupyter notebooks and shell scripts. One of the cornerstones of the DevOps toolchain is logging. Traces and metrics are built on top of logs, thus enabling monitoring and feedback loops. What would a good logging tool look like in an ML system? In this talk, we present a powerful new tool—a logging library—that enables data logging for AI applications. We discuss how this solution enables testing, monitoring and debugging of both an AI application and its upstream data pipeline(s). We offer a deep dive into some of the key properties of the logging library that enable it to handle TBs of data, run with a constraint memory footprint, and produce statistically accurate log profiles of structured and unstructured data. Attendees will leave the talk equipped with best practices to supercharge MLOps in their teams.

What you’ll Learn:
How to enable logging in data & ml applications in order to unlock MLOps activities such as testing, debugging, and monitoring.

With a joint background in data science and engineering, Gulsen leads data product and AI/ML efforts globally across Udemy, including the development of Udemy’s catalog search, recommendations, knowledge graph, and personalization technologies as well as ML platform. She has 15+ years of experience in the field of software engineering and management with a particular focus on ML applications and platforms for the last 7+ years. She has 4+ years of experience in the field of data science including hands-on experience in search and recommendations data science. Prior to Udemy, Gulsen worked in various areas including project management, full-stack development, game development, and mobile applications.

Alessya Visnjic is the CEO and co-founder of WhyLabs, the AI Observability company on a mission to build the interface between AI and human operators. Prior to WhyLabs, Alessya was a CTO-in-residence at the Allen Institute for AI (AI2), where she evaluated commercial potential for the latest advancements in AI research. Earlier in her career, Alessya spent 9 years at Amazon leading Machine Learning adoption and tooling efforts. She was a founding member of Amazon’s first ML research center in Berlin, Germany. Alessya is also the founder of Rsqrd AI, a global community of 1,000+ AI practitioners who are committed to making AI technology Robust & Responsible.

Abstract:
This talk describes Marius, a software system that aims to make scaling of modern AI models over billion-edge graphs dramatically easier. Marius focuses on a key bottleneck in the development of machine learning systems over large-scale graph data: data movement during training. Marius addresses this bottleneck with a novel data flow architecture that maximizes resource utilization of the entire memory hierarchy (including disk, CPU, and GPU memory). Marius’ no-code paradigm allows users to simply define a model and enjoy resource-optimized training out of the box. This talk will describe how Marius can train deep learning models over graphs with more than a billion edges and 550GB of total parameters 10x faster and 5x cheaper than competing industrial systems.

What you’ll Learn:
1) How we can enable training of large-scale machine learning models over sparse structured data in a resource-optimizer manner
2) Learn about Marius and its functionalities

Technical Level: 5

Theodoros (Theo) Rekatsinas is an Assistant Professor in the Department of Computer Sciences at the University of Wisconsin-Madison. His work focuses on machine learning methods and systems over structured data. Theo is also a co-founder of Inductiv (now part of Apple), which developed technology that uses artificial intelligence to automate processes that involve identifying and correcting errors in data.

Abstract:
At Udemy, multi-faceted growth created different challenges in terms of ML platform and tooling as well as the related processes. One such challenge was to come up with a scalable platform to train and execute different types of ML models in real-time or batch. We overcame this particular challenge by building generic components that increased reuse and led to faster delivery and lower maintenance costs. Another challenge was to find a good way to efficiently serve the needs of the different parts of the organization despite their varying requirements. For instance, distributed applications cannot be avoided for many of our use cases that deal with web-scale data, but there are still a significant number of cases which can be more efficiently handled without the complexities of distributed applications.

To overcome this challenge, we had to unify the frameworks (and tooling whenever possible) for both cases, mainly to decrease the learning curve for data scientists and increase the maintainability.
Another challenge we faced was the need for increased focus on developer and data science ergonomics as the organization grew.

In this talk, we will present an overview of the main ML-related challenges we faced at Udemy as the company experienced explosive growth in users, product complexity, and organization size. We will also present best practices that we developed and outstanding problems in our current state.

What you’ll Learn:
1) Best practices for ML operations in different company sizes
2) Challenges for ML-related areas in real-life settings

What is unique about this, which can’t be found online?
The content is based on years of observations and trial-and-error experiments in a fast-growing company. Given that every stage of growth introduced different challenges, the experience conveyed in this talk will be a unique one.

Technical Level: 3

With a joint background in data science and engineering, Gulsen leads data product and AI/ML efforts globally across Udemy, including the development of Udemy’s catalog search, recommendations, knowledge graph, and personalization technologies as well as ML platform. She has 15+ years of experience in the field of software engineering and management with a particular focus on ML applications and platform for the last 7+ years. She has 4+ years of experience in the field of data science including hands-on experience in search and recommendations data science. Prior to Udemy, Gulsen has worked in various areas including project management, full-stack development, game development, and mobile applications.

Sam Cohan is a Principal Machine Learning Engineer at Udemy. Sam’s current focus at Udemy is enabling efficient workflows for building data products. Prior to joining Udemy, Sam was an early employee and founding team member at multiple data-heavy startups in the Bay Aarea ranging from revenue intelligence to mobile advertising to crypto trading. Sam is a big believer in the power of building teams that can blur the lines between engineering and data science. Sam has a degree in Computer Engineering from the University of Waterloo and a Masters in Financial Engineering from UC Berkeley.

Abstract:
At Yelp, we use ElasticSearch(ES) to power most of our Search. However, the process for updating a model or replacing it was slow and error-prone; engineers needed to spend time implementing any new feature transformations when training and serving, as well as ensuring parity between the two. We improve this situation by building an ES plugin that integrates neatly into Yelp’s Model Platform. Spark Pipelines trained and stored as MLeap bundles to MLFlow using the Model Platform can now be deployed directly to ES as MLeap pipelines. These Spark/MLeap pipelines encapsulate not only the ML model itself but also it’s feature engineering pipeline. Subsequently, this allows the ES plugin to swap one modeling pipeline for another as long as the base features for their pipelines are available through the ES index!

In this talk, we will discuss this ES plugin, as well as lessons learnt in making model pipelines performant.

What you’ll Learn:
The audience will walk through a working framework for easy training and deployment of ML models for online ranking tasks. They will learn about pitfalls we ran into when trying to make our models more performant.

What is unique about this, which can’t be found online?
A framework for quick model deployments when powering search. Experiences in making model pipelines more performant.

Technical Level: 4

Why is this takeaway(s) important?
ML models for online ranking tasks are used widely. However, automated frameworks for easy training and deployment of such models are not.

Software Engineer with the ML Infrastructure team at Yelp. PhD with a focus on Randomized and Distributed Algorithms from the University of Iowa.

Abstract:
MLOps and ModelOps are different. In this session, we will cover how ModelOps not only encompasses the basic model monitoring and tuning capabilities of MLOps, but also includes a continuous feedback loop and 360 view of your models throughout the enterprise, providing reproducibility, compliance and auditability of all your business critical models.

What you’ll Learn:
You will learn about good practices for:
– Continuous model monitoring for early problem detection
– Automated remediation for accelerating time to resolution
– Establishing a continuous feedback loop model improvement

What is unique about this, which can’t be found online?
We will cover model operational practices that you can apply to AI models along with other types of analytical models.

Jim is the CTO at ModelOp, where he leads the technical design of ModelOp Center, the leading ModelOps platform. Jim has been a technical innovator for the past 30 years, leading architectural design for products at a variety of companies, including Teradata, Qualtrics, and Novell. Jim holds two software patents and has taught several ModelOps master classes. For the last year, Jim has been living off the grid in the Rocky Mountains of Colorado.