DevConf.cz 2021

In this talk you will learn how to build an end-to-end analytics application using (almost) only jupyter notebooks as the basic unit of development.

When developing AI driven applications, there is often a friction point in the development life cycle when porting your code from a Data Scientistโ€s experimental notebook into the production ready code a Software Engineer expects. What if we could find a solution to this issue, and let the notebooks themselves be run in a DAG workflow, connected to each other, allowing them to share and exchange data and avoid this porting step all together, making it simpler for data scientists and software engineers to collaborate and quickly iterate on their AI driven application?

We will walk you through a case study where we did just that. Using the Open Data Hub toolkit, specifically: Jupyterhub, Ceph, Hive, Hue, Superset and Argo on Openshift, to build a recurring email list analytics and dashboard application. Highlighting some pitfalls we made along the way, how we could improve in the future with Elyra and how this process is general enough to be applied to many AI driven application development use cases.