3#8 - Software Development: An Inspiration for Data Management? (Eng)

MetaDAMA - Data Management in the Nordics

«The journey Software development went through during the last 10 years, working towards DevOps and agile development, is something that we can really benefit from in the data space.»
Uncover the synergy between agile software development and data management as we sit down with Alexandra Diem, head of Cloud Analytics and MLOps at Gjensidige, who bridges the gap between these two dynamic fields. In a narrative that takes you from the structured world of mathematics to the true data-driven insurance data sphere, Alexandra shares her insights on Cloud Analytics, Software Development, Machine Learning and much more. She illustrates how software methodologies can revolutionize data work.
This episode peels back the layers of MLOps, drawing parallels with the established tenets of software engineering. As we dissect the critical role of continuous development, automated testing, and orchestration in data product management, we also navigate the historical shifts in software project strategies that inform today's practices. Our conversation ventures into the realm of domain knowledge, product mindset, and federated governance, providing you with a well-rounded understanding of the complexities at play in modern data management.
Finally, we cast a pragmatic eye over the challenges and solutions within data engineering, advocating for a focus on practical effectiveness over the elusive pursuit of perfection. With Alexandra's expert perspective, we delve into the strategy of time-boxed approaches to data product development and the indispensable role of cross-functional teams. Join us for an episode that promises to enrich your view on the interplay between software and data.
Here are some key takeaways:

• There is a certain push in the insurance industry towards data, AI and autiomation.
• Gjensidige has over 20 decentralized analyst teams.
• Data Mesh is about empowering analyst teams to take control over their data.
• By taking responsibility over their own data, analyst teams take off the load from Data engineering teams, so they can focus on the tricky stuff.
• MLOps, DataOps, or classic DevOps in the Data Space is about using System Development principles in the Data Space.
• The questions that arise within data today, are questions that software engineering went through 10 years ago.
• Software development also went through a maturing, that brought forth a domain driven focus, best practice focus, product thinking, etc.
• Documentation should live, where the code also lives. It should be part of the code.
• Introduce more software development best practices into the data teams.
• Do not think about the solution you want to develop, but the problem you want to solve.
• Time-box exploratory efforts into sprints.

The pitfalls

1. Software Development Lifecycle vs. Data Lifecyle – they overlap, but there are clear differences, especially in the late phases.
2. Feature-driven (or functionality-driven) vs. Data-driven: Is there a problem with software engineering mindset in data?
3. Hypothesis - Data Science vs. Engineering mindset: Explorational vs. structural thinking can cause friction
4. Environmental challenges: How does Test-Dev-Prod split fit with data?