3#3 - Data Management as Code (Eng)

MetaDAMA - Data Management in the Nordics

«There should be very little reason to say: Hey, I need a human to look these operational things for me. They are all defined as code.»
Lars Albertsson has a long career in Data and Software Engineering, including Google and Spotify. Lars is on a mission to spread the superpowers of working with data, with the vision to: «Enable companies outside of the absolute technical elite to work with data with the same efficiency or effectiveness as the technical elite companies in an industrial manner.»

4 types of companies:

1. Born digital - Data is the basis of their business model.
2. Born digital in a traditional market - completely natural to use data as a competitive advantage.
3. Traditional industries «born before the internet» - big difference wether they handle information or are in the physical world.
4. Information Handlers - Banks, Media, etc have digitalized their whole activity chain a long time ago.

The differences

• Significant differences in cycle-time in different industries and businesses.
• The only way to beat this cycle is to try out, fail fast, learn, try again.
• «Successful companies have been really good at failing fast.»
• Fast moving cultures are more effective and therefore have a better risk focus, without slowing down.
• To move fast in a slow moving industry, you need to choose your technology and approach wisely, keeping complexity down .
• Cultural slowness - «The challenge to change the way people work and people think is extraordinarily difficult.»
• Risk and Governance are addressed by rituals, rather then tasks.
• The value chain data to client outcome, needs to be anchored in a company. Have a clear picture of what this means.

Getting close

• Success can be measured by how close you are to the end user. The closer you get to a customer, the better the changes of success.
• «There is no substitute in value creation, than talking to the people you actually want to make happy.»

Automation is Innovation

• You need to find ways to ignite people's domain innovation capacity.
• Automation is a gradual process. People don’t loose their work to machines over night.
• Human-oversight is still really important, and there is a long journey with humans as part of the process.
• The focus on automation now is in knowledge workers, yet those have a different stand in society and are able to resist better, compared to the workforce during the Industrial Revolution.
• «If it changes quicker than one generation, there won’t be natural attrition that matches the changes in the need of the workforce.»

Automated Data Management

• Automating and industrializing data management processes is lower risk then software development, but still not as common.
• Great value to gain, from delaying simple automation processes to data management.
• You need to build everything from raw data to end product to find ways to automate.
• The raw data is the soul of the end product and the other way around. You need to keep these two outer points of the pipeline in mind, when think of data quality and data products.
• The limitations in Hadoop forced to work in a certain way. That way can be adopted to data management.
• Hadoop really pushed people in the functional Big-data patterns, that are still the basis of much of the work we are doing today.
• Workflow orchestration can help to know, which data you choose for a certain computation.
• Data Management as code is an area that is underdeveloped and under-appreciated.
• Minimize the technical barriers from Governance, and focus on the social aspects.

Ford CEO on Software: https://www.youtube.com/shorts/HrNN6goQe50