DevOps approach to creating demo data?

Question

Is there a good DevOps method to construct a workflow that creates demo data for a product? This situation is difficult when the app constantly changes. There are a few scenarios to consider.

1. Either old demo data becomes obsolete in the new build or the old data can be considered corrupted in the new build.

2. New business logic is introduced which requires interaction with the app for the new data to be formed. You cannot look at the new schema, static rules or old data to derivative/fill in such data.

3. Program managers have the insider info on what the demo data needs to look like, not the engineers. So the workflow needs to match up during the process, from (build -> test -> deploy) somehow.

One approach is to run tests that have demo data in it. So after the tests were ran, we can extract the data for our demos. But I can see that the tests can be very long and not as instant as importing a csv file. Also the coordination between PMs and engineers seems to be tightly coupled.

Another approach is to start with the new scheme from the repository and generate data from there. But this requires overhead of creating rules for generation which may not be in sync with the logic of app itself.

It seems a top down approach is better, so you generate dynamic data sets per release. Instead of working with a backward compatibility mindset with older static data.

The problem is to keep the app and the data in sync and valid for each iteration, while allowing the PMs and engineers to work in parallel from one another without relying on each other.

Answer 1

Welcome to DevOps SE!

Is it possible to make data management and operation of data sets to be an asset on its own? It seems like data has become more important so a more data-centric approach could help.

Program managers have the insider info on what the demo data needs to look like

This proves also organizational silo and lack of communication in the process: even before DevOps, in an agile team e.g. scrum you could have not only engineers but also a product owner and testers who take responsibility to provide not only useful tests but also useful testdata to the team.

That is, the fourth possible scenario could be that you take a data set from production and scramble/anonymize it. If it is too large, you need to reduce it to a representative mix. A DevOps way to do it would be to automate all these steps after they are well understood. => Working real life scenario.

There are some legal workaround methods though which require though considerable additional effort:

• data scrambling
• data anonymization
• synthetic data

Depending on your domain, there could be also applicable and more or less flexible enterprise tools for master data management where product owners could manage and version various data profiles.

Since DevOps approach is more than just finding a tool to use, but also doing experiments to find out more, here are examples of recent academic research on these topics.

An example study: Going Beyond Obscurity: Organizational Approaches to Data Anonymization (Hargitai et al. 2018)

Rather than being a purely technical question of applying the right algorithms, anonymization in practice is a complex socio-technical process that relies on multi-stakeholder collaborations.

Or: Machine learning using synthetic and real data: Similarity of evaluation metrics for different healthcare datasets and for different algorithms. (Heyburn et al., 2018)

In this paper, we carry out an experiment to study the validity of conducting machine learning on synthetic data.

Answer 2

Would it be possible to segregate data by level of maturity/stability of its functionality? The same approach which is used to version the API might work here quite well.

Conceptually it would mean that data for stable functionality could be copied from build to build via snapshots of the underlying data store (maybe with few alterations depending on your context).

Data for preview (unstable, in development) functionality is best to generate - this way no matter how functionality is altered, it is always possible to get the fresh view of how the product is going to look like. Generation can happen (for example) as a pre-demo step, or straight in the app (auto-fill in the UI, it can even be made to look neat).

As soon as preview functionality graduates to stable, its data can be included into the snapshot (as an additional release process step)

It would be best to align with engineering on common way to generate data for new functionality and then ship generators for it’s data along with code itself.

This hybrid approach allows to achieve following benefits:

• Set up and align expectations between devops, engineering and product about how test data is obtained

• No need to deal with exponential growth of time to generate entire data set

• Enable quick iteration of features under development

• Educate developers and make them more DevOps aware (they’d have to take last stable data snapshot and roll it out in their dev env when they start working on a feature)

• Enable fast feedback loop between all the parties involved which leads to better product quality (if snapshot doesn’t rollout or app crashes, then the release process is broken)

There aren’t many cons of this approach, but I’d highlight a few most important:

• It requires work. Main point of this approach is to fix (or actually define) the process, so it would require taking an extra mile to weigh, explain, prototype and build the alignment

• It would not solve problems with depencies in preview functionality, eg if feature B depends on A and they both are under development, data for both have to be generated and A has to be stabilized before B.

Overall, it’s not an easy problem you’re dealing with, and I would be appreciste if you could share your experience on how did it go, what path have your project had taken and reasoning behind it. Cheers!