1

One goal of DevOps is to create reproducible, production-like environments from source-code repositories. To achieve this, I believe it's necessary to also restore all the data that is needed to setup the environment, so that automated & manual tests can be run there.

For example, let's take a CMS product that provides content (labels, images, rich-text) to a user-facing web application. We want to run integrations tests or explorative UI tests against that web application using production-like data from the CMS. Due to the nature of the product, the data itself can be manipulated at any time by users using the UI of the CMS.

There are two main use cases

  • Developers create example content or add required label translations for new features
  • Content managers publish new content for production, which may include static content like label translations

In a typical setup, there are at least two environments, let's say staging and production. Due to the use cases described above, the CMS databases where the contents are stored will deviate quite quickly between these environments.

Without taking any measures here, after a while, the two databases will be completely out-of-sync and the staging database will not be "production-like" anymore. Additionally, each release of a new feature will require manual steps to add data like label translations into the production database, a step that is neither verified nor tested before the release.

So my question is: What is a good way to bring new "baseline" data from a staging database into a production database without overriding the changes there. And was is a good way to bring back data from production into the staging environment, to provide production-like data for testing purposes. Ideally, all automated and all part of version-control.

Side Note: If we say "different CMS content doesn't change anything about testing", let's think of another use-case where the data behaves even more like configuration. For example, payment or delivery options that are offered on an e-Commerce site.

1 Answer 1

1

it depends a bit what kind of cms you use, most cms offer a way to package data in to packages that you can check in to git and deploy to databases almost like code to webapps.

For example, we work with Sitecore and it lets you use TDS or Unicorn, other big CMS use different tools but mostly the same concept.

The tools let you pick items from your Database and serialise them in to code/text files that are packaged and check in to Git, lets say you have a copy of the database on your Development Workstation, you make some updates to the database and you serialise and package the change with TDS in to a package that you check in to your Development branch.

When you deploy the development branch to your Staging environment the TDS package is applied and makes the same updates to the database of your Staging environments.

6
  • Thanks! I read a bit into both tools for SiteCore and as I understand, developers make changes to their local Sitecore DB and sync the changes to disk, then commit the changes to Git. How would we go about doing this from Production to Staging? e.g. after each Production deployment, run this in an automated pipeline and commit the production changes to a repository that is then deployed to staging?
    – ggradnig
    Sep 26, 2022 at 20:01
  • we only use TDS from lower to higer environment, for the other way have a separate pipeline that we run on demand that will take a backup of the production database(s) and recreate the database of the lower environments (Dev/QA, UAT), we can then apply the TDS packages again from Git if we have new changes that haven't been already on the prod database(s)
    – qBasicBoy
    Sep 26, 2022 at 20:12
  • makes completely sense - thank you for that insight! I suppose we can also generalize this solution for other products: 1) a tool to export changes from a local database to disk and have developers commit the changes to version control 2) a tool to apply changes from version control to a shared database, run by an automated pipeline 3) a tool to backup and restore a production database to a lower tier environment, run on demand or by an automated pipeline
    – ggradnig
    Sep 26, 2022 at 20:28
  • maybe just a last follow-up question: you mentioned "if we have new changes that haven't been already on the prod database"... does this mean that the changes are stored as (historical) differences with timestamps that can be applied onto each other, i.e. after each "sync to disk" a new file is generated, or does the repository only reflect the current state of the database and TDS somehow knows what's new and what's old?
    – ggradnig
    Sep 26, 2022 at 20:31
  • 1
    yeah so in the case of SiteCore and TDS, There always is an ItemID, TDS keeps the itemID in the package, you can also set a "Deploy Once" or "Always Deploy" property in TDS, so if you set it to always deploy it always overwrite whats in the DB with the same ID, if you set it to deploy once it will only create the item if it doesn't already exist. There is also a time stamp so on your development machine it can show you which has been modified last, the item in the TDS (git) package or in the database. PS. Liebe Gruesse nach Wien aus Birmingham :)
    – qBasicBoy
    Sep 26, 2022 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.