8

Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.

With the rise of data science there is a new type of artefact - monolitic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can have a size many GB and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, they have their version and associated collections of training data (corpora) which tend to grow rapidly as well.

What are best practices to address this new challenge using DevOps methods - if even possible?

  • 2
    I don't see the difference between a large blob and an uberjar in java context. Same practices applies, the size of an artefact has few reasons to come into play. – Tensibai Sep 13 '17 at 12:24
  • Hi - I thought with uber-jars from 2gb upwards you would tell them the microservice architecture story, or?.. But model blobs just start there, 8gb will be soon not rare. – Peter Sep 13 '17 at 13:39
  • 1
    I just mean a dB snapshot of 350Go is not a different asset than a 5Mo jar, it has to be stored somewhere anyway and artifact repository can handle that – Tensibai Sep 13 '17 at 14:53
  • I agree - just because the resulting program is large doesn't mean it isn't still compiled, versioned and stored like everything else (albeit with perhaps a few storage challenges), so I fail to see how this "brings organizations back to the pre-CI age" If an organization thinks that, I'm not sure that they actually understand DevOps/CI. – James Shewey Sep 13 '17 at 15:40
8

Personally I don't see any reason for which an Artefact Repository - the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts.

The artefact size might have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.

As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:

  • are produced from some sort of specification which supports change versioning (equivalent to software source code)
  • are built via a repeatable and automatable process
  • are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)

Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.

  • Perfect answer. I store all my heavy models in git lfs and pull them when necessary [serverless paradigm] :) – Dawny33 Sep 14 '17 at 2:41
  • @Dawny33 but would you now consider moving away from git lfs? – Peter Sep 14 '17 at 7:38
  • @J.Doe So far so good with lfs. Would probably move if I find a really good better alternative. – Dawny33 Sep 14 '17 at 8:14
  • then I do not get why you say the answer about is "perfect" while it suggests using an artefact repository?! @Dawny33 – Peter Sep 14 '17 at 8:16
  • 2
    DVC can be considered as a better alternative to git-lfs – Shcheklein Jun 20 '18 at 21:06
4

I would recommend taking a look at DVC - an open source version control system for data science projects.

One of the basic things that it perfectly handles is managing data files (along with code) - inputs, outputs (models), intermediate results. Semantically it's similar to git-lfs but unlike git-lfs it is capable of managing files like 100GB and what is more important it does not rely on proprietary storage/format. It's completely open-source and is compatible with any network storage as a server to keep data files - S3, GCP cloud storage, SSH, FTP, etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.