10

Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.

With the rise of data science there is a new type of artefact - monolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can be many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, such blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.

What are best practices to address this new challenge using DevOps methods - if even possible?

Improve this question
4
  • 3
    I don't see the difference between a large blob and an uberjar in java context. Same practices applies, the size of an artefact has few reasons to come into play.
    – Tensibai
    Commented Sep 13, 2017 at 12:24
  • Hi - I thought with uber-jars from 2gb upwards you would tell them the microservice architecture story, or?.. But model blobs just start there, 8gb will be soon not rare.
    – Ta Mu
    Commented Sep 13, 2017 at 13:39
  • 1
    I just mean a dB snapshot of 350Go is not a different asset than a 5Mo jar, it has to be stored somewhere anyway and artifact repository can handle that
    – Tensibai
    Commented Sep 13, 2017 at 14:53
  • I agree - just because the resulting program is large doesn't mean it isn't still compiled, versioned and stored like everything else (albeit with perhaps a few storage challenges), so I fail to see how this "brings organizations back to the pre-CI age" If an organization thinks that, I'm not sure that they actually understand DevOps/CI.
    – James Shewey
    Commented Sep 13, 2017 at 15:40

2 Answers 2

Reset to default
9

Personally I don't see any reason for which an Artefact Repository - the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts.

The artefact size might have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.

As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:

  • are produced from some sort of specification which supports change versioning (equivalent to software source code)
  • are built via a repeatable and automatable process
  • are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)

Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.

Improve this answer
9
  • Perfect answer. I store all my heavy models in git lfs and pull them when necessary [serverless paradigm] :)
    – Dawny33
    Commented Sep 14, 2017 at 2:41
  • @Dawny33 but would you now consider moving away from git lfs?
    – Ta Mu
    Commented Sep 14, 2017 at 7:38
  • @J.Doe So far so good with lfs. Would probably move if I find a really good better alternative.
    – Dawny33
    Commented Sep 14, 2017 at 8:14
  • then I do not get why you say the answer about is "perfect" while it suggests using an artefact repository?! @Dawny33
    – Ta Mu
    Commented Sep 14, 2017 at 8:16
  • 2
    DVC can be considered as a better alternative to git-lfs
    – Shcheklein
    Commented Jun 20, 2018 at 21:06
4

I would recommend taking a look at DVC - an open source version control system for data science projects.

One of the basic things that it perfectly handles is managing data files (along with code) - inputs, outputs (models), intermediate results. Semantically it's similar to git-lfs but unlike git-lfs it is capable of managing files like 100GB and what is more important it does not rely on proprietary storage/format. It's completely open-source and is compatible with any network storage as a server to keep data files - S3, GCP cloud storage, SSH, FTP, etc.

Improve this answer
1
  • DVC has some nice features, but I don't believe Git LFS uses proprietary storage - it's an open source project.
    – RichVel
    Commented Apr 9, 2021 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.