Skip to main content
DevOps

Return to Question

typo/grammar
Source Link
edit approved Feb 17, 2021 at 10:14
RichVel
edit approved Feb 17, 2021 at 10:14
RichVel
  • 902
  • 6
  • 16

Data Science pipelines and monoliticmonolithic model blobs

Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.

With the rise of data science there is a new type of artefact - monoliticmonolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can have a sizebe many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, theysuch blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.

What are best practices to address this new challenge using DevOps methods - if even possible?

Data Science pipelines and monolitic model blobs

Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.

With the rise of data science there is a new type of artefact - monolitic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can have a size many GB and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, they have their version and associated collections of training data (corpora) which tend to grow rapidly as well.

What are best practices to address this new challenge using DevOps methods - if even possible?

Data Science pipelines and monolithic model blobs

Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.

With the rise of data science there is a new type of artefact - monolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can be many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, such blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.

What are best practices to address this new challenge using DevOps methods - if even possible?

Source Link
Ta Mu
Ta Mu
  • 6.8k
  • 5
  • 42
  • 83

Data Science pipelines and monolitic model blobs

Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.

With the rise of data science there is a new type of artefact - monolitic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can have a size many GB and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, they have their version and associated collections of training data (corpora) which tend to grow rapidly as well.

What are best practices to address this new challenge using DevOps methods - if even possible?