With typical continuous integration environments, you configure an environment capable to execute compilation and test batches (agent, slave..) coordinated by a scheduler (master, server..)

But what if your "client" environment is a Graphics Processing Unit (GPU) cluster used to perform model trainings in different configurations? Is there any difference or would you just for example let the head cluster node incorporate a Jenkins slave? (or Bamboo agent etc)

  • You have a GPU cluster at the moment?
    – 030
    Sep 14, 2017 at 7:52
  • please elaborate your question better - I tend to answer yes
    – Ta Mu
    Sep 14, 2017 at 8:00
  • 1
    But what if your "client" environment is a GPU cluster Is it a hypothesis or is there a real GPU cluster?
    – 030
    Sep 14, 2017 at 8:20
  • real, indeed it is
    – Ta Mu
    Sep 14, 2017 at 8:25
  • 2
    But what exactly is the problem? It's just a resource, it shouldn't matter if it's a GPU cluster or a self-driving car you run tests on. You just have to schedule it's use. Sep 14, 2017 at 8:56

1 Answer 1


Since you're talking about CI/CD I presume you have the possibility to automate the model trainings in those configurations. Let's call the scripts able to do that train_model_config_A, train_model_config_B, etc.

Then you could have a wrapper script which checks an environment variable used to select which client environment you desire and invokes the corresponding train_model_config_<blah> script. Ideally translating the outcome of the training (whatever that is) into one or more results of the pass/fail type. Then such wrapper script can be integrated in a CI/CD pipeline like a custom test step/stage (or even a build one, if it produces any artifacts you might want to archive). Just like any test executed on a testbed incorporating some non-generic piece of hardware. In other words the CPU cluster makes no real difference.

You might not need to install the slave directly on the GPU cluster if the train_model_config_<blah> scripts (and thus the wrapper script as well) can be executed on some other hosts and remotely control the GPUs - you can then have the slave on some other host, leaving the GPU cluster free to only do its stuff.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.