Skip to main content
Commonmark migration
Source Link

This is a textbook example of server [orchestration][1]orchestration and is something that Chef inherently is not intended to do. As noted by Tensibai, a server running Chef is a convergent system that achieves its own desired state based upon configuration settings set by recipes, attributes, data bags, etc. Without getting in to specific details about your infrastructure, a few approaches that you might be able to take are:

Create independent idempotent operations

As you stated in your question, creating a state of operation where your nodes could run repeatedly until all tasks have completed does not scale well. It may be possible however to redesign your nodes so that it doesn't matter. If nodes a and b run tasks to output their logs in parallel, and b completes before a, it could run the task that node a would normally run and vice-versa.

Use an external orchestrator for delegation

Using a delegator node will definitely scale much better if you intend to have many nodes to orchestrate. However, this could create conflicts with your chef client runs on the nodes being managed by the delegator. It would be very difficult to verify that your node configurations and the delegator node tasks do not conflict with each other. A clever way to manage this could be to incorporate the tasks in the configuration of each node and have the delegator set a value in a data bag or and attribute of the server to signal how it should configure itself (i.e. what tasks it needs to perform).

Combine your infrastructure

If each node runs it tasks serially depending upon the other nodes, and you have no cost/technical dependencies on running tasks on different nodes, You may want to consider combining your node configurations into one single node. This would eliminate any configuration conflicts you would have between any of your nodes. I imagine there are clear intentions for running your tasks on different nodes, but this is definitely an option to consider (maybe even at the cost of time to rewrite tasks for different nodes). [1]: https://en.wikipedia.org/wiki/Orchestration_(computing)

This is a textbook example of server [orchestration][1] and is something that Chef inherently is not intended to do. As noted by Tensibai, a server running Chef is a convergent system that achieves its own desired state based upon configuration settings set by recipes, attributes, data bags, etc. Without getting in to specific details about your infrastructure, a few approaches that you might be able to take are:

Create independent idempotent operations

As you stated in your question, creating a state of operation where your nodes could run repeatedly until all tasks have completed does not scale well. It may be possible however to redesign your nodes so that it doesn't matter. If nodes a and b run tasks to output their logs in parallel, and b completes before a, it could run the task that node a would normally run and vice-versa.

Use an external orchestrator for delegation

Using a delegator node will definitely scale much better if you intend to have many nodes to orchestrate. However, this could create conflicts with your chef client runs on the nodes being managed by the delegator. It would be very difficult to verify that your node configurations and the delegator node tasks do not conflict with each other. A clever way to manage this could be to incorporate the tasks in the configuration of each node and have the delegator set a value in a data bag or and attribute of the server to signal how it should configure itself (i.e. what tasks it needs to perform).

Combine your infrastructure

If each node runs it tasks serially depending upon the other nodes, and you have no cost/technical dependencies on running tasks on different nodes, You may want to consider combining your node configurations into one single node. This would eliminate any configuration conflicts you would have between any of your nodes. I imagine there are clear intentions for running your tasks on different nodes, but this is definitely an option to consider (maybe even at the cost of time to rewrite tasks for different nodes). [1]: https://en.wikipedia.org/wiki/Orchestration_(computing)

This is a textbook example of server orchestration and is something that Chef inherently is not intended to do. As noted by Tensibai, a server running Chef is a convergent system that achieves its own desired state based upon configuration settings set by recipes, attributes, data bags, etc. Without getting in to specific details about your infrastructure, a few approaches that you might be able to take are:

Create independent idempotent operations

As you stated in your question, creating a state of operation where your nodes could run repeatedly until all tasks have completed does not scale well. It may be possible however to redesign your nodes so that it doesn't matter. If nodes a and b run tasks to output their logs in parallel, and b completes before a, it could run the task that node a would normally run and vice-versa.

Use an external orchestrator for delegation

Using a delegator node will definitely scale much better if you intend to have many nodes to orchestrate. However, this could create conflicts with your chef client runs on the nodes being managed by the delegator. It would be very difficult to verify that your node configurations and the delegator node tasks do not conflict with each other. A clever way to manage this could be to incorporate the tasks in the configuration of each node and have the delegator set a value in a data bag or and attribute of the server to signal how it should configure itself (i.e. what tasks it needs to perform).

Combine your infrastructure

If each node runs it tasks serially depending upon the other nodes, and you have no cost/technical dependencies on running tasks on different nodes, You may want to consider combining your node configurations into one single node. This would eliminate any configuration conflicts you would have between any of your nodes. I imagine there are clear intentions for running your tasks on different nodes, but this is definitely an option to consider (maybe even at the cost of time to rewrite tasks for different nodes).

Source Link

This is a textbook example of server [orchestration][1] and is something that Chef inherently is not intended to do. As noted by Tensibai, a server running Chef is a convergent system that achieves its own desired state based upon configuration settings set by recipes, attributes, data bags, etc. Without getting in to specific details about your infrastructure, a few approaches that you might be able to take are:

Create independent idempotent operations

As you stated in your question, creating a state of operation where your nodes could run repeatedly until all tasks have completed does not scale well. It may be possible however to redesign your nodes so that it doesn't matter. If nodes a and b run tasks to output their logs in parallel, and b completes before a, it could run the task that node a would normally run and vice-versa.

Use an external orchestrator for delegation

Using a delegator node will definitely scale much better if you intend to have many nodes to orchestrate. However, this could create conflicts with your chef client runs on the nodes being managed by the delegator. It would be very difficult to verify that your node configurations and the delegator node tasks do not conflict with each other. A clever way to manage this could be to incorporate the tasks in the configuration of each node and have the delegator set a value in a data bag or and attribute of the server to signal how it should configure itself (i.e. what tasks it needs to perform).

Combine your infrastructure

If each node runs it tasks serially depending upon the other nodes, and you have no cost/technical dependencies on running tasks on different nodes, You may want to consider combining your node configurations into one single node. This would eliminate any configuration conflicts you would have between any of your nodes. I imagine there are clear intentions for running your tasks on different nodes, but this is definitely an option to consider (maybe even at the cost of time to rewrite tasks for different nodes). [1]: https://en.wikipedia.org/wiki/Orchestration_(computing)