Using Chef for multi-node operations

Question

I have an application I would like to configure using Chef that spans multiple nodes. Let's say that the process of doing it consists of

1. Do a thing on node A capturing the output
2. Do another thing on node B with that output
3. Back to node A now for some operation
4. Now on node B again
5. ...

One way to do this would be to write a recipe that stored the stage it was on on each node and just keep running it on both nodes repeatedly until eventually it had a chance to do all operations on all nodes. But this is clumsy and won't scale if there's node C, D, etc. I obv know about notify to sequence dependencies within a single node, but that won't work across multiple nodes. I can't be the only person who needs this, so is there a mechanism or a design pattern/best practice/TTP for this style of activity?

Answer 1

In short, Chef may not be the tool to use here.

Chef is a convergent model, so you have to write your recipes in an idempotent way such as after some runs it will be at the desired state according to the others nodes around.

Your approach about storing the state sounds the way to go, and you'll have to use an external orchestrator to schedule the chef-client runs alternatively on A and B.
If you have a chef-server, push jobs could be the way to go sticking in chef ecosystem, main advantage I see with it compared to others orchestrators is a pull model not needing an incoming administrative port nor ssh on the target nodes.

Answer 2

This is a textbook example of server orchestration and is something that Chef inherently is not intended to do. As noted by Tensibai, a server running Chef is a convergent system that achieves its own desired state based upon configuration settings set by recipes, attributes, data bags, etc. Without getting in to specific details about your infrastructure, a few approaches that you might be able to take are:

Create independent idempotent operations

As you stated in your question, creating a state of operation where your nodes could run repeatedly until all tasks have completed does not scale well. It may be possible however to redesign your nodes so that it doesn't matter. If nodes a and b run tasks to output their logs in parallel, and b completes before a, it could run the task that node a would normally run and vice-versa.

Use an external orchestrator for delegation

Using a delegator node will definitely scale much better if you intend to have many nodes to orchestrate. However, this could create conflicts with your chef client runs on the nodes being managed by the delegator. It would be very difficult to verify that your node configurations and the delegator node tasks do not conflict with each other. A clever way to manage this could be to incorporate the tasks in the configuration of each node and have the delegator set a value in a data bag or and attribute of the server to signal how it should configure itself (i.e. what tasks it needs to perform).

Combine your infrastructure

If each node runs it tasks serially depending upon the other nodes, and you have no cost/technical dependencies on running tasks on different nodes, You may want to consider combining your node configurations into one single node. This would eliminate any configuration conflicts you would have between any of your nodes. I imagine there are clear intentions for running your tasks on different nodes, but this is definitely an option to consider (maybe even at the cost of time to rewrite tasks for different nodes).

Answer 3

Use SaltStack to orchestrate your chef runs.

Your cross-node orchestration logic goes here https://docs.saltstack.com/en/latest/topics/orchestrate/orchestrate_runner.html#more-complex-orchestration

And you can declaratively https://docs.saltstack.com/en/latest/ref/states/all/salt.states.chef.html

or imperatively https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.chef.html

drive Chef nodes to their desired state.