6

I'm running Ansible 2.2, but can upgrade if it helps.

I saw this and was pretty excited, but it doesn't seem to be in this (or any) version of Ansible documentation.

The problem I'm trying to solve is I've got 1000 users that I need to manage on a Centos box.

It takes quite a while to run this task serially. And even more annoying, everything shows up as changed because the "expires" command on the user module always marks the thing as changed.

this also looked promising, but it took the same amount of time to run each command in the with_items loop and didn't go any faster (I never bothered to wait long enough to get to the end).

Skipping tasks is fast now (a lot faster than it was in Ansible 2.0), if I can't figure out how to make this work in parallel, I think I'll go back and figure out how to skip pointless tasks and if all else fails, I'll write my own module. But it seems like I should be able to do all this faster in Ansible.


This is what I want to run in parallel, host_authorizations is a list of usernames and other data.

  - name: Create/modify OS user accounts
    user: name={{ item.username }} group=sshusers shell=/bin/bash home="/home/selinux-modules/{{ item.username }}" state=present expires={{item.expiredate|default(omit)}}
    with_items: "{{ host_authorizations }}"
    tags: full_maintenance
  • Please provide a code snippet. Otherwise it is hard to help. – 030 Apr 11 '18 at 18:31
  • @030 there's a snippet, I guess it helps a bit for context. I'm more interested conceptually if there really is a way to run tasks (in a loop) in parallel on the same host. I know I could do a ton of individual things with async, but not so much with with_items. – Peter Turner Apr 11 '18 at 18:45
  • So basically if 1000 users have to be created then it should be finished as fast as creating just one user. Interesting, why not using something like LDAP? – 030 Apr 11 '18 at 20:53
  • 1
    Seriously, you're heading toward a path of pain, I don't think anyone handle more than a dozen of accounts with local account base, as soon as the number of users grow, I assume everyone move to a centralized accounting system, usualy some ldap backend (could be active directory) and then set the expiry time and public key as attributes of this central base then use things like sss_ssh_authorizedkeys to let the ssh server get the authorized keys from this central base. – Tensibai Apr 12 '18 at 11:32
  • 2
    I disagree this is what ansible is for (hint being it doesn't do bulk user creation/management). I stand by the point that accounts should not be managed on local account bases at large volume (fact they are not human accounts is irrelevant to the problem anyway) – Tensibai Apr 12 '18 at 14:33
7

As @webKnja mentioned this is possible with async mode. I have recently discovered it myself and learned that you can use it in 3 different ways depending on your needs.

  1. Execute and poll the results, notice the poll:5, This will poll the results every 5 seconds. You may save some time with this method.

    - name: My long runing task
      some_module_name:
        ip: "{{item.fabric}}"
        username: "{{user}}"
        password: "{{password}}"
        secret: "{{secret}}"
      loop: "{{zoning_list}}"
      register: _alias_vc_0
      async: 60
      poll: 5
    
  2. Fire and forget poll: 0, This is very quick option since Ansible it's just shooting out those tasks. The down side is that we don't know what was the outcome of the task i.e. changed: True/False. Of course it's a downside if you care about the feedback ;).

    name: My long runing task
    some_module_name:
      ip: "{{item.fabric}}"
      username: "{{user}}"
      password: "{{password}}"
      secret: "{{secret}}"
    loop: "{{zoning_list}}"
    register: _alias_vc_0
    async: 60
    poll: 0
    
  3. Fire and forget with async_status, the syntax for the task is the same as example 2 whowever it will require additional task async_status. This is my favorite since it's relatively fast (faster then normal looping or the execute and poll) and allows you to capture the feedback although will need to deal with new register for your async_task.

    retries: 20 -- how many attempts before failing.

    delay: 2 -- how many second to wait between polls.

    - name: My long runing task
      some_module_name:
        ip: "{{item.fabric}}"
        username: "{{user}}"
        password: "{{password}}"
        secret: "{{secret}}"
      loop: "{{zoning_list}}"
      register: _alias_vc_0
      async: 60
      poll: 0
    
    
    - name: Wait for My long running task to finish
      async_status:
        id: "{{ item.ansible_job_id }}"
      register: _jobs_alias_vc_0
      retries: 20
      delay: 2
      until: _jobs_alias_vc_0.finished
      loop: "{{_alias_vc_0.results}}"
    

A word of caution, depending on the task yo may not be able to use the async option. I had examples where I was interacting with system which was not able to handle multiple requests for the same resource. I found async option best working if I have to perform the same task across multiple hosts. That's where I was able to "save" the most time.

Since you posted the link to Ansible documentation in the question I'm not going to do that.

  • @chicks you might want to change the poll value to 0 in example 3. This is an amazing explanation!! Thnx. – Debanjan Basu May 3 at 18:43
  • @DebanjanBasu Anybody can make suggested edits. I might be the one who approves it in the review queues, but you should get credit for the edit itself. – chicks May 3 at 19:37
  • One character edits aren't allowed sadly! :( – Debanjan Basu May 8 at 10:32
4

To answer your question: No, as of now Ansible can't run loops in parallel.

I'd use newusers instead, which is made for bulk user creation. Create a file with all users in it, copy it over to the host, and run newusers /path/to/user/list in a command task.

1

It is possible to achieve this using async mode. Please find some references for how to do this below.

Refs:

---

- name: Run tasks in parallel
  hosts: localhost
  connection: local
  gather_facts: no
  tasks:
    - name: Pretend to create instances
      command: "sleep {{ item }}"  # Instead of calling a long running operation at a cloud provider, we just sleep.
      with_items:
        - 6
        - 8
        - 7
      register: _create_instances
      async: 600  # Maximum runtime in seconds. Adjust as needed.
      poll: 0  # Fire and continue (never poll)

    - name: Wait for creation to finish
      async_status:
        jid: "{{ item.ansible_job_id }}"
      register: _jobs
      until: _jobs.finished
      delay: 5  # Check every 5 seconds. Adjust as you like.
      retries: 10  # Retry up to 10 times. Adjust as needed.
      with_items: "{{ _create_instances.results }}"
  • While those links may answer the question if they come to break there's nothing left in your answer for future readers, please try to showcase how this would help solving the problem with your own words/exemple and leave the links for more detailed informations only. – Tensibai Sep 17 '18 at 7:46
  • yeah, I can't mark this as the answer until A.) I test it and B.) relevant code is placed in here. But thank you for pointing me in this direction nevertheless. – Peter Turner Sep 17 '18 at 13:31
  • Sorry, I was it hurry :) – webKnjaZ Sep 19 '18 at 11:40
-1

Look at Mitogen for Ansible. I assume it would be useful for your case because it accelerate loops execution greatly (it's from my practice).

From site's overview:

Mitogen cannot improve a module once it is executing, it can only ensure the module executes as quickly as possible.

One connection is used per target, in addition to one sudo invocation per user account. This is much better than SSH multiplexing combined with pipelining, as significant state can be maintained in RAM between steps, and system logs aren’t spammed with repeat authentication events.

A single network roundtrip is used to execute a step whose code already exists in RAM on the target. Eliminating multiplexed SSH channel creation saves 4 ms runtime per 1 ms of network latency for every playbook step.

Processes are aggressively reused, avoiding the cost of invoking Python and recompiling imports, saving 300-800 ms for every playbook step.

Code is ephemerally cached in RAM, reducing bandwidth usage by an order of magnitude compared to SSH pipelining, with around 5x fewer frames traversing the network in a typical run.

Fewer writes to the target filesystem occur. In typical configurations, Ansible repeatedly rewrites and extracts ZIP files to multiple temporary directories on the target. Security issues relating to temporary files in cross-account scenarios are entirely avoided.

  • May you extend a little on why and how ? as is the answer is a bit terse and sounds like more a comment. You may check How to Answer for more guidance. – Tensibai Jan 21 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.