Traverse nested dictionary with keeping the keys in ansible

Question

I'm not able to loop a list in a dict and keep the key of the original dict.

Bellow an example of my data

vars:
 nginx_users:
    'instance1':
       roles:
          - product
          - nginx_instance
       cert_source: internal-ca
       URL: page-nginx.domain.internal
       port: 8090
       downtime: true
       basic_auth:
          - username1
          - username2
       ip_restriction:
          - '0.0.0.0/0'
    'instance2':
       roles:
          - product
          - nginx_instance
       cert_source: internal-ca
       URL: page2-nginx.domain.internal
       port: 8091
       downtime: true
       basic_auth:
          - username1
          - username2
       ip_restriction:
          - '0.0.0.0/0'
    'instance3':
       roles:
          - product
          - nginx_instance

I need to loop it in a manner that It iterates like this

"msg": [
        "instance1",
        [
            "username1"
        ],
        "instance1",
        [
            "username2"
        ],
       "instance2",
        [
            "username1"
        ],
        "instance2",
        [
            "username2"
        ]
...
    ]

I am using

when: item.value.basic_auth is defined

In a loop which will filter out the items without the desired value/item and return only the "items" that I need to parse, like.

...
loop: "{{ nginx_users | dict2items }}"
when: item.value.basic_auth is defined
...

However from there I don't know how to loop those "items" to get the list of users on a way that it returns the single user per iteration and preserve the key.

I know a work around would be to have use another variable instead of the key so I could just select it. but that wouldn't work for me in this escenario.

I have tried using subelements and lookup but I don't seem to be able narrow down the exact combination also thought about using product too and try and traverse the data. But I wasn't able to use that either.

After reading this:

https://github.com/ansible/ansible/pull/6370

https://github.com/ansible/ansible/pull/4982

I'm starting to think I will have to use Jinja to be able to do this.

This is as close as I have been able to get to what I'm trying to achieve

- name: loop
  debug:
    msg:
    - "{{ item.key }}"
    - "{{ item.value.basic_auth }}"
  loop: "{{ nginx_users | dict2items }}"
  when: item.value.basic_auth is defined

- name: loop
  debug:
    msg:
    - "{{ item.1 }}"
  loop: "{{ lookup('subelements', nginx_users, 'basic_auth', {'skip_missing': True}) }}"

which results on this:

 TASK [loop]
 ************************************************************************ skipping: [nginx-test-01.domain.internal] => (item={'key':
 u'instance1', 'value': {u'roles': [u'product', u'nginx_instance']}})
 ok: [nginx-test-01.domain.internal] => (item={'key': u'username2',
 'value': {u'roles': [u'product', u'nginx_instance'], u'URL':
 u'page-nginx.symplectic.internal', u'ip_restriction': [u'0.0.0.0/0'],
 u'basic_auth': [u'username1', u'username2'], u'port': 8090,
 u'zerodowntime': True, u'cert_source': u'internal-ca'}}) => {
     "msg": [
         "instance1",
         [
             "username1",
             "username2"
         ]
     ]}

 TASK [loop]
 ************************************************************************ ok: [nginx-test-01.domain.internal] => (item=[{u'roles':
 [u'product', u'nginx_instance'], u'URL':
 u'page-nginx.domain.internal', u'ip_restriction': [u'0.0.0.0/0'],
 u'port': 8090, u'downtime': True, u'cert_source':
 u'internal-ca'}, u'username1']) => {
     "msg": [
         "username1"
     ] } ok: [nginx-test-01.domain.internal] => (item=[{u'roles': [u'product', u'nginx_instance'], u'URL':
 u'page-nginx.symplectic.internal', u'ip_restriction': [u'0.0.0.0/0'],
 u'port': 8091, u'downtime': True, u'cert_source':
 u'internal-ca'}, u'username2']) => {
     "msg": [
         "username2"
     ] }

Any thought/suggestions would be appreciated.

UPDATE: I have learned that using loop control inner with and outer variable I can achieve this. But this requires me to use a separated .yml Is there any way to do this on the same .yml file?

Answer 1

Let's simplify the dictionary in the first task and loop the subelements in the second. The tasks below

    - set_fact:
        nginx_users_selected: "{{ nginx_users_selected|
                                  default({})|
                                  combine({item.key: item.value})
                                  }}"
      loop: "{{ nginx_users|
                dict2items|
                json_query('[*].{key: key, value: value.basic_auth}')
                }}"
      when: item.value
    - debug:
        msg: "{{ item.0.key }} {{ item.1 }}"
      loop: "{{ nginx_users_selected|
                dict2items|
                subelements('value')
                }}"

give

    "msg": "instance2 username1"
    "msg": "instance2 username2"
    "msg": "instance1 username1"
    "msg": "instance1 username2"

Then we can create the structure

    - set_fact:
        my_list: "{{ my_list|default([]) +
                     [ item.0.key, [ item.1 ]] }}"
      loop: "{{ nginx_users_selected|
                dict2items|
                subelements('value') }}"
    - debug:
        var: my_list

give

    "my_list": [
        "instance2", 
        [
            "username1"
        ], 
        "instance2", 
        [
            "username2"
        ], 
        "instance1", 
        [
            "username1"
        ], 
        "instance1", 
        [
            "username2"
        ]
    ]