0

I have been working on deployment of a GitLab container with Ansible.

At one point the whole thing broke with a terribly unspecific error message. After that point the container kept rebooting about once a minute.

The docker logs <container> command gave me the following output:

Thank you for using GitLab Docker Image!
Current version: gitlab-ce=12.7.2-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab vim /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

Cleaning stale PIDs & sockets
Preparing services...
Starting services...
Configuring GitLab...
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
Starting Chef Client, version 14.14.29
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - postgresql (0.1.0)
  - registry (0.1.0)
  - mattermost (0.1.0)
  - consul (0.1.0)
  - gitaly (0.1.0)
  - gitlab (0.0.1)
  - letsencrypt (0.1.0)
  - runit (4.3.0)
  - crond (0.1.0)
  - redis (0.1.0)
  - nginx (0.1.0)
  - acme (4.0.0)
  - package (0.1.0)
  - monitoring (0.1.0)
  - praefect (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...

================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
================================================================================

Psych::SyntaxError
------------------
(<unknown>): did not find expected key while parsing a block mapping at line 2 column 5

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:105:in `block in from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:29:in `auto_vivify'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:105:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:105:in `block in from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:29:in `auto_vivify'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:105:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:22:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:

 98:
 99:    def from_file(_file_path)
100:      # Throw errors for unrecognized top level calls (usually spelling mistakes)
101:      config_strict_mode true
102:      # Turn on node deprecation messages
103:      Gitlab::Deprecations::NodeAttribute.log_deprecations = true
104:      # Allow auto mash creation during from_file call
105>>     Gitlab::ConfigMash.auto_vivify { super }
106:    ensure
107:      config_strict_mode false
108:      Gitlab::Deprecations::NodeAttribute.log_deprecations = false
109:    end
110:
111:    # Enhance set so strict mode errors aren't thrown as long as the setting is witin our defined config
112:    def internal_set(symbol, value)
113:      if configuration.key?(symbol)
114:        configuration[symbol] = value

System Info:
------------
chef_version=14.14.29
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

(<unknown>): did not find expected key while parsing a block mapping at line 2 column 5

Running handlers complete
Chef Client failed. 0 resources updated in 01 seconds
1

While someone with knowledge about the innards of GitLab may have known what the error pointed to, it took me some time and bisecting my revision history to find the root cause.

I had introduced a Jinja2 conditional like this in the inlined YAML for LDAP servers:

gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below
main: # 'main' is the GitLab 'provider ID' of this LDAP server
  somekey: somevalue
  {% if some_variable is defined %}
  key: value
  {% endif %}
EOS

The # remember to close this block with 'EOS' below remark isn't the only caveat you have to worry about here. Indentation is too, as I found out.

Provided the condition became true, the about would have evaluated to:

gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below
main: # 'main' is the GitLab 'provider ID' of this LDAP server
  somekey: somevalue
    key: value
  EOS

... where the additional indentation tripped up the YAML parser and caused the unspecific error symptoms (and message).

Hope this helps someone else to avoid a lengthy analysis of the cause, if they run into a similar condition.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.