Sensu TCP Connection Issue

Question

I have a fresh sensu/uchiwa stack running on a virtual machine. It is a "standalone" installation - redis, rabbitmq, sensu-server, sensu-api, sensu-client, and uchiwa are all installed on the same machine. I have a single check which the client.json subscribes to.

In short, something appears to be wrong. I logged into the uchiwa dashboard, and there is a warning message that "Datacenter sensu-81 returned: 500 INTERNAL SERVER ERROR".

The sensu-client, sensu-api and sensu-server logs are filled with these messages and only these messages:

==> /var/log/sensu/sensu-api.log <==
{"timestamp":"2017-05-11T21:00:34.758243+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:34.758784+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

==> /var/log/sensu/sensu-client.log <==
{"timestamp":"2017-05-11T21:00:35.973060+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:35.974858+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

==> /var/log/sensu/sensu-server.log <==
{"timestamp":"2017-05-11T21:00:37.489540+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:37.489704+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

It suggests that there is an error connecting to the transport layer, so I checked that my transport.json was configured as expected, and it was (the sensu default):

{
  "transport": {
    "name": "rabbitmq",
    "reconnect_on_error": true
  }
}

So I checked sensu's rabbitmq.json to check that it was configured as expected, which again it was. (Note: I have a line in my hosts file which translates monitor to 127.0.0.1. I am aware of the issue with using "localhost" causing problems with IPv6, so I double checked that the hosts file maps "monitor" to 127.0.0.1). To further confirm that these credentials are working, I turned on the rabbitmq web-based management and logged in successfully using those credentials.

{
  "rabbitmq": {
        "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
        "host": "monitor",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "sensu"
  }
}

I also checked that the configuration of rabbitmq itself was as expected (ssl turned on and such):

[
    {rabbit, [
        {ssl_listeners, [5671]},
    {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                   {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                   {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                   {verify,verify_peer},
                   {fail_if_no_peer_cert,true}]}
      ]}
].

I'm at a loss where to continue debugging.

I also checked the api.json and uchiwa.json config files, just to make sure they had credentials which matched, which they do:

api.json:

{
  "api": {
    "host": "monitor",
    "port": 4567,
    "user": "admin",
    "password": "secret"
  }
}

uchiwa.json:

{
 "sensu": [
   {
       "name": "",
       "host": "monitor",
       "ssl": false,
       "port": 4567,
       "user": "admin",
       "pass": "secret",
       "path": "",
       "timeout": 5000
   }
 ],
 "uchiwa": {
   "users": [
    {
        "password": "admin",
        "username": "admin"
    }
],
   "port": 3000,
   "refresh": 5
  }
}

Attempting to talk to the sensu api directly from inside the box gives a 500 error also:

vagrant@vagrant-ubuntu-trusty-64:/etc/default$ curl -I http://localhost:4567/stashes -u admin
Enter host password for user 'admin':
HTTP/1.1 500 Internal Server Error
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Origin: *
Connection: close
Content-length: 0

Answer 1

Ends up the rabbitmq logs held the answer. I saw that something was wrong, but the messages were so cryptic I couldn't tell what was actually breaking. Turns out it was an SSL issue caused by an old version of Erlang, which is what apt was installing by default on ubuntu 14.04 (erlang version R16B03).

This amqp issue was what pointed me to the solution:

https://github.com/squaremo/amqp.node/issues/224

I needed to upgrade to Erlang >= 17.5, then it worked as expected.