Skip to main content
1 of 5
fildred13
  • 313
  • 1
  • 9

Sensu TCP Connection Issue

I have a fresh sensu/uchiwa stack running on a virtual machine. It is a "standalone" installation - redis, rabbitmq, sensu-server, sensu-api, sensu-client, and uchiwa are all installed on the same machine. I have a single check which the client.json subscribes to.

In short, something appears to be wrong. I logged into the uchiwa dashboard, and there is a warning message that "Datacenter sensu-81 returned: 500 INTERNAL SERVER ERROR".

The sensu-client, sensu-api and sensu-server logs are filled with these messages and only these messages:

==> /var/log/sensu/sensu-api.log <==
{"timestamp":"2017-05-11T21:00:34.758243+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:34.758784+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

==> /var/log/sensu/sensu-client.log <==
{"timestamp":"2017-05-11T21:00:35.973060+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:35.974858+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

==> /var/log/sensu/sensu-server.log <==
{"timestamp":"2017-05-11T21:00:37.489540+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:37.489704+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

It suggests that there is an error connecting to the transport layer, so I checked that my transport.json was configured as expected, and it was (the sensu default):

{
  "transport": {
    "name": "rabbitmq",
    "reconnect_on_error": true
  }
}

So I checked sensu's rabbitmq.json to check that it was configured as expected, which again it was. (Note: I have a line in my hosts file which translates monitor to 127.0.0.1. I am aware of the issue with using "localhost" causing problems with IPv6, so I double checked that the hosts file maps "monitor" to 127.0.0.1). To further confirm that these credentials are working, I turned on the rabbitmq web-based management and logged in successfully using those credentials.

{
  "rabbitmq": {
        "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
        "host": "monitor",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "sensu"
  }
}

I also checked that the configuration of rabbitmq itself was as expected (ssl turned on and such):

[
    {rabbit, [
        {ssl_listeners, [5671]},
    {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                   {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                   {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                   {verify,verify_peer},
                   {fail_if_no_peer_cert,true}]}
      ]}
].

If I can provide any more details let me know. I'm at a loss where to continue debugging.

fildred13
  • 313
  • 1
  • 9