I have a fresh sensu/uchiwa stack running on a virtual machine. It is a "standalone" installation - redis, rabbitmq, sensu-server, sensu-api, sensu-client, and uchiwa are all installed on the same machine. I have a single check which the client.json subscribes to.
In short, something appears to be wrong. I logged into the uchiwa dashboard, and there is a warning message that "Datacenter sensu-81 returned: 500 INTERNAL SERVER ERROR".
The sensu-client
, sensu-api
and sensu-server
logs are filled with these messages and only these messages:
==> /var/log/sensu/sensu-api.log <==
{"timestamp":"2017-05-11T21:00:34.758243+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:34.758784+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
==> /var/log/sensu/sensu-client.log <==
{"timestamp":"2017-05-11T21:00:35.973060+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:35.974858+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
==> /var/log/sensu/sensu-server.log <==
{"timestamp":"2017-05-11T21:00:37.489540+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:37.489704+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
It suggests that there is an error connecting to the transport layer, so I checked that my transport.json was configured as expected, and it was (the sensu default):
{
"transport": {
"name": "rabbitmq",
"reconnect_on_error": true
}
}
So I checked sensu's rabbitmq.json to check that it was configured as expected, which again it was. (Note: I have a line in my hosts file which translates monitor to 127.0.0.1. I am aware of the issue with using "localhost" causing problems with IPv6, so I double checked that the hosts file maps "monitor" to 127.0.0.1). To further confirm that these credentials are working, I turned on the rabbitmq web-based management and logged in successfully using those credentials.
{
"rabbitmq": {
"ssl": {
"cert_chain_file": "/etc/sensu/ssl/cert.pem",
"private_key_file": "/etc/sensu/ssl/key.pem"
},
"host": "monitor",
"port": 5671,
"vhost": "/sensu",
"user": "sensu",
"password": "sensu"
}
}
I also checked that the configuration of rabbitmq itself was as expected (ssl turned on and such):
[
{rabbit, [
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/cert.pem"},
{keyfile,"/etc/rabbitmq/ssl/key.pem"},
{verify,verify_peer},
{fail_if_no_peer_cert,true}]}
]}
].
If I can provide any more details let me know. I'm at a loss where to continue debugging.