16
votes
Accepted
How to have multiple log streams in docker
I'm still looking for a merging/splitting approach myself, but meanwhile this approach recommended by Kubernetes documentation seems like a sound solution: Use a sidecar container for each of your ...
- 276
12
votes
Accepted
What is a good logging practice for distributed tasks?
"Serverless" mostly just means you've got relatively simple microservices, generally just a little webapp or a single function that is automatically connected to a REST frontend. The same concepts ...
- 1,197
11
votes
Accepted
What exactly is the cloud term 'a Firehose'?
The fire hose is a reference to an old Weird Al Yankovick movie, UHF (1989). In this movie, George Newman (portrayed by Yankovick) starts a local TV channel in the UHF band and is responsible for ...
- 3,742
9
votes
Accepted
Automatic periodic conditional checks
All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...
- 11.4k
8
votes
How to have multiple log streams in docker
The idea of merging them into one stream just to split them later sounds painful. I've not had a reason to do this myself, but here's where I'd start:
When you run the docker container, create a ...
- 556
6
votes
What is a good logging practice for distributed tasks?
This answer is more about scalability considerations - if the number of workers can be high and/or multiple of them can produce logs at high rate at the same time.
Yes, using multiple logfiles ...
- 6,750
5
votes
Accepted
Should I use filebeat or an app-logging module with lumberjack capabilities to collect my logs
Both are common and comes with their own pro and cons.
It depends if you can accept loosing logs if logstash is dead and your app restart and on the other hand if you can write logs to disk and ...
- 11.4k
5
votes
Accepted
Ideal methods of handling server logs on AWS
I like to use ELK, Elasticsearch, Logstash and Kibana with Beats.
Filebeat, that belongs to Beats series will forward the logs that read from a file to logstash.
In logstash you could tag, filter, ...
- 305
5
votes
Accepted
How to get memory usage per process with sar, sysstat?
As Tensibai mentioned, you can extract this info from the /proc filesystem, but in most cases you need to determine the trending yourself. There are several places which could be of interest:
/proc/[...
- 6,750
4
votes
Accepted
What strategies can be employed to secure sensitive data in log files?
I think the solution comes down to a broad spectrum of approaches that ensures data protection:
Data Classification: The most efficient technical strategy is to categorise the data at the point of ...
- 11.7k
4
votes
How Do I Log Task Output To A File?
[Converting my comment to an answer]
One way to do it would be to write the logs to some external file, and then having a task after it which makes use of failed_when condition, and remove the log ...
- 2,816
4
votes
How Do I Log Task Output To A File?
I think that all you need to do is to register the output of every command you need (store it in a variable) and then simply dump the variable in a file. That way you can review it later.
tasks:
- ...
- 757
4
votes
How to get memory usage per process with sar, sysstat?
Is it a hard requirement to use only sar and sysstat? If not you might want to look at collectl or collectd. These will enable you to study memory usage over time on a per-process granularity. It's ...
- 1,086
4
votes
Accepted
How to parse command output (docker log) through goaccess?
What do you mean by "it doesn't work"? I got an error suggesting that the log format wasn't correct:
Fatal error has occurred
Error occured at: src/goaccess.c - set_curses - 1394
No time format was ...
- 156
3
votes
How to redirect tomcat logs TO console?
There is Tomcat configuration file for the logging: TOMCAT_HOME/conf/logging.properties.
Edit the file and remove all the handlers except the console handler:
.handlers = java.util.logging....
- 131
3
votes
How Do I Log Task Output To A File?
I solved this by adding
ignore_errors: true
register: results
to the no_log-task. This makes ansible continue to the next task, even when the task fails. Then for the next task define a debug task, ...
3
votes
Ideal methods of handling server logs on AWS
Completely agree with RuBiCK's answer.
I handle logs of 3 servers. We keep only really important logs in the system. Rest are pushed instantly to elasticsearch database.
My view is that Using ...
- 223
3
votes
Accepted
Dynamically grep with less/tail
I implemented this in node.js - I don't know of a unix utility that can dynamically change grep. Here is how it works:
before dygrep:
many-child-procs > logfile.log # in terminal ...
- 395
3
votes
Accepted
What are the best logging/metrics reporting tools for use with OpenShift?
The question hasn't had an answer in seven months so I will promote my upvoted comment to an answer:
Finding the lifecycle of a specific message across services is known as "distributed tracing". ...
- 1,172
3
votes
Accepted
How to increase jenkins logger size?
I'm pretty sure this Jenkins system property will do the trick:
hudson.util.RingBufferLogHandler.defaultSize
tuning
Since: 1.563
Default: 256
Description: Number of log entries in loggers available ...
- 449
2
votes
Automatic periodic conditional checks
Try Sparky. This is a lightweight but powerful alternative to linux crontab. It comes with nice UI to see cronjob reports and statues. You can also run tasks remotely over ssh or through docker. The ...
- 149
2
votes
Automatic periodic conditional checks
Another option is to use Sensu. There are several plugins including a http one, that is able to check whether a response contains a certain string:
check-http.rb -u http://my.site.com/health -q 'OK'...
- 13.3k
2
votes
Ideal methods of handling server logs on AWS
Also check the content of the log files and ask "is this relevant logging"? Sometimes, complete stack traces are logged. If this is the case, what is causing this and how to suppress it? Some apps ...
- 13.3k
2
votes
How Do I Log Task Output To A File?
What I do when I've a command to execute and wish to get the log only in case of failure is as follow (prefixed by a shell comamnd like /bin/sh -c '...' in case the initiator doesn't use a system call ...
- 11.4k
2
votes
Accepted
What considerations should be taken into account for a good log file format?
If you are dealing with application logs, here are the three broad categories of information that a good log file should cover:
basic context (timestamp, log level, application name, source file name,...
- 391
2
votes
Accepted
Unable to send cloudwatch logs to loggly using a blueprint lambda - Invalid ciphertext
Follow these steps to fix InvalidCiphertextException:
Open the Lambda function in AWS console.
Scroll down to the Environment Variables section & click Edit:
Delete the value of ...
- 311
2
votes
Accepted
What is the difference between Fluent Bit and Prometheus?
This is a little off. You are comparing a "Time Series Database" to a "Log Shipper".
How Prometheus Works
Any service/entity prometheus wants to run monitor has to provide an &...
- 1,550
2
votes
Accepted
".format(name, e)) ... google.protobuf.json_format.ParseError: Failed to parse textPayload field: expected string or bytes-like object." GCF logging
Workaround
You can cast all of your exceptions (and errors) to string.
Example
except Exception as e:
logger.log_text(str(e))
raise
This is still a strange workaround, I would ...
- 143
1
vote
How to redirect tomcat logs TO console?
That doc say next:
When running Tomcat on unixes, the console output is usually redirected to the file named catalina.out. The name is configurable using an environment variable. (See the startup ...
- 121
1
vote
Monitoring K8s Pods in case of outage
For health monitoring, I would recommend the prometheus-operator. If you are looking for application logs, you should consider a logging stack such as elasticsearch-fluentd-kibana. I personally would ...
- 246
Only top scored, non community-wiki answers of a minimum length are eligible