Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
16 votes
Accepted

How to have multiple log streams in docker

I'm still looking for a merging/splitting approach myself, but meanwhile this approach recommended by Kubernetes documentation seems like a sound solution: Use a sidecar container for each of your ...
Rhubarb's user avatar
  • 276
12 votes
Accepted

What is a good logging practice for distributed tasks?

"Serverless" mostly just means you've got relatively simple microservices, generally just a little webapp or a single function that is automatically connected to a REST frontend. The same concepts ...
coderanger's user avatar
  • 1,197
11 votes
Accepted

What exactly is the cloud term 'a Firehose'?

The fire hose is a reference to an old Weird Al Yankovick movie, UHF (1989). In this movie, George Newman (portrayed by Yankovick) starts a local TV channel in the UHF band and is responsible for ...
James Shewey's user avatar
  • 3,734
9 votes
Accepted

Automatic periodic conditional checks

All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...
Tensibai's user avatar
  • 11.4k
8 votes

How to have multiple log streams in docker

The idea of merging them into one stream just to split them later sounds painful. I've not had a reason to do this myself, but here's where I'd start: When you run the docker container, create a ...
bradym's user avatar
  • 556
6 votes

What is a good logging practice for distributed tasks?

This answer is more about scalability considerations - if the number of workers can be high and/or multiple of them can produce logs at high rate at the same time. Yes, using multiple logfiles ...
Dan Cornilescu's user avatar
5 votes
Accepted

Should I use filebeat or an app-logging module with lumberjack capabilities to collect my logs

Both are common and comes with their own pro and cons. It depends if you can accept loosing logs if logstash is dead and your app restart and on the other hand if you can write logs to disk and ...
Tensibai's user avatar
  • 11.4k
5 votes
Accepted

Ideal methods of handling server logs on AWS

I like to use ELK, Elasticsearch, Logstash and Kibana with Beats. Filebeat, that belongs to Beats series will forward the logs that read from a file to logstash. In logstash you could tag, filter, ...
RuBiCK's user avatar
  • 305
5 votes
Accepted

How to get memory usage per process with sar, sysstat?

As Tensibai mentioned, you can extract this info from the /proc filesystem, but in most cases you need to determine the trending yourself. There are several places which could be of interest: /proc/[...
Dan Cornilescu's user avatar
4 votes
Accepted

How to parse command output (docker log) through goaccess?

What do you mean by "it doesn't work"? I got an error suggesting that the log format wasn't correct: Fatal error has occurred Error occured at: src/goaccess.c - set_curses - 1394 No time format was ...
Tom's user avatar
  • 156
4 votes
Accepted

What strategies can be employed to secure sensitive data in log files?

I think the solution comes down to a broad spectrum of approaches that ensures data protection: Data Classification: The most efficient technical strategy is to categorise the data at the point of ...
Richard Slater's user avatar
4 votes

How Do I Log Task Output To A File?

[Converting my comment to an answer] One way to do it would be to write the logs to some external file, and then having a task after it which makes use of failed_when condition, and remove the log ...
Dawny33's user avatar
  • 2,816
4 votes

How Do I Log Task Output To A File?

I think that all you need to do is to register the output of every command you need (store it in a variable) and then simply dump the variable in a file. That way you can review it later. tasks: - ...
13dimitar's user avatar
  • 757
4 votes

How to get memory usage per process with sar, sysstat?

Is it a hard requirement to use only sar and sysstat? If not you might want to look at collectl or collectd. These will enable you to study memory usage over time on a per-process granularity. It's ...
Gaius's user avatar
  • 1,076
3 votes

How Do I Log Task Output To A File?

I solved this by adding ignore_errors: true register: results to the no_log-task. This makes ansible continue to the next task, even when the task fails. Then for the next task define a debug task, ...
Robin Roevens's user avatar
3 votes

Ideal methods of handling server logs on AWS

Completely agree with RuBiCK's answer. I handle logs of 3 servers. We keep only really important logs in the system. Rest are pushed instantly to elasticsearch database. My view is that Using ...
luv.preet's user avatar
  • 223
3 votes
Accepted

Dynamically grep with less/tail

I implemented this in node.js - I don't know of a unix utility that can dynamically change grep. Here is how it works: before dygrep: many-child-procs > logfile.log # in terminal ...
Alexander Mills's user avatar
3 votes
Accepted

What are the best logging/metrics reporting tools for use with OpenShift?

The question hasn't had an answer in seven months so I will promote my upvoted comment to an answer: Finding the lifecycle of a specific message across services is known as "distributed tracing". ...
simbo1905's user avatar
  • 1,172
3 votes
Accepted

How to increase jenkins logger size?

I'm pretty sure this Jenkins system property will do the trick: hudson.util.RingBufferLogHandler.defaultSize tuning Since: 1.563 Default: 256 Description: Number of log entries in loggers available ...
Ian W's user avatar
  • 449
2 votes

Automatic periodic conditional checks

Try Sparky. This is a lightweight but powerful alternative to linux crontab. It comes with nice UI to see cronjob reports and statues. You can also run tasks remotely over ssh or through docker. The ...
Alexey Melezhik's user avatar
2 votes

Automatic periodic conditional checks

Another option is to use Sensu. There are several plugins including a http one, that is able to check whether a response contains a certain string: check-http.rb -u http://my.site.com/health -q 'OK'...
030's user avatar
  • 13.2k
2 votes

Ideal methods of handling server logs on AWS

Also check the content of the log files and ask "is this relevant logging"? Sometimes, complete stack traces are logged. If this is the case, what is causing this and how to suppress it? Some apps ...
030's user avatar
  • 13.2k
2 votes

How Do I Log Task Output To A File?

What I do when I've a command to execute and wish to get the log only in case of failure is as follow (prefixed by a shell comamnd like /bin/sh -c '...' in case the initiator doesn't use a system call ...
Tensibai's user avatar
  • 11.4k
2 votes
Accepted

What considerations should be taken into account for a good log file format?

If you are dealing with application logs, here are the three broad categories of information that a good log file should cover: basic context (timestamp, log level, application name, source file name,...
codeforester's user avatar
2 votes
Accepted

Unable to send cloudwatch logs to loggly using a blueprint lambda - Invalid ciphertext

Follow these steps to fix InvalidCiphertextException: Open the Lambda function in AWS console. Scroll down to the Environment Variables section & click Edit: Delete the value of ...
Harish KM's user avatar
  • 311
2 votes

How to redirect tomcat logs TO console?

There is Tomcat configuration file for the logging: TOMCAT_HOME/conf/logging.properties. Edit the file and remove all the handlers except the console handler: .handlers = java.util.logging....
Dimitar II's user avatar
2 votes
Accepted

".format(name, e)) ... google.protobuf.json_format.ParseError: Failed to parse textPayload field: expected string or bytes-like object." GCF logging

Workaround You can cast all of your exceptions (and errors) to string. Example except Exception as e: logger.log_text(str(e)) raise This is still a strange workaround, I would ...
questionto42's user avatar
1 vote

How to redirect tomcat logs TO console?

That doc say next: When running Tomcat on unixes, the console output is usually redirected to the file named catalina.out. The name is configurable using an environment variable. (See the startup ...
Ivan Gurzhiy's user avatar
1 vote

Monitoring K8s Pods in case of outage

For health monitoring, I would recommend the prometheus-operator. If you are looking for application logs, you should consider a logging stack such as elasticsearch-fluentd-kibana. I personally would ...
Mikhail Advani's user avatar
1 vote

Looking for a good book explaining how things work in the industrial system test environment and production environment

Dude, will depends of what you want to use or follow. Actually cloud + DevOps it's a buzzword in use. So you can start reading about Cloud Architecture, Modern Infrastructure. Refer about one book, ...
Vader's user avatar
  • 466

Only top scored, non community-wiki answers of a minimum length are eligible