Questions tagged [logging]
Systematic recording of events from services and application is a key process within DevOps adoption. Use this tag to describe logging and telemetry questions.
41
questions
-2
votes
0
answers
32
views
Suggestions on a possible simplification of an infrastructure
I have a somewhat complex infrastructure situation to manage. I have 3 logging source systems (standalone applications logging on linux machines, frontend applications logging on Appinsights and ...
- 1
0
votes
0
answers
9
views
containerd to graylog
I was using /etc/docker/daemon.json to easily configure logs from containers to be sent via UDP to a remote Graylog server and it worked fine.
It's easily achievable with the following config in ...
0
votes
0
answers
10
views
What is the easiest way to admin a server with multiple nodes?
One issue as a sysadmin that I'm facing is the logs of 1 day are in 1 server, and the logs of another day are in other server. It's absolute hassle to figure out where the logs of particular day went ...
- 101
0
votes
0
answers
29
views
sending logs to splunk between docker containers
I have an app that runs in a docker container (say A). The app includes a log4j2 yaml configuration file, which has an Http_Appender routing logs to http://host.docker.internal:8080/log ready to be ...
- 1
0
votes
1
answer
43
views
How can I log every command for which an Ansible playbook uses elevated permissions?
I have an Ansible playbook that runs against a Linux target host and uses become to gain elevated permissions for those actions that require it.
For auditing purposes, I need a list of all the Unix ...
- 123
3
votes
1
answer
300
views
Keeping Docker logs only in memory
There are various log drivers available for Docker but it looks like all of them persist the logs to disk.
To reduce disk usage I would like to keep the logs only in memory (and limited to a certain ...
- 131
1
vote
0
answers
79
views
Can Loki accept/combine a log stream (unique set of labels) from multiple Promtail instances running simultaneously? (applies to Prometheus too)
I am using ec2_sd_configs in Loki (and Prometheus) to discover my EC2 instances, most of which are autoscaled, and therefore come and go. They are identical load balanced API instances, so I don't ...
- 130
0
votes
1
answer
120
views
Does GitLab support searching logs across jobs for specific strings?
I need to find all jobs that failed in a specific fashion, a few methods are viable
One method that would be viable is finding where GitLab stores its log files, and searching them with something ...
- 1,794
0
votes
0
answers
19
views
Print application log in an external directory from Golang application deployed in Kubernetes cluster
I am new to containerization. I have a Golang application deployed in a Kubernetes cluster inside an Ubuntu VM. I can access the application log using k logs -f <pod-name> command. However, I ...
3
votes
1
answer
856
views
Azure Function App logs without Application Insights instance?
I'm a bit confused about my options for Function App logging on Azure. Is Application Insights the only mechanism available to observe the output logged through the function application's context? In ...
- 131
1
vote
1
answer
751
views
".format(name, e)) ... google.protobuf.json_format.ParseError: Failed to parse textPayload field: expected string or bytes-like object." GCF logging
I changed the logging in a Google Cloud Function from the built-in Python module logging to google.cloud.logging and I cannot seem to export exceptions and error messages since they have the wrong ...
- 143
3
votes
1
answer
3k
views
What is the difference between Fluent Bit and Prometheus?
From what I've seen, Prometheus scrapes data and metrics and displays them and Fluent Bit collects logs and metrics and can output them.
So, are they doing the same thing, i.e. should I choose one ...
- 33
0
votes
0
answers
149
views
Why Jenkins Blue Ocean with inheritance Job does not log ? Yet logfile is created
I'm willing to switch to blue ocean however I can't get the pipeline logs to show in inheritance "classic" jobs (https://plugins.jenkins.io/project-inheritance/) :
I see that freestyle jobs ...
- 101
1
vote
1
answer
145
views
How can I get logs in real time from a docker container hosted on Digital Ocean
I have a docker container in which I am running Flask and Scrapy.
I noticed that when I run and build my container on my local machine I get logs in real time. However, on an Ubuntu instance hosted on ...
- 121
6
votes
0
answers
481
views
How do I find out exactly when a Github personal access token was last used?
A few days ago, I got an email from github saying that a personal access token that we used is in an 'outdated format':
We noticed that an application, TeamCity(ci.support.example.com), owned by an ...
0
votes
1
answer
718
views
How to deploy loki-stack helm chart with CRIO?
I am trying to configure loki-stack using a helm chart to my k8s cluster.
Since my k8s cluster uses CRIO and not docker (which is common nowdays) I need to change the pipeline_stages: config map to ...
- 151
2
votes
1
answer
1k
views
How to increase jenkins logger size?
I use Jenkins log recorders as described in docs:
https://www.jenkins.io/doc/book/system-administration/viewing-logs/
It works, but log is only about 255 lines long, which is too short for my ...
- 23
3
votes
1
answer
1k
views
Logging & Monitoring on multiple servers
I lead a small project with 2 groups of servers (servers in a single group are clones with the same microservices, like pods in the k8s), a balance loader, and a DB. Docker & k8s aren't used ...
- 181
1
vote
1
answer
163
views
Unable to send cloudwatch logs to loggly using a blueprint lambda - Invalid ciphertext
I am trying to use the loggly Lambda blueprint to send cloudwatch logs to loggly. I am doing this because I have a bunch of lambdas that I need to have a single point of viewing logs. I've followed ...
- 465
1
vote
1
answer
44
views
Monitoring K8s Pods in case of outage
K8s has self-healing builtin features, if a pod crash then it will be replaced with the new pod but its log is gone. So what is the best way to monitor K8s pods in case of outage?
- 213
0
votes
1
answer
21
views
Looking for a good book explaining how things work in the industrial system test environment and production environment
Any advice on a good book explaining how things work in the industrial system test environment and production environment?
It's my first time in this topic, so I would need a general overview.
Key ...
- 109
1
vote
1
answer
26
views
What software ihandles shipping to ElasticSearch the standard output of a .NET Core web application running in a container?
I have a ballpark idea of how the ELK stack works. I read that by default, standard output is captured and that output will make its way to the Elasticsearch repository. If I post something to stdout, ...
- 111
1
vote
1
answer
945
views
How to parse command output (docker log) through goaccess?
How to parse docker logs output / tail output through goaccess? When I do:
docker container logs mycontainer > logs
and then goaccess -f logs it works perfectly, but if I do docker logs --tail ...
- 113
1
vote
0
answers
149
views
Fluentd log shipment. Index assigning [closed]
I'm using Fluentd for shipping two types of logs to Elasticsearch cluster (application and syslog). And I'd like to assign different indices to them.
Here is my try but it doesn't work. Could anybody ...
- 11
0
votes
1
answer
427
views
Problem with logging in docker when using fluentd for logging of three flask apps running in a docker network behind an nginx
I am facing a strange problem with logging when I set up a network of three docker containers each running a small flask app.
My python application looks as follows:
from flask import Flask, request
...
- 99
2
votes
0
answers
249
views
How to write output per builder into separate log files in Packer? [closed]
I have a few packer templates which have several builders - up to 5 per template. This makes it very easy to provision all the resources necessary in a pipeline -- however by default packer writes all ...
- 3,570
1
vote
1
answer
60
views
What considerations should be taken into account for a good log file format?
We have a practical question I struggle to answer :
What is a good logfile schema?
Here are the caveats:
There are several workloads, each writing more than one log file
Applications are on more ...
- 3,570
0
votes
1
answer
31
views
View logrotated log continuously
Is there a command to view logrotated logs as a continuous stream with proper ordering?
I need to view all entries in syslog in historical order starting from the oldest entry available. So far I was ...
- 143
0
votes
1
answer
165
views
azure activity log, how to find releases
My security department has asked to ingest logs from Azure with the goal being to identity releases. The security team wants to be able to say:
Hey, there was a release on mm/dd/yyyy at noon (...
- 101
4
votes
2
answers
6k
views
How to redirect tomcat logs TO console?
So I have a problem that is the reverse of what everyone else on the internet wants.. I want to take tomcat logs that are being written out to /usr/local/tomcat/logs and have them be output to console ...
- 161
0
votes
1
answer
138
views
Dynamically grep with less/tail [closed]
when tailing a log file, we might use:
tail -f foo.log | grep bar
if we want to change the grep argument, we have to ctrl-c and then change the argument and then restart the command.
my question is ...
- 395
1
vote
1
answer
105
views
What are the best logging/metrics reporting tools for use with OpenShift?
OpenShift includes EFK and I'm looking into using that or ELK, but there are so many options (found a link with 50+ options) that I'm trying to narrow down. Ideally something free/cheap. What is the ...
- 163
3
votes
1
answer
597
views
Should I use filebeat or an app-logging module with lumberjack capabilities to collect my logs
I have a pretty basic elk setup which aggregates logs from around 15 VMs that are running various services.
All of them have configured filebeat syslog forwarder with a single input in logstash.
I ...
- 133
2
votes
3
answers
179
views
Automatic periodic conditional checks
I need a simple tool that will carry out some conditional checks periodically and give me a red/green status that I can use as an indication that something is wrong.
These are examples of checks that ...
- 171
4
votes
3
answers
119
views
Ideal methods of handling server logs on AWS
We currently have some ruby applications hosted on EC2 servers on AWS, and every few weeks we see the disk space get completely consumed by the app log as well as the worker logs.
What are the best ...
- 143
10
votes
1
answer
693
views
What exactly is the cloud term 'a Firehose'?
I found a Firehose definition from Overview of the Loggregator System Cloud Foundry docs.
The Firehose is a WebSocket endpoint that streams all the event data
coming from a Cloud Foundry ...
- 452
12
votes
7
answers
66k
views
How Do I Log Task Output To A File?
One of my ansible tasks imports an Oracle database using impdp.
This generates a lot of output to the console so I have set no_log: True.
However, when this fails I want to see the log!
How can I ...
- 491
3
votes
2
answers
13k
views
How to get memory usage per process with sar, sysstat?
Can I get memory usage per process with Linux? we monitor our servers with sysstat/sar. But besides seeing that memory went off the roof at some point, we can't pinpoint which process was getting ...
- 151
7
votes
2
answers
743
views
What strategies can be employed to secure sensitive data in log files?
Working in highly regulated environments data is classified in different ways depending on the sensitivity. In some cases, this is legally enforced and must be treated differently.
Examples of a data ...
- 11.6k
28
votes
2
answers
14k
views
How to have multiple log streams in docker
We have an application that writes three types of logs into three separate files: access logs, generic application logs and system logs. The format (and purpose) of these logs are very different. And ...
- 1,577
15
votes
2
answers
398
views
What is a good logging practice for distributed tasks?
I have the following setting:
Create multiple workers, do a computation and terminate them after the
computation is done.
So, every-time it’ll be a different instance running the task, so each ...
- 2,826