Questions tagged [logging]

Systematic recording of events from services and application is a key process within DevOps adoption. Use this tag to describe logging and telemetry questions.

Filter by
Sorted by
Tagged with
0
votes
0answers
2 views

Azure Data Factory and Salesforce Marketing Cloud

I'm attempting to set up the configuration found at this guide: https://docs.microsoft.com/en-us/azure/data-factory/connector-salesforce-marketing-cloud Things seem to have gone smoothly, except that ...
0
votes
1answer
19 views

How can I get logs in real time from a docker container hosted on Digital Ocean

I have a docker container in which I am running Flask and Scrapy. I noticed that when I run and build my container on my local machine I get logs in real time. However, on an Ubuntu instance hosted on ...
1
vote
0answers
17 views

How do I find out exactly when a Github personal access token was last used?

A few days ago, I got an email from github saying that a personal access token that we used is in an 'outdated format': We noticed that an application, TeamCity(ci.support.example.com), owned by an ...
0
votes
0answers
5 views

How to trace logs in pipelines?

I wanted to get some advice on how to trace log documents being processed in various stages. e.g. file->logstash->kafka->logstash -> elasticsearch The information I want to get is whether ...
0
votes
0answers
17 views

fluentd logs all processing logs in fluent logs

I've noticed that in my configuration where fluent logs in his own logs all processed message. So any message is processed at least twice. My configuration looks like this <source> @type ...
0
votes
1answer
77 views

How to deploy loki-stack helm chart with CRIO?

I am trying to configure loki-stack using a helm chart to my k8s cluster. Since my k8s cluster uses CRIO and not docker (which is common nowdays) I need to change the pipeline_stages: config map to ...
-1
votes
0answers
3 views

Is there a way to filter missing events in the sequence of events in Loggly?

We'd like to monitor the cron jobs through Loggly. Our app sends a log for all successful attempts of the cron job. We'd like to build a dashboard so that only missing cron jobs alone to be listed in ...
1
vote
1answer
150 views

How to increase jenkins logger size?

I use Jenkins log recorders as described in docs: https://www.jenkins.io/doc/book/system-administration/viewing-logs/ It works, but log is only about 255 lines long, which is too short for my ...
3
votes
1answer
371 views

Logging & Monitoring on multiple servers

I lead a small project with 2 groups of servers (servers in a single group are clones with the same microservices, like pods in the k8s), a balance loader, and a DB. Docker & k8s aren't used ...
10
votes
7answers
58k views

How Do I Log Task Output To A File?

One of my ansible tasks imports an Oracle database using impdp. This generates a lot of output to the console so I have set no_log: True. However, when this fails I want to see the log! How can I ...
1
vote
1answer
77 views

Unable to send cloudwatch logs to loggly using a blueprint lambda - Invalid ciphertext

I am trying to use the loggly Lambda blueprint to send cloudwatch logs to loggly. I am doing this because I have a bunch of lambdas that I need to have a single point of viewing logs. I've followed ...
1
vote
1answer
21 views

Monitoring K8s Pods in case of outage

K8s has self-healing builtin features, if a pod crash then it will be replaced with the new pod but its log is gone. So what is the best way to monitor K8s pods in case of outage?
1
vote
0answers
56 views

Kubernetes pod eviction and logging

EDIT: I found this Github issue and I'm waiting for their response: https://github.com/kubernetes/kubernetes/issues/26923#issuecomment-608581984. I have been tasked with improving our cluster-level ...
0
votes
1answer
17 views

Looking for a good book explaining how things work in the industrial system test environment and production environment

Any advice on a good book explaining how things work in the industrial system test environment and production environment? It's my first time in this topic, so I would need a general overview. Key ...
14
votes
2answers
374 views

What is a good logging practice for distributed tasks?

I have the following setting: Create multiple workers, do a computation and terminate them after the computation is done. So, every-time it’ll be a different instance running the task, so each ...
1
vote
0answers
129 views

Fluentd log shipment. Index assigning [closed]

I'm using Fluentd for shipping two types of logs to Elasticsearch cluster (application and syslog). And I'd like to assign different indices to them. Here is my try but it doesn't work. Could anybody ...
1
vote
0answers
101 views

How to rite output per builder into separate log files in Packer? [closed]

I have a few packer templates which have several builders - up to 5 per template. This makes it very easy to provision all the resources necessary in a pipeline -- however by default packer writes all ...
3
votes
2answers
3k views

How to redirect tomcat logs TO console?

So I have a problem that is the reverse of what everyone else on the internet wants.. I want to take tomcat logs that are being written out to /usr/local/tomcat/logs and have them be output to console ...
1
vote
1answer
322 views

How to parse command output (docker log) through goaccess?

How to parse docker logs output / tail output through goaccess? When I do: docker container logs mycontainer > logs and then goaccess -f logs it works perfectly, but if I do docker logs --tail ...
1
vote
1answer
21 views

What software ihandles shipping to ElasticSearch the standard output of a .NET Core web application running in a container?

I have a ballpark idea of how the ELK stack works. I read that by default, standard output is captured and that output will make its way to the Elasticsearch repository. If I post something to stdout, ...
0
votes
1answer
259 views

Problem with logging in docker when using fluentd for logging of three flask apps running in a docker network behind an nginx

I am facing a strange problem with logging when I set up a network of three docker containers each running a small flask app. My python application looks as follows: from flask import Flask, request ...
1
vote
1answer
50 views

What considerations should be taken into account for a good log file format?

We have a practical question I struggle to answer : What is a good logfile schema? Here are the caveats: There are several workloads, each writing more than one log file Applications are on more ...
0
votes
1answer
27 views

View logrotated log continuously

Is there a command to view logrotated logs as a continuous stream with proper ordering? I need to view all entries in syslog in historical order starting from the oldest entry available. So far I was ...
0
votes
1answer
100 views

azure activity log, how to find releases

My security department has asked to ingest logs from Azure with the goal being to identity releases. The security team wants to be able to say: Hey, there was a release on mm/dd/yyyy at noon (...
1
vote
1answer
87 views

What are the best logging/metrics reporting tools for use with OpenShift?

OpenShift includes EFK and I'm looking into using that or ELK, but there are so many options (found a link with 50+ options) that I'm trying to narrow down. Ideally something free/cheap. What is the ...
2
votes
3answers
165 views

Automatic periodic conditional checks

I need a simple tool that will carry out some conditional checks periodically and give me a red/green status that I can use as an indication that something is wrong. These are examples of checks that ...
0
votes
1answer
82 views

Dynamically grep with less/tail [closed]

when tailing a log file, we might use: tail -f foo.log | grep bar if we want to change the grep argument, we have to ctrl-c and then change the argument and then restart the command. my question is ...
10
votes
1answer
513 views

What exactly is the cloud term 'a Firehose'?

I found a Firehose definition from Overview of the Loggregator System Cloud Foundry docs. The Firehose is a WebSocket endpoint that streams all the event data coming from a Cloud Foundry ...
3
votes
1answer
548 views

Should I use filebeat or an app-logging module with lumberjack capabilities to collect my logs

I have a pretty basic elk setup which aggregates logs from around 15 VMs that are running various services. All of them have configured filebeat syslog forwarder with a single input in logstash. I ...
3
votes
2answers
11k views

How to get memory usage per process with sar, sysstat?

Can I get memory usage per process with Linux? we monitor our servers with sysstat/sar. But besides seeing that memory went off the roof at some point, we can't pinpoint which process was getting ...
4
votes
3answers
95 views

Ideal methods of handling server logs on AWS

We currently have some ruby applications hosted on EC2 servers on AWS, and every few weeks we see the disk space get completely consumed by the app log as well as the worker logs. What are the best ...
26
votes
2answers
11k views

How to have multiple log streams in docker

We have an application that writes three types of logs into three separate files: access logs, generic application logs and system logs. The format (and purpose) of these logs are very different. And ...
7
votes
2answers
611 views

What strategies can be employed to secure sensitive data in log files?

Working in highly regulated environments data is classified in different ways depending on the sensitivity. In some cases, this is legally enforced and must be treated differently. Examples of a data ...