19

When people talk about running a database in Docker, they do not mean to store the data in a container; they are talking about having a docker image with the DB software, and mounting the data as a volume (a bind volume, not a container volume). Volumes are an essential part in Docker, and are not something that is flakey or just tacked on. Docker is not ...


13

I've had two runs at doing environment variables in a scalable way and neither has ended up perfect because, as I've discovered, is a very tricky thing to get right. I'll give a summary of both of my experiences below: Common Factors Environment variables are stored in a separate repository from the original source code (they are submoduled together but ...


13

Pramod Sadalage and Scott Ambler wrote a book Refactoring Databases: Evolutionary Database Design that is an incredibly solid primer to the subject of DBs in a CD org/team.


11

The Challenges I am aware that there are practices such as only adding database objects, i.e. tables and columns, never modifying or removing them At one company I worked for, a rolling window of raw data equated to about 6 months and ate up 10 TB. The data was then processed into an RDBMS format which cost 6 TB of usable data which accounted for about ...


8

I would integrate the schema management into the application itself (or along with it). Any change to the schema should be committed along the application code (and hence tagged also). There's already a bunch of possibilities listed in this question: What practices or tools enable Continuous Deployment of Databases With this kind of tools, using an in ...


7

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels. Let's start with Prometheus official documentation, it gives a good high-level explanation why: CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase ...


6

I always saw a DBA as someone who fits in between developers and operations when it comes to database management. On the one hand, they often take care of backups, clustering, replication, actual binary installation, file system management, etc. On the other, they also take care of relational DB schema, do performance or query optimization, and advise ...


5

Back when Kubernetes announced the new StatefulSet feature with K8s v1.5 (converting it from the old PetSet name), they put out a really good blog post walking through an example of its usage: https://kubernetes.io/blog/2016/12/statefulset-run-scale-stateful-applications-in-kubernetes/ The first paragraph has a really good description of differentiating ...


5

We use Flyway at work for managing Postgres schemas in the app, and Pillar for managing Cassandra schemas. We have found it best if the app manages its own schema. We had a horrible experience having ansible manage schemas before the apps managed their own schemas.


5

I'd argue a tool alone won't really help unless you shift the schema responsibility to the application team. We do use liquibase or flyway at work, where the application team is responsible to create the changesets. Along with this,you can avoid a purely additive way. Each application is required to be compatible with its precedent version, when an ...


4

I don't know much about Redis/ElasticSearch, but GCS is not really a database-like solution, it is closer to a file storage solution. If you're looking for database-like storage Google Cloud offers: Cloud SQL Cloud Bigtable Cloud Datastore Cloud Spanner If you expect a modest app traffic or alternating high/low traffic it might be more cost-effective (and ...


4

To answer this question in the context of a mainframe environment, and specific to DB2® databases, there are typically 2 commonly used (not cheap ...) alternatives to pick from: Object Administration for DB2®, from BMC. Here are some details about it (quote from the linked page): Making changes to objects in your database—or even just performing routine ...


4

Why not something like Redis? (can use it through ElastiCache). Hard to answer this without knowing the operational requirements of this project. Is this data strictly for logging/research, or is it operational (i.e. this "table" is consulted in order to know which lambda to invoke). Dynamo vs. RDS - depends how spiky your traffic is. Dynamo is very ...


4

We use liquibase at our work and I'll speak highly for it. It's also used by our QA tool QASymphony. We're utilizing it against MSSQL and Oracle databases internally and QASymphony uses/has used it with both postgres + mysql instances.


4

There is plenty of information about this. The basic idea goes like so - Create a script that does the database migration, but make sure that it only adds tables and fields - never removes, renames, or changes the type of existing tables or fields. The new version of the application will start using the new tables/fields when it is an existing field from ...


3

Keep the DB containers separate. The concept of microservices and containers is to allow each part to be independently updated without introducing changes and outages to the rest of the application stack. Containers should also be running a single app per container to allow error detection and log gathering from that application. Each container is simply an ...


3

One of the advantages of using Docker would be that you can easily run tests with different versions of the (DB) apps, which could be quite difficult with Homebrew (some apps don't easily support multiple versions installed on the same system).


3

If your environments are per customer, I would suggest in your specific case to have a repository per customer. (In general it is repository per environment.) This repository would have a standard directory structure for environment variables, ansible variables and inventories, strongly encrypted secrets (account access tokens, private keys, etc.). You ...


3

In our company we manage our application code within a VCS (Git) and most of the applications we work with install their core database from within the setup scripts that ship with the application. If we would have to extend or customize an application for one of our customers which involves having to extend the database or do database customizations, then ...


3

I wrote about this in depth but here's the summary: Preventing split brain (electing more than one master node) needs to be solved. Failure to do so can be catastrophic There are no production ready shared storage solutions to enable databases to be shutdown on one instance and brought up on another without losing all your data.


2

We had the similar requirement as there needs to be a set of data cached in a server for serving the application (fast processing) and also by end of the day, we would need to sync the data from the cache server to the origin server. In the longer run, I would suggest to go for Redis as you can have Redis as an intermediate database (caching) and it will ...


2

For question #1) I think this could perhaps be better answered on SoftwareEngineering SE. Nonetheless, I'll risk an answer: for this kind of information (states) coming from that kind of architecture (distributed), I recommend event sourcing. It removes the complexity and many headaches that come with the nature of distributed systems if someone try to ...


2

Yes, it is possible, but it takes a bit of work. I assume that the concern with duplicating the database is 1) time and 2) size/storage, otherwise it seems like it would be trivial to spawn databases for testing against. Based on the fact that you are targeting PL/SQL i'm guessing you are working with Oracle. You will either need: 1) a smaller-scale dataset ...


2

One of the cons of Homebrew for the use of testing your DB stuff is how tightly coupled the DB is to your local environment. You'll inevitably end up needing to manage the details of what services are running, and possibly needing to manage the versions of multiple installations. This can get complex if you need to install software that needs different ...


2

The DBA is the person/role who is responsible for the availability, integrity and security of the organizations data. The responsibilities are the same regardless of the technology that happens to be in use. The way the tasks are carried out may have changed, but they still need to be done. A modern DBA might not be writing backup scripts and changing tapes ...


2

... [I] can't say what exact app (shop, historic info, social media, forum) This makes this question very difficult to answer. This is because you have two basic types of queries: OLTP and OLAP queries Online transaction processing (OLTP) is information systems that facilitate and manage transaction-oriented applications, typically for data entry and ...


2

The most elegant way would be to use dynamic inventory, rather than static inventory and dynamic group variables. However, if for some reason you don't want to do that, one other option is to use custom facts; these are scripts that live in /etc/ansible/facts.d on the client machines. Like dynamic inventory, they be executable scripts (and therefore, query ...


2

Here's a solution I've used in cases like this - I utilize Ansible to manage Docker containers, and Ansible Vault to store secrets for those containers. Ansible Playbook to run MongoDB container Your playbook.yml may look something like this: - name: run mongodb docker container docker_container: name: mongo-container image: mongo ports: ...


2

I read in Git, Cherry picking is bad. Be careful of such blanket statements. They take on a life of their one, and after a while you have a lot of myths in your team (or workplace) about no cherry-picking, no re-basing, no xyz. These are all just tools; there are times to use them, and times to not use them. Nothing is generally bad. Sometimes , we do ...


2

A variation of method 2 is infinitely scalable: write the data to s3. from s3 trigger lambda to break s3 file into sqs queue events of good size from sqs queue trigger lambdas to insert sqs can be fifo queue if it matters.


Only top voted, non community-wiki answers of a minimum length are eligible