People who code: we want your input. Take the Survey
10

The desired_capacity in Terraform is marked in the documentation as optional. So with a proper min_size value, Terraform can wait until the minimum capacity is reached before continuing. The above, with the addition of scale policy can effectively manage capacity without being specific about desired_capacity in your Terraform code. This will prevent you ...


9

While technologically, containers and virtual machines are very different, there is no apparent difference from the perspective of your software. It seems like the argument in your question is that data is special and will always be a unique snowflake, so your question basically boils down to what to do about it in terms of DevOps, CI and Automation. This ...


7

CloudWatch is absolutely the way to go on this front, you can surface Kafka Metrics in CloudWatch either by: Having a separate process that pushes the metrics into CloudWatch. Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch. The process you follow will broadly be broken down into: Publish Custom Metrics. ...


6

I would set the leave_on_terminate option to true. As per the documentation leave_on_terminate If enabled, when the agent receives a TERM signal, it will send a Leave message to the rest of the cluster and gracefully leave. The default behavior for this feature varies based on whether or not the agent is running as a client or a server (prior to ...


5

Autoscaling has nothing to do with NixOS or Docker. This is just the feature of cloud backed applications. However it is true that the best results are achievable using immutable infrastructure, but you do not need neither of these tools to achieve such. Instead you can build your own AMI (using Packer) and then use AWS CodeDeploy to deploy your application ...


4

Disclaimer: this is certainly not the best solution for backend workers, I'm just trying to be creative within the given constraints. First, you'll need to be familiar with SQS's visibility timeout (documentation) for this to work. Let's assume the following, then I'll walk through how it might work: We only have 1 request (henceforth "message") coming in ...


4

If I am understanding you correctly, you have one AMI used in several autoscaling groups and want to run a specific “bootstrap” sequence on it depending on the autoscaling group it belongs to. This is not difficult to accomplish because the launch configuration you created to map the autoscaling group to the AMI can accomodate user data which can be a shell ...


3

Yes, it is certainly possible, but it entirely depends on the architectural design of the entire system. Separating application services into manageable components (i.e. microservices) is one approach to this. While Netflix as a whole is one application in terms of the user experience, it is comprised of many applications/services that provide a small ...


3

Auto Scaling groups have a useful feature for this, named lifecycle hooks. Worflow taken from the documentation above: As you can notice there's a Scale in step, triggering a Terminating:Wait for the autoscaling group and notifying the instance to be terminated, the instance has now to do it's work and once done signal it is ok to be terminated. If the ...


3

The problem can easily be solved using the following components: On the instances serving your webapp continue to monitor the number of incoming requests – and anything else you see fit. Publish the number of incoming requests to a monitoring system. If it is not yet implemented, this step will improve your monitoring capacities, and will help you to ...


3

Here's a rough approach to scaling any stateless app on AWS: Run the app in an Auto Scaling Group (ASG). An ASG makes it easy to manage multiple servers, will automatically replace failed servers, and allows you to automatically scale the number of servers up or down in response to load. To run an ASG, you need to create a Launch Configuration. This is a "...


2

One possible approach is to allow such instances to make demands for new instances based on local request counters, but instead of directly reacting to those demands you would funnel them to a central instance creation logic. That logic would immediately react to the first demand, but also start a "cool off" countdown timer. Any subsequent demand received ...


2

You should look into affinity and anti-affinity as that allows you to control what pods go on which node. You can have it so that with anti-affinity there is only 1 pod of each deployment on a node. This is a bit overkill and wouldn't work for you exactly though IIRC you can have multiple pods on a node just with a limit. https://kubernetes.io/docs/tasks/...


2

What you're going to do is only calling for inconsistencies between your nodes. I wouldn't do any deployment during the instance boot-up, instead install AWS CodeDeploy as part of your boot-up (or bake it into the AMI) and once the instance boots up it contacts the CodeDeploy service and obtains the newest code version. Likewise if you need to roll-out a ...


2

Historically, F5's Application Delivery Controller has been the enterprise-grade industry solution for doing this. With this solution you can: Use iRules LX to communicate with other software to auto-scale on an as-needed basis Load balance at Layer 4 (to achieve better throughput/acceleration) or Layer 7 Share memory between Virtual Servers (there may be a ...


2

My concern is that when scaling down, the ASG may terminate instances that are in the middle of a build. It the builds are running then one should prevent down scaling. When the build has been completed then one could down scale as the resources will not be required anymore.


1

Take a look at the Descheduler. This project runs as a Kubernetes Job that aims at killing pods when it thinks the cluster is unbalanced. The LowNodeUtilization strategy seems to fit your case: This strategy finds nodes that are under utilized and evicts pods, if possible, from other nodes in the hope that recreation of evicted pods will be scheduled on ...


1

Many orm's store migration state in the DB itself, but if you do it 'manually' its not hard to build yourself either. Just have a "migrationstate" table where all executed migrations are stored. That way when a node comes online it can simply check that table against its local "migrations" folder, and execute what's missing - this should cause only your ...


1

How are you doing your deploys? Nuke + rebuild the ASG (whether one node at a time, or by replacing the whole group at once), or do you have a script that redeploys all of your active nodes? And also, how are you triggering your deploys? Ideally you want to be running/triggering deploys from some form of CI server like Bamboo or Jenkins. If you're doing ...


1

Use AWS Automation document on state change to modify autoscaling group to 0 and return to 1 after. It should terminate and recreate.


1

Very good question; I'm interested to see what comes up here. In terms of cost, obviously each company and team is different. Ideally, everyone should be resource and cost conscious... but most people really don't seem to penny pinch by heavily monitoring and tuning their workloads (probably as that takes a certain skill level and tool set). My Thoughts ...


1

There is a range of common scenarios when you want to use private subnets to be used in an auto scaling group: Your traffic is terminated by reaches your infrastructure on a Elastic Load Balancers and your web server instances are behind the load balancer. This reduces the attack surface of the web servers. ( I think I started to use "terminate" since ...


1

Resources: TargetGroup: ...


1

I am assuming that the backend application has been set up in its own subnet that has been configured for private IP addresses only. If not I would recommend that you do that as a starter for 10. With that in place, you could set up an internal application load balancer to work on private only and have your backend ASG use this. from the AWS docs https://...


1

Yes. And there are tools out there covering this piece of functionality. Probably the most popular one is kubernetes (see also kubernetes): Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications. Depending on the provider for your deployment environment the actual orchestration tool/...


1

If the repo is on AWS Codecommit, you should be able to access it via a role setup for each type of instance. See under IAM role here https://docs.aws.amazon.com/codecommit/latest/userguide/auth-and-access-control.html


1

Another option is to use saltstack's product: salt This Configuration Management System (CMS) can protect sensitive data in three ways. 1) Pillars; these are securely stored and encrypted key:value or key:list dictionaries stored on your primary computer(s) that you manage the entire cluster from. This will allow you to create salt scripts that can have ...


1

Option 1: You could place your private repo in the same VPC as the instance and then pull code from there. Just make sure the repo cannot be accessed from outside the VPC. Option 2: Ssh into the new instance and SCP the binary over. Option 3: Use docker and deploy a private container registry (e.g. Amazon ECR). Build the image on your machine / build ...


1

Recently Kubernetes started preview work on https://cluster-api.sigs.k8s.io which is meant to be the standard way to solve the way how High Availability clusters are deployed. The kubeadm tool has also been updated with features to deploy multiple masters since this question was first asked, which help tremendously. With the exception of deploying cloud-...


1

I think you can do this by Nodeselector. Firstly add a label for node selector in your daemonset config. Then label your nodes with the attached label. Now if you can set the autoscaling thresholds, it will be deployed on that node automatically on nodes that matches the label. Maybe you can tweak somehow to attach a label to your node when it is added to ...


Only top voted, non community-wiki answers of a minimum length are eligible