7

I have autoscaling groups that I want to dynamically scale up/down based on the amount of lag showing in certain Kafka topics. There's a 1:1 relationship between topic and autoscaling group that needs to be scaled. I'm fairly sure I can't use CloudWatch metrics to approach this.

Is there a way I could accomplish this using Jenkins without creating race conditions and other issues?

Note: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.

  • Maybe you can explain a little more why you think cloudwatch is not a good approach ? – Tensibai Apr 4 '17 at 12:53
  • My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics. – danielhklein Apr 4 '17 at 14:05
7

CloudWatch is absolutely the way to go on this front, you can surface Kafka Metrics in CloudWatch either by:

  1. Having a separate process that pushes the metrics into CloudWatch.
  2. Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch.

The process you follow will broadly be broken down into:

  1. Publish Custom Metrics.
  2. Create an AutoScaling Launch Configuration.
  3. Create an AutoScaling Group and Policies based upon your Custom Metric.

Depending on your cost model I would also consider how you can use spot instances to keep the costs down, i.e. if you have non-critical but high compute workloads use Spot Instances as part of your autoscaling configuration.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.