22

This is actually a great example of people slightly abusing spot. People are saying 'Our workload is really important but we don't want to pay full on demand price', so they set a bid price higher than on-demand on the assumption that it is very unlikely to be terminated, but still want to get the 'cheapest possible' spot price on offer. There have been ...


15

You are looking for Dynamic Inventory Please look into ec2.py and ec2.ini of Ansible, and also the Dynamic Inventory docs of Ansible are very good for understanding how to manage your AWS inventory.


15

Spotted an open source tool called autospotting that just might help with this: Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand ...


12

There are 2 reasons for this: A lot of users are using the spot instance at some times (Think about batch processing, boot up 100 machines as spot instance and crunch away). For a spot instance you don't pay the bidding price, you pay the current spot price. The bidding price is the cutoff point. If the current spot price exceeds that of the bidding price ...


12

Amazon AWS just released (on 13th February 2017) a new feature that allows to change a size of an EBS volume. source: https://aws.amazon.com/blogs/aws/amazon-ebs-update-new-elastic-volumes-change-everything/ This allows to increase the size of an EBS volume on an existing instance, while it is running. It is important to note that changing the volume size,...


12

Largely inspired by the blog posts and gist already linked in the other answers, here is my take to the problem. I did use some convoluted JMESpath functions to get a list of snapshots and not require tr. Disclaimer: Use at your own risks, I did my best to avoid any problem and keep sane defaults, but I won't take any blame if it cause problem to you. #!/...


8

If you want something quick to get this sorted without much more knowledge I'd recommend elastic beanstalk. It's another AWS application which will handle the load balancer configuration and instance scaling for you. There's no extra cost on top of the load balancer and instances so you can keep using t2 type instances but let elastic beanstalk scale as ...


6

Snapshot Permissions Boto3 has a function that allows you to create volume permissions, which is what AMI Sharing with AWS Marketplace requires you to do. snapshot.modify_attribute will allow you to share your AMI with the marketplace account like so (you can also use a JSON representation if you prefer, it's in the docs): response = snapshot....


6

A useful bit of information for understanding why someone would bid over the on-demand price can be found in the Introduction to Spot Instances: Spot Instances can be used to help you meet occasional needs for large amounts of compute capacity (note that the default limit for Spot Instances is 100 versus the default limit of 20 for On-Demand Instances....


6

I would personally consider a model like this: Timed Lambdas -> Checks spot price -> Push to ElastiCache Then when you need instances: Timed lambdas -> Pulls spot price from ElastiCache, sets it as environment variable on your Machine where you spin up IaC from -> This is parsed as argument to IaC code and pushes out the spot price You could ...


6

So what you have actually is this: As your API ELB is in a private zone it can't be accessed from the internet. Your frontend in React.js just run in User's browser and not on the UI servers, those server just serve static files. You have two options, configure your frontend servers to redirect API calls to the API ELB or just update the API ELB to be ...


6

You notice that you now have access to 6 availability zones in us-east-1, where you previously only had 4 or 5. Adding capacity is the reason that new availability zone is now available. Because of the way the spot market works, there is no way to know what the highest winning bid was, at any given time... because the current market price is based on the ...


6

you create a Docker Swarm stack file: --- version: '3.1' services: ubuntu: image: ubuntu # or your custom Docker image deploy: replicas: 10000 Then, with docker stack you can run your 10000 Ubuntu's on one or - probably better in this case - a set of Swarm hosts. This could be bare metal or AWS. Enjoy! For MAC address is very ...


6

A good way to find comparison information about things like this is googling for "X vs Y", e.g. "Chef vs AWS stacks", "Chef vs Puppet" or something like that. That does turn out subjective information, and while it is nothing like having hands-on experience, you still get a few nice nuggets here or there. For example, Chef gives you the full Ruby language ...


6

To run ipconfig from the AWS Systems Manager Run Command: $ aws ssm send-command --document-name "AWS-RunPowerShellScript" --instance-ids "<your instance id>" --parameters commands=ipconfig Note: If you've got the error, consider specifying the right --region. This assumes you have your AWS credentials and CLI configured properly. See Systems ...


6

For a given instance, you would first use aws ec2 describe-instances to get the information JSON for your instance. The information also contains the keypair name used to create that instance. E.g. for an instance i-0e2x8xd7xxx (Note: I use the awesome tool jq to do JSON parsing but you can use any other solution) aws ec2 describe-instances --instance-...


5

How about user-data ? I believe adding the "#cloud-boothook" allow to force the user-data to run at every restart. #cloud-boothook #!/bin/bash echo 'test' > /home/ec2-user/user-script-output.txt If so, you could fix your sudo scripts hopefully... or install / add AWS run commands configuration https://aws.amazon.com/blogs/aws/new-ec2-run-command-remote-...


5

I used the following script on GitHub by Rodrigue Koffi (bonclay7) and it works pretty good. https://github.com/bonclay7/aws-amicleaner Command: amicleaner --check-orphans From the documentation blog post it does some more things: It actually does a bit more than that, at of today it allows: Removing a list of images and associated snapshots ...


5

You might not be able to get this done on the free tier. Puppet for example isn't going to want to start because of the RAM limitations. The AWS free tier uses thje t2.micro instance which only has 1GB of RAM. Your operating system alone probably needs 512 to run at idle. This leaves you a mere 512 MB or ram for all of the things that you listed. While the ...


5

The article you are referencing mentions encrypted RAM but that was only a reference to possibilities of enhancing the technology more in the future, not about the current state of confidential computing. Confidential computing means, that when data is in use these things are prevented: Malicious insiders with administrative privilege or direct access ...


4

Let me give you a tool-agnostic way of trying to do this. every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc We faced the same problem in the infrastructure which we're building. So, we ...


4

Generally recovering from this sort of mistake involves editing the boot loader to force a root shell. Since you don't have physical access to the (non-physical) machine, things are a bit different. If this is an EBS boot drive, you can try detaching it, fixing it on another machine, then reattaching. This is a good opportunity to learn why it's useful to ...


4

Essentially yes. From what I've observed when there is limited capacity in that availability zone the spot price jumps to 10x the on demand price. Be sure to use as many AZs as possible (6 currently in us-east-1) when using spot instances to provide better guarantees you can launch an instance.


4

I like to use ELK, Elasticsearch, Logstash and Kibana with Beats. Filebeat, that belongs to Beats series will forward the logs that read from a file to logstash. In logstash you could tag, filter, parse and modify the log entries that are stored in Elasticsearch. For visualizing you can use Kibana. You can create a great dashboards based on the logs with ...


4

If editing critical flat-files across multiple servers in one place is too scary.. One approach I have often recommended is to let system users reside in /etc/passwd files, and to add an additional Name Service Switch (NSS) source for humans that need to log in. For big projects (lots of users authenticating against the PAM user data), that means LDAP, and ...


4

EBS Snapshots aren't really disk image files. They're logical containers of pointers to chunks of compressed and deduplicated backup data that EBS stores in S3 when you create snapshots -- but these are stored in buckets that EBS owns and controls. They are user-accessible via only one mechanism -- creating a new EBS volume from the snapshot. When you do ...


4

Look at AWS Fargate - it lets you run your Docker containers without having to spin up the EC2 instances for an ECS cluster. You simply schedule the Tasks (or Services), each gets is own IP address and they can talk to each other. You can also use AWS Service Discovery to facilitate the registration and lookup of the tasks' IP addresses. BTW Note that the ...


4

We faced exactly the same choices for our live service that deploys a laravel API and multiple react apps. There is no generic correct answer to a question "what tool/provider is better?" as it depends on your circumstances. Cost is a particularly tricky question. Are we just talking about the cost of getting something live, or the cost of running an ...


3

Although i have not found out what leads to this. But i am finally able to login into my instance by following these steps: 1. As you are logged in create a user in the instance and give it a password. $ adduser user1 $ passwd user1 2. Enable password authentication in /etc/sshd_config file. 3. Login with the new user. Interestingly, this user does not ...


3

This can be achieved by using Post build task plugin and by adding the following Script command in Post-build Action, Post build task solved the problem: vagrant destroy -f Plus adding rule with Log text as "Build was aborted" make sure that above command is only invoked on aborted builds. For example:


Only top voted, non community-wiki answers of a minimum length are eligible