Assume I have two clusters devcluster and qacluster on ECS.
How to remove an EC2 instance (ecs-optimized amazon-linux) from qacluster and add to devcluster?
Add a comment
|
3 Answers
You need to follow these steps:
- Set
ECS_CLUSTER=devclusterin/etc/ecs/ecs.config - Stop all tasks/containers
- Remove checkpoint file -
/var/lib/ecs/data/ecs_agent_data.json - Start ECS agent again as explained here - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html
To make sure it uses correct cluster, check logs - docker logs ecs-agent. It should have a message like:
[INFO] Registration completed successfully. I am running as 'arn:aws:ecs:eu-west-1:ACCOUNT_ID:container-instance/CLUSTER_ID' in cluster 'devcluster'
-
1You should avoid publishing any ressource arn without masking your account id and the ressource id in my opinion. I've edited and cleared the the revision history.– TensibaiJan 24, 2018 at 15:54
-
@Tensibai you're totally right. But it was actually masked as I have replaced last 6 digits with zeros ;) But this change would not harm.– lagivanJan 24, 2018 at 22:31
-
There's a small chance it match someone else account number, prefer replacing with xxxx than with 0 to avoid bots scanning the internet launching attacks :)– TensibaiJan 24, 2018 at 23:13
The way a cluster becomes aware of the EC2 instances associated to it, is a configuration file used by the ECS agent. You can modify this file, located at /etc/ecs/ecs.config, and name a different cluster. Then restart the ECS agent. This will effectively "move" the EC2 instance to the other cluster.
More information about ecs.config in the documentation - http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html
After the modification, it is possible to verify by sending an HTTP request to the ECS Agent metadata service and see the new cluster displayed. http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html
The ECS cluster itself also needs to be notified. Can use the Register/Deregister API calls for this, for example using the AWS CLI:
- http://docs.aws.amazon.com/cli/latest/reference/ecs/register-container-instance.html
- http://docs.aws.amazon.com/cli/latest/reference/ecs/deregister-container-instance.html
More housekeeping might be required on the ECS agent itself. Like removing cached checkpoint files, etc. https://github.com/aws/amazon-ecs-agent/issues/139#issuecomment-123061463
answered Sep 18, 2017 at 4:45
-
i modified
/etc/ecs/ecs.configtoECS_CLUSTER=devclusterand restarted ec2 instance. but the instance still appear onqacluster– sithumcSep 18, 2017 at 5:19 -
@sith I updated the answer with details about registration of instances in the ECS Cluster metadata. Might need to be updated as well, or else the cluster does not know about a new instance or about a removed one. Sep 18, 2017 at 7:06
Updated for ECS optimized AMI image as of May 2021. The sudo start command is not present and the checkpoint file /var/lib/ecs/data/ecs_agent_data.json changed to agent.db.
- Edit
/etc/ecs/ecs.configas previously described (add ECS_CLUSTER=devcluster). - Rename the agent state file with
mv /var/lib/ecs/data/agent.db /var/lib/ecs/data/agent.db.old - Restart the ECS agent using
sudo systemctl restart ecs
Check the status of the agent with systemctl status ecs, you want it to display Active: active (running).
This was a bit frustrating to figure out because the Amazon online documentation I found still displays the old information. Couldn't have done it without devops!
