I have a process (exe) on a Windows ec2 that is using up lots of memory. I would like to create a custom metric that will collect data and send to CloudWatch. I would like to also create an alarm when a certain threshold has been reached or surpassed.
Memory consumption is a metric that has to be reported from inside the guest (EC2 instance). You can use a CloudWatch agent to collect metrics from the guest directly. The agent also supports custom metrics now.
Alternatively, install a third party monitoring system. Which also means that you'll have to get an agent running on the EC2 instance. There are many of those, choose whatever fits your environment.