How to create custom AWS CloudWatch metric for i-node

Question

I have created custom CloudWatch alarm for Disk Utilization for root and mount volume to monitor the disk space for my Ubuntu (16.04) instances, but quite often it happens that disk is only 60% full (df -h) and still it's unable to create any more file as the i-node (df -i) already reached its limit. To keep an eye on this, I want to introduce a custom metric for i-node utilization and then want to configure an alarm on it.

Please provide a solution, considering I am relatively new in AWS.

Answer 1

AWS have published a plugin for collectd which allows to send metrics collected using collectd to CloudWatch.

https://github.com/awslabs/collectd-cloudwatch

The collectd monitoring tool runs on the server and can be configured to collect various metrics, among them metrics on disks and filesystems using their DF plugin. https://collectd.org/wiki/index.php/Plugin:DF

You can turn the ReportInodes option of the CollectD DF plugin and then have these metrics reported to CloudWatch as custom metrics using the AWS CollectD CloudWatch plugin.

Answer 2

AWS have a CloudWatch agent which will submit a number of useful (configurable) metrics, including inodes_free, inodes_used, and inodes_total. They'll appear, by default, under the CWAgent namespace in CloudWatch metrics from where you can view them, add them to dashboards, create alarms etc.

For the full details, these links will be useful:

• installing the agent
• configuring the agent
• troubleshooting the agent

Alternatively, the quickest way to get started is to:

1. Download the agent from https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
2. Unzip and run install.sh
3. Start the agent by running /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start (this will create a default config file)
4. Edit the default config file at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json and modify the metrics_collected to include the inode metrics under the disk category (see below for sample).
5. Stop (above command but with -a stop instead) and start the agent for the new config to take effect.

You should then see the new metrics appear in CloudWatch within a minute or so. The agent should also automatically start at boot.

Here's a sample of defining the inode metrics (for full details see the configuration docs):

{
  "metrics": {
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}"
    },
    "metrics_collected": {
      "disk": {
        "measurement": [
          "inodes_free",
          "inodes_used",
          "inodes_total"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ]
      }
    }
  }
}

Answer 3

SSH into the instance and add the script that will collect the inode usage and send metrics to the alarm

!/bin/bash
USAGE=80
INODE_USED=$(df -ih | sed -n '4p'| awk '{print $5}'| grep -v U | cut -d% -f1)
aws cloudwatch put-metric-data --metric-name inode-usage --dimensions Instance=<Instance id>  --namespace "Custom" --value $INODE_USED