Two solutions for tagging elusive AWS EBS volumes
Looking for more info about controlling EBS volume costs? Check out our How to Keep Track of EBS Volumes and Control EBS Costs post from 2018!
When doing cost allocation for your AWS environment, one challenge is to make sure that untagged EBS volumes aren’t skewing your accounting. Here are two tagging solutions that can help wrangle EBS cost and usage data.
It’s no surprise to anyone that EC2 is by far the biggest cost that most AWS users encounter, and here at Cloudability, we see this average 70-75% of cloud spending. Being able to allocate these storage costs to individual business units, teams, or people is often an important, ongoing battle.
As we discussed here, other costs related to EC2 usage, such as data transfer and EBS volumes, are non-trivial and we’ve certainly seen cases where they are costing more than their actual instance hours.
We generally see fairly good compliance when it comes to tagging the actual EC2 instances themselves. However, one common situation we see out in the wild is that throughout all the effort to tag EC2 instances, other AWS resources tend to be neglected in the process.
EBS can be a particular tagging challenge: root volumes of instances can’t be easily tagged through CloudFormation. Because of this, we see a prevalence of EBS volumes continuing to go untagged, with their costs unallocated. There can also be a downstream effect as related data transfer costs will go untagged. What results is a big chunk of the EC2 component of your bill being difficult to allocate accurately.
In this blog post, we’ll go over a few tagging solutions to help account for EBS volumes.
Run a background tagging job (aka Tag Monkey)
One straightforward solution is to run a background job using a scripting language with an AWS SDK, or script it with the AWS CLI. This job would cycle through all attached EBS volumes and look for cases where a tag is wrong or missing.
When you hit a problem volume, the job will look up the instance it’s attached to and copy the tags from there. Here is a straightforward example of doing this through a ruby script and the AWS SDK:
This is handy for older volumes that might have been created outside of CloudFormation. It’s a great way to make sure things get into a good state and the approach could be used for tagging items like snapshots. Ideally, this script could run every hour.
Another “tag monkey” approach that would work really well would be to create a lambda function which works using the same principles. Lambda functions, by default, have access to the AWS SDK, which simplifies things. Doing this solution through lambda is also going to save the time in having to provision your own server for running the script.
While all of the above (done correctly) present great upsides, there is a downside to this solution. If the script stops working, you could have instance hours not tagged, and depending on your script execution cadence there could be a delay in those volumes getting tagged.
Use CloudFormation with EC2 User Data
When you spin up any EC2 instance, you have the option of passing in user data as a shell script to perform important configuration tasks. We can also apply this through CloudFormation when provisioning the EC2 instance to make sure the attached root volume is tagged as desired.
Using this technique requires the instance to have AWS CLI tools installed. If your instance is based on the AWS Linux AMI you’ll already have this. Otherwise, it’s a pretty simple matter of installing the CLI into your AMI.
Here’s an example with user data that can be applied directly through your CloudFormation stack:
This is a pretty straightforward bash script. We’re able to use the metadata endpoint from within the instance to get the instance id. Then using the AWS CLI we can get the volume ID of the volume attached to the instance and then tag it as desired.
The best thing about this solution is that your volumes are tagged pretty much straight away and you have fine grained control of your tags through CloudFormation. The obvious negative is needing to have the AWS CLI installed into your AWS AMI.
Once a solution is in place, visualize its progress
Once you have put a strategy in place to tag your EBS volumes, you’re going to want to find a way to visualise and validate that the EBS volumes are being tagged correctly. Doing this also helps inform ops or finance teams that you now know where the unallocated EBS costs are coming from.
Creating a focused report within Cloudability is going to be a great way to achieve this. Here’s an example you can start with:
This is an example of a ‘required’ tag Application for EBS volume costs. As tagging gets put in place we’d hope to see the (not set) costs be replaced by actual tag values. This is going to help your cost allocation efforts tremendously.
So, if you find yourself dealing with unidentified costs in your next AWS bill, and you have EBS backed instances, try one of these solutions to improve how you track your storage use. If you need any help, or want to explore ways to visualize your cost and usage data once everything is properly tagged, let us know!