How Carsales.com Debugs Shifts in AWS Spending Patterns to Improve Cloud Efficiency
One key point we teach here at Cloudability is how the cloud is different from traditional infrastructure. There are many more stakeholders involved, with hourly decisions and actions that can have a real impact on your AWS spend. This is why it’s so important for organizations of every size (whether enterprise, business or growing startup) to have a strong culture around cloud management and develop processes that let you understand what drives changes in your spending and to deliver the right business intelligence for growth.
Something we are seeing our customers increasingly use to support this quest toward cloud efficiency is what we call the “Cloudability Diff!” For the engineers out there, you can think of this as something similar to a “git diff.” For the non-engineers, this is an exercise in comparing two bodies of work and identifying changes between them both.
There are three typical scenarios where this process is proving powerful:
- Use these “diff” reports to debug the exact cause of a distinct spike or shift in AWS spending that you may notice at any time.
- Use the technique to gauge or monitor the success of a cost optimization initiative.
- Run a repeating set of these reports at the end of each month to establish exactly what has changed compared to the previous month and what remediation/actions may follow.
This is all about being proactive, hoping to pick up trends early and works well being rolled into a regular cloud center of excellence (COE) meetup.
Carsales.com Puts the “Diff Report” to Work to Improve Cloud Management
Carsales.com is a Cloudability customer and the largest online automotive, motorcycle and marine classifieds service in Australia. They run plenty of services on AWS and use Cloudability as a cornerstone of their cloud management solution to know exactly what they’re spending on the cloud, how to eliminate waste and to get the most value out of AWS as their cloud infrastructure grows.
They use the “Cloudability Diff” frequently to maximize the return on every dollar spent on the cloud. For Michael Ridgway, Senior Development Manager, it helps him manage costs across their growing team.
“Until we utilized Cloudability as our cost management tool, it was close to impossible to reconcile who or what was driving changes in our AWS expenditure. The motivator for me was gaining the ability to seamlessly manage the cost of a complex, single payer account that includes over 15 teams.” – Michael Ridgway, Senior Development Manager, Carsales.com
Let’s take a look at their process and how they get it done.
Debugging an Unexpected Shift or Spike in Spending
We saw this exact scenario at Carsales.com during the previous year. In the chart below, we have a clear, if temporary, shift in daily AWS spending. How would we go about working out what the discrete drivers of this were?
* Note: the values throughout this post have been modified from their original source. However the concepts and characteristics are still valid and illustrate the Carsales.com exercise.
We can see a clear shift in the range of the 15th through 21st of September. We created a report in Cloudability comparing this period to that immediately preceding it. The datepicker works as such:
Pro-tip: Start with Account and AWS Service Name (Cloudability Dimension)
So what should the report look like? What dimensions and metrics should we include and what is the best starting point?
Everyone’s environment is going to be a bit different, but using Account and AWS Service Names is generally a great start and may be enough (note that Service Name is a special Cloudability dimension which allows us to delineate logical services such as EBS from the official AWS product of EC2. Read more here). For folks on a Free Trial or our Cloudability users, here is the example report for Carsales.com:
What we saw here was that the spending increase was almost driven entirely by Redshift usage in one particular account. We can easily dig deeper by clicking the account and then adding further dimensions such as “Application” or “Name” tag.
Measuring the Success of a Cost Optimization Initiative
If you’ve been using AWS for a good period of time, you’ve likely gone through some form of a cloud cost optimization initiative. This may involve numerous activities such as deleting unattached EBS volumes, reducing your provisioned IOPS, power cycling pre-prod EC2 instances or simply terminating unused resources.
In the background, teams may be increasing usage of other resources, and with so many moving parts day to day, it can be hard to gauge how successful you were. Let’s create a report similar to above but with a bit more color.
If your cost optimization initiative has been focused on usage, we can add a transaction type filter for that. Adding the “Usage Family” dimension to the report will further break down what exactly has changed. Make sure you pick a date range which matches your initiative and compare the period beforehand. For example, try this report:
You’ll be able to see exactly how much your cloud cost reductions (or increases are). To create less noise and zone in on the effects of your initiative, you may want to add further filters which are specific on the account, or a particular tag value (or add them as dimensions in the table).
Being Proactive: Compare Recent Spending to a Previous Period
This would be a great exercise as part of your monthly cloud cost optimization process. Reviewing last month and comparing it to the previous month is a great way to answer questions like:
- What has recently changed in the infrastructure?
- Is there anything we need to get on top of cost-wise?
- Are there stakeholders we should educate in helping us become more cloud efficient?
The dimensions you are going to be using here are a bit dependant on your setup, but Account Name, your highest level tag (such as department, business unit etc) and Service Name would be a good start. By doing this, you’ll no doubt make yourself aware of shifts and unexpected workloads that need to be addressed.
Pro-Tip: Another feature you could make use of is Cloudability’s automated anomaly detection. This feature is based around detecting statistically significant shifts in your spending and letting you know about it without needing to setup specific reports. If this is something you are interested in doing, and you’re currently a customer, reach out to your dedicated TAM to get started.
Awareness of Cloud Cost Changes is Key to Maximizing Value
This exercise is one of many tactics that help businesses and enterprise quickly identify the causes of spending spikes on their cloud and take action to remedy them. These steps can quickly provide the right people with the right cloud cost and usage data and lead to cost efficiency actions that produce significant savings over time.
Using a robust cloud management tool like Cloudability helps team members ingest cloud cost and usage data and build the insights that your team needs to optimize cloud spending while maximizing value. To see our analytics at work with your data, try us out for free or get in touch with one of our cloud cost optimization experts today.