Following the significant disruption of EBS volumes in April, the importance of backups was once again brought to the forefront. Amazon’s method of generating EBS backups is the ‘snapshot’ – a differential backup stored in S3.
While Amazon provides tools for creating snapshots (either command line, or via their API), a script that is quite popular for the task is ec2-consistent-snapshot. I have previously made mention of this script, as well as its installation on Amazon’s Linux and the remainder of this article presumes you have the Perl modules required by that script installed.
Once you begin accumulating regular snapshots, however, it soon becomes necessary to begin deleting the older snapshots (if for no other reason than that Amazon imposes a limit on snapshots, recently increased from 500 to 10000). The above script does not perform any rotation or deletion of snapshots, so an additional script must be employed.
There are several scripts available online which will do this, however few are written in Perl (the majority are in PHP). While most scripting languages should be able to perform the task, it would stand to reason that if one is already using a Perl script (ec2-consistent-snapshot) that a conserved set of parameters and dependencies would be favourable.
Having not found a script that matched my requirements, I decided to try my hand at writing one of my own over the past few days and have included it below in case it might be of use to anyone else.
I have tested the script under a variety of circumstances (Windows/Linux, different parameters, etc), and it has worked without issue; it is currently in use on my system and has been performing its task admirably thus far.
(While I have programmed in a variety of languages, this is my first Perl script – so undoubtedly there are some non-standard code practises – use at your own risk. It was written in Perl for the reasons outlined above, and because it presented a practical opportunity to use the language.)
Objectives
- Support Grandfather-Father-Son style rotation
- Use the same basic command line arguments as ec2-consistent-snapshot
- Allow the specification of the number of hourly, daily, weekly, and monthly snapshots to be kept
- This script was not designed to keep the x most recent snapshots
- Allow the specification of a preferred hour/day for snapshots
- e.g. keep Sunday’s snapshot for the weekly snapshot, or keep the 1st of the month for the monthly snapshot.
- Do not delete preferred snapshots if they will be used in the future
- e.g. if the snapshot from the 1st of the month would not be kept for weekly snapshot, but will later be needed for the monthly snapshot, it will be kept.
- Allow the specification of a preference for ‘newer/older’ in case the preferred snapshot is not available
- e.g. if preferred snapshot is Tuesday for weekly snapshot, but there is only a Monday and Friday snapshot, newer would keep Friday, older would keep Monday.
- Script can be run from any computer (it does not need to be run from the server hosting the EBS volumes)
- Script can be run at any time (while it may be desirable to run after each snapshot, there is no necessity to do so)
Basic Methodology
- The script, gets a list of all snapshots owned by you, and calculates the time limits for hourly, daily, weekly, and monthly snapshots.
- All snapshots which are for the volumes passed, not on the exclusion list, and fall within the accepted date range are processed.
- The script then calculates which time point (1st, 2nd, 3rd, etc) the snapshot applies to (e.g. it matches the 1st hourly snapshot, or the 3rd weekly snapshot, etc).
- If more than one snapshot falls within the same time point, one is kept (based on the preferences passed) and the others are marked for deletion. Before marking a snapshot for deletion, the script will check if it will be needed later on.
- e.g. if you keep Wednesday snapshots for your weekly backup, and the 15th of the month for your monthly backup, but the 15th is a Thursday, the script will keep both the Wednesday back and the Thursday backup for that week.
- All snapshots for the specified volumes that are not to be kept will be deleted.
The script uses the Net::Amazon::EC2 module for EBS specific functions (e.g. getting the list of snapshots, performing the deletion, etc) and the DateTime module for all date calculations.
Installation
Download (below) and extract (or copy and paste)
In addition to the Perl modules required by ec2-consistent-snapshot, you need: DateTime and DateTime::Format::DateParse (install via yum (below) or CPAN).
yum install perl-DateTime perl-DateTime-Format-DateParse -y
This script can be run from anywhere, but it may be desirable to place it with the AWS command line tools (/opt/aws/bin/)
Note: It appears that v0.14 of Net::Amazon::EC2 is required for proper functioning of this script. See this comment, below, for further details.
Remember to make the file executable:
chmod +x ec2-prune-snapshots
Known/Suspected Bugs
- Times may not be fully consistent across time zones (Amazon uses UTC)
- Unexpected behaviour may occur if DAYOFMONTH exceeds number of days in month (e.g. if DAYOFMONTH=31 in Feb) [workaround is to specify an appropriate FAVOURRECENT]
- Multiple snapshots on the same day may be kept for future purposes, instead of being narrowed to the single necessary one (if more than one snapshot for the same volume is taken in an hour) [Edge case]
- Some error checking missing (e.g. does not confirm valid ec2-endpoint)
Changelog
Jul. 29, 2011 – add ‘use File::Slurp;’ (line 19) to resolve dependency for AWS keys passed via file.
Dec. 15, 2011 – fix typo in usage ‘–favour-recent’
Mar. 7, 2012 – fixed line endings on download
Mar. 30, 2012 – added ‘–exclude’ parameter
Usage
ec2-prune-snapshots.pl --aws-access-key-id KEY --aws-secret-access-key SECRET --region us-east-1 --months 3 --weeks 4 --days 7 --noaction VOL1 VOL2 VOL3…
(Keys may be passed in the same manner as used with ec2-consistent-snapshot: as a parameter, an individual file, or combined in a credentials file)
Change the months, weeks, and days, values to suit your needs, add ‘hours’ if desired; and specify additional parameters for added control (see help).
Remove the --noaction parameter to actually perform the deletions
Run with --debug for detailed action reporting
Run with --quiet to suppress all output other than errors
Run ec2-prune-snapshots --help for full instructions
The Script
ec2-prune-snapshots: (Download: ec2-prune-snapshots.gz [4.9kB, MD5: 8787BB6987F05B3946626EA7C2D40C0A])
#!/usr/bin/perl
#
# (C) 2011-2012 cyberx86 [ThatsGeeky.com]
# Some sections (C) 2008-2011 Eric Hammond
#
#Todo:
#Ensure consistency across timezones (Amazon uses UTC)
#Possible problems: if DAYOFMONTH exceeds number of days in month (e.g. if DAYOFMONTH=31 in Feb)
#Edge case: multiple snapshots on the same day may be kept for future purposes, instead of being narrowed to the single necessary one.
use strict;
use warnings;
(our $Prog) = ($0 =~ m%([^/]+)$%);
use Getopt::Long;
use DateTime;
use DateTime::Format::DateParse;
use Pod::Usage;
use File::Slurp;
use Net::Amazon::EC2 0.11;
#---- OPTIONS ----
my $Help = 0;
my $Debug = 0;
my $Quiet = 0;
my $Noaction = 0;
my $aws_access_key_id = $ENV{AWS_ACCESS_KEY_ID};
my $aws_secret_access_key = $ENV{AWS_SECRET_ACCESS_KEY};
my $aws_access_key_id_file = $ENV{AWS_ACCESS_KEY_ID};
my $aws_secret_access_key_file = $ENV{AWS_SECRET_ACCESS_KEY};
my $aws_credentials_file = $ENV{AWS_CREDENTIALS};
my $region = undef;
my $ec2_endpoint = undef;
my @exclusions;
my $hours = 0;
my $days = 0;
my $weeks = 0;
my $months = 0;
# Always keep the dayofweek and dayofmonth to be consistent.
my $hourofday = 0; #Midnight
my $dayofweek = 7; #Sunday
my $dayofmonth = 1;
my $favour_recent = 0; #0=keep older; 1=keep newer
Getopt::Long::config('no_ignore_case');
GetOptions(
'help|?' => \$Help,
'debug' => \$Debug,
'quiet' => \$Quiet,
'noaction' => \$Noaction,
'aws-access-key-id=s' => \$aws_access_key_id,
'aws-secret-access-key=s' => \$aws_secret_access_key,
'aws-access-key-id-file=s' => \$aws_access_key_id_file,
'aws-secret-access-key-file=s' => \$aws_secret_access_key_file,
'aws-credentials-file=s' => \$aws_credentials_file,
'region=s' => \$region,
'ec2-endpoint=s' => \$ec2_endpoint,
'hours=i' => \$hours,
'days=i' => \$days,
'weeks=i' => \$weeks,
'months=i' => \$months,
'hourofday=i' => \$hourofday,
'dayofweek=i' => \$dayofweek,
'dayofmonth=i' => \$dayofmonth,
'favour-recent=i' => \$favour_recent,
'exclude=s' => \@exclusions,
) or pod2usage(2);
pod2usage(1) if $Help;
my @volume_ids = @ARGV;
pod2usage(2) unless scalar @volume_ids;
pod2usage(2) unless (($hours >=0 && $days >= 0 && $weeks >=0 && $months >=0) && ($hours > 0 || $days > 0 || $weeks > 0 || $months > 0));
$ec2_endpoint ||= "https://ec2.$region.amazonaws.com" if $region;
#---- MAIN ----
($aws_access_key_id, $aws_secret_access_key) = determine_access_keys(
$aws_access_key_id, $aws_secret_access_key,
$aws_access_key_id_file, $aws_secret_access_key_file,
$aws_credentials_file,
);
die "$Prog: ERROR: Can't find AWS access key or secret access key"
unless $aws_access_key_id and $aws_secret_access_key;
$Debug and warn "$Prog: Using AWS access key: $aws_access_key_id\n";
prune_snapshots(\@volume_ids, $ec2_endpoint, $hours, $days, $weeks, $months, $hourofday, $dayofweek, $dayofmonth, $favour_recent);
exit 0;
END {
$Debug and warn "$Prog: ", scalar localtime, ": done\n";
}
#---- METHODS ----
sub prune_snapshots {
my ($volume_ids, $ec2_endpoint, $hours, $days, $weeks, $months, $hourofday, $dayofweek, $dayofmonth, $favour_recent) = @_;
$Debug and warn "$Prog: ", scalar localtime, ": begin snapshot pruning\n";
$Debug and warn "$Prog: Endpoint: $ec2_endpoint\n" if $ec2_endpoint;
$Debug and warn "$Prog: Vols: @$volume_ids\n";
my $now = DateTime->now( time_zone => 'UTC' );
# EC2 API object
my $ec2 = Net::Amazon::EC2->new(
AWSAccessKeyId => $aws_access_key_id,
SecretAccessKey => $aws_secret_access_key,
($ec2_endpoint ? (base_url => $ec2_endpoint) : ()),
# ($Debug ? (debug => 1) : ()),
);
my $current_snapshots = $ec2->describe_snapshots(Owner => 'self');
my $min_hours = $now->clone->subtract(hours => $hours);
my $min_days = $min_hours->clone->subtract(days => $days);
my $min_weeks = $min_days->clone->subtract(weeks=>$weeks);
my $min_months = $min_weeks->clone->subtract(months=>$months);
$Debug and warn "$Prog: Earliest for 'hours': ", $min_hours->datetime(), "\n";
$Debug and warn "$Prog: Earliest for 'days': ", $min_days->datetime(), "\n";
$Debug and warn "$Prog: Earliest for 'weeks': ", $min_weeks->datetime(), "\n";
$Debug and warn "$Prog: Earliest for 'months': ", $min_months->datetime(), "\n";
my %kept_snaps;
my %del_snaps;
foreach my $snapshot (@$current_snapshots) {
my $snapnum;
my $snapshot_date = DateTime::Format::DateParse->parse_datetime($snapshot->start_time);
if ($snapshot->status ne 'completed'){
#snapshots that are still in progress
$Debug and warn $snapshot->snapshot_id , " not completed", "\n";
}elsif(!grep $_ eq $snapshot->volume_id, @$volume_ids){
#snapshots for volumes not specified
$Debug and warn "$Prog: Snapshot [", $snapshot->snapshot_id, "] skipped; volume [",$snapshot->volume_id,"] not in list", "\n";
}elsif(grep $_ eq $snapshot->snapshot_id, @exclusions){
#snapshot should be excluded
$Debug and warn "$Prog: Snapshot [", $snapshot->snapshot_id, "] skipped; on exclusions list", "\n";
}elsif (DateTime->compare($snapshot_date, $min_hours)>=0){
#snapshots in hourly range
#$snapnum = $now->clone->subtract_datetime($snapshot_date)->hours();
$snapnum = $now->clone->delta_ms($snapshot_date)->in_units('hours');
process_snaps ('hour', $snapnum, \%kept_snaps, \%del_snaps, $snapshot, $snapshot_date, $hourofday, $dayofweek, $dayofmonth, $favour_recent);
}elsif (DateTime->compare($snapshot_date, $min_days)>=0){
#snapshots in daily range
$snapnum = $min_hours->clone->delta_days($snapshot_date)->delta_days();
process_snaps ('day', $snapnum, \%kept_snaps, \%del_snaps, $snapshot, $snapshot_date, $hourofday, $dayofweek, $dayofmonth, $favour_recent);
}elsif (DateTime->compare($snapshot_date, $min_weeks)>=0){
#snapshots in weekly range
$snapnum = $min_days->clone->delta_days($snapshot_date)->weeks();
process_snaps ('week', $snapnum, \%kept_snaps, \%del_snaps, $snapshot, $snapshot_date, $hourofday, $dayofweek, $dayofmonth, $favour_recent);
}elsif (DateTime->compare($snapshot_date, $min_months)>=0){
#snapshots in monthly range
$snapnum = abs($min_weeks->clone->subtract_datetime($snapshot_date)->in_units('months'));
process_snaps ('month', $snapnum, \%kept_snaps, \%del_snaps, $snapshot, $snapshot_date, $hourofday, $dayofweek, $dayofmonth, $favour_recent);
}else{
#anything that makes it here is to be deleted because it is too old
push @{ $del_snaps{$snapshot->volume_id} }, $snapshot->snapshot_id;
$Debug and warn "$Prog: ". $snapshot->snapshot_id . " [" . $snapshot->volume_id . "; " . $snapshot_date ."] to be DELETED (3)\n";
}
}
#Debug: Display kept_snaps and del_snaps (snaps kept for future not shown);
if ($Debug){
my ($vol_id,$period, $num, $snap_id);
foreach $vol_id (keys %kept_snaps) {
foreach $period (keys %{$kept_snaps{$vol_id}}){
foreach $num (sort keys %{$kept_snaps{$vol_id}{$period}}){
warn "$Prog: Snapshot [", $kept_snaps{$vol_id}{$period}{$num}{'snap_id'}, "] ", $kept_snaps{$vol_id}{$period}{$num}{'time'}->day_abbr() . " " . $kept_snaps{$vol_id}{$period}{$num}{'time'}->month_abbr() . " " . $kept_snaps{$vol_id}{$period}{$num}{'time'}->day_of_month() . ", " . $kept_snaps{$vol_id}{$period}{$num}{'time'}->year() . " " . $kept_snaps{$vol_id}{$period}{$num}{'time'}->hms(':'), " to be KEPT ($period)\n";
}
}
}
foreach $vol_id (keys %del_snaps) {
foreach $snap_id (@{$del_snaps{$vol_id}}){
warn "$Prog: Snapshot [", $snap_id, "] to be DELETED\n";
}
}
}
#Do the actual deletion:
my ($volume_id, $snaps_to_del, $snap_to_del);
while (($volume_id, $snaps_to_del) = each(%del_snaps)){
$Debug and warn "$Prog: Deleting snaps for ". $volume_id.":\n";
foreach $snap_to_del (@$snaps_to_del){
if ( $Noaction ) {
warn "$Prog: Snapshot [", $snap_to_del, "] deletion SKIPPED [--noaction]\n";
}else{
#could pass snapshots as array ref...
if ($ec2->delete_snapshot(SnapshotId => $snap_to_del)){
$Quiet or print "Snapshot [", $snap_to_del, "] deleted\n";
}else{
warn "$Prog: Snapshot [", $snap_to_del, "] deletion FAILED\n";
}
}
}
}
}
sub process_snaps {
my ($timeperiod, $snapnum, $kept_snaps, $del_snaps, $snapshot, $snapshot_date, $hourofday, $dayofweek, $dayofmonth, $favour_recent) = @_;
if (!defined $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}){
#no snap for this time point
$kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'}=$snapshot->snapshot_id;
$kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}=$snapshot_date;
$Debug and warn "$Prog: ". $snapshot->snapshot_id . " [" . $snapshot->volume_id . "; " . $snapshot_date ."] to be KEPT [".$timeperiod." #". $snapnum ."] (1)\n";
}elsif (
($timeperiod eq 'hour') ||
($timeperiod eq 'day' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->hour() != $hourofday || $snapshot_date->hour() == $hourofday)) ||
($timeperiod eq 'week' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_week() != $dayofweek || $snapshot_date->day_of_week() == $dayofweek)) ||
($timeperiod eq 'month' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_month() != $dayofmonth || $snapshot_date->day_of_month() == $dayofmonth))
){
#already a snap for this time point AND (currently kept snap is not preferred date OR new one is preferred date)
if (
($timeperiod ne 'hour') && (
($timeperiod eq 'day' && ($snapshot_date->hour() == $hourofday && $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->hour() !=$hourofday)) ||
($timeperiod eq 'week' && ($snapshot_date->day_of_week() == $dayofweek && $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_week() !=$dayofweek)) ||
($timeperiod eq 'month' && ($snapshot_date->day_of_month() == $dayofmonth && $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_month() !=$dayofmonth))
)){
#currently kept snap is not preferred date AND new one is preferred date, so keep new and delete old
if (
#($timeperiod eq 'hour' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->hour != $hourofday)) ||
($timeperiod eq 'day' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_week != $dayofweek && $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_month != $dayofmonth)) ||
($timeperiod eq 'week' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_month != $dayofmonth)) ||
($timeperiod eq 'month')
){
push @{ $del_snaps->{$snapshot->volume_id} }, $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'};
$Debug and warn "$Prog: ". $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'} . " [" . $snapshot->volume_id . "; " . $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'} ."] to be DELETED (switch) [".$timeperiod." #". $snapnum ."] (1)\n";
}else{
#don't delete the currently kept snap if it will be needed for a future time point
$Debug and warn "$Prog: ". $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'} . " [" . $snapshot->volume_id . "; " . $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'} ."] to be KEPT (for future) [".$timeperiod." #". $snapnum ."] (1)\n";
}
$kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'}=$snapshot->snapshot_id;
$kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}=$snapshot_date;
$Debug and warn "$Prog: ". $snapshot->snapshot_id . " [" . $snapshot->volume_id . "; " . $snapshot_date ."] to be KEPT [".$timeperiod." #". $snapnum ."] (2)\n";
}else{
#new and currently kept both correct OR both incorrect, check if we should switch them
if(DateTime->compare( $snapshot_date, $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'} )==($favour_recent*2-1)){
#new snapshot is favoured
if (
($timeperiod eq 'hour' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->hour != $hourofday)) ||
($timeperiod eq 'day' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_week != $dayofweek && $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_month != $dayofmonth)) ||
($timeperiod eq 'week' && ($kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}->day_of_month != $dayofmonth)) ||
($timeperiod eq 'month')
){
#delete currently kept snap
push @{ $del_snaps->{$snapshot->volume_id} }, $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'};
$Debug and warn "$Prog: ". $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'} . " [" . $snapshot->volume_id . "; " . $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'} ."] to be DELETED (switch) [".$timeperiod." #". $snapnum ."] (2)\n";
}else{
#keep the currently kept snap for a future time point
$Debug and warn "$Prog: ". $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'} . " [" . $snapshot->volume_id . "; " . $kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'} ."] to be KEPT (for future) [".$timeperiod." #". $snapnum ."] (2)\n";
}
$kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'snap_id'}=$snapshot->snapshot_id;
$kept_snaps->{$snapshot->volume_id}{$timeperiod}{$snapnum}{'time'}=$snapshot_date;
$Debug and warn $snapshot->snapshot_id . " [" . $snapshot->volume_id . "; " . $snapshot_date ."] to be KEPT [".$timeperiod." #". $snapnum ."] (3)\n";
}else{
#currently kept snapshot favoured - delete new snapshot
if (
($timeperiod eq 'hour' && ($snapshot_date->hour != $hourofday)) ||
($timeperiod eq 'day' && ($snapshot_date->day_of_week != $dayofweek && $snapshot_date->day_of_month != $dayofmonth)) ||
($timeperiod eq 'week' && ($snapshot_date->day_of_month != $dayofmonth)) ||
($timeperiod eq 'month')
){
#delete new snap
push @{ $del_snaps->{$snapshot->volume_id} }, $snapshot->snapshot_id;
$Debug and warn "$Prog: ". $snapshot->snapshot_id . " [" . $snapshot->volume_id . "; " . $snapshot_date ."] to be DELETED [".$timeperiod." #". $snapnum ."] (1)\n";
}else{
#keep the new snap for a future time point
$Debug and warn "$Prog: ". $snapshot->snapshot_id . " [" . $snapshot->volume_id . "; " . $snapshot_date ."] to be KEPT (for future) [".$timeperiod." #". $snapnum ."] (3)\n";
}
}
}
}else{
#already a snap for this time point AND currently kept snap is preferred date AND new one is NOT preferred date
if (
($timeperiod eq 'hour' && ($snapshot_date->hour != $hourofday)) ||
($timeperiod eq 'day' && ($snapshot_date->day_of_week != $dayofweek && $snapshot_date->day_of_month != $dayofmonth)) ||
($timeperiod eq 'week' && ($snapshot_date->day_of_month != $dayofmonth)) ||
($timeperiod eq 'month')
){
#delete new kept snap
push @{ $del_snaps->{$snapshot->volume_id} }, $snapshot->snapshot_id;
$Debug and warn "$Prog: ". $snapshot->snapshot_id . " [" . $snapshot->volume_id . "; " . $snapshot_date ."] to be DELETED [".$timeperiod." #". $snapnum ."] (2)\n";
}else{
#keep the new snap for a future time point
$Debug and warn "$Prog: ". $snapshot->snapshot_id . " [" . $snapshot->volume_id . "; " . $snapshot_date ."] to be KEPT (for future) [".$timeperiod." #". $snapnum ."] (4)\n";
}
}
}
# Figure out which AWS access keys to use
sub determine_access_keys {
my ($aws_access_key_id, $aws_secret_access_key,
$aws_access_key_id_file, $aws_secret_access_key_file,
$aws_credentials_file,
) = @_;
# 1. --aws-access-key-id and --aws-secret-access-key
return ($aws_access_key_id, $aws_secret_access_key)
if $aws_access_key_id;
# 2. --aws-access-key-id-file and --aws-secret-access-key-file
if ( $aws_access_key_id_file ) {
die "$Prog: Please provide both --aws-access-key-id-file and --aws-secret-access-key-file"
unless $aws_secret_access_key_file;
$aws_access_key_id = File::Slurp::read_file($aws_access_key_id_file);
$aws_secret_access_key= File::Slurp::read_file($aws_secret_access_key_file);
chomp($aws_access_key_id);
chomp($aws_secret_access_key);
return ($aws_access_key_id, $aws_secret_access_key);
}
# 3. $AWS_CREDENTIALS or $HOME/.awssecret
return read_awssecret($aws_credentials_file);
}
# Look for the access keys in $AWS_CREDENTIALS or ~/.awssecret
sub read_awssecret {
my ($aws_credentials_file) = @_;
$aws_credentials_file ||= "$ENV{HOME}/.awssecret";
my ($aws_access_key_id, $aws_secret_access_key);
eval {
($aws_access_key_id, $aws_secret_access_key) =
File::Slurp::read_file($aws_credentials_file);
chomp $aws_access_key_id;
chomp $aws_secret_access_key;
};
return ($aws_access_key_id, $aws_secret_access_key);
}
=head1 NAME
ec2-prune-snapshots - Delete old snapshots in grandfather-father-son style
=head1 SYNOPSIS
ec2-prune-snapshots [opts] VOLUMEID1 [VOLUMEID2]...
At least one of hours, days, weeks, or months is required.
=head1 OPTIONS
-h --help Print help and exit.
-d --debug Debug mode.
-q --quiet Quiet mode.
-n --noaction Don't do it. Just say what you would have done.
--aws-access-key-id KEY
--aws-secret-access-key SECRET
Amazon AWS access key and secret access key. Defaults to
environment variables or .awssecret file contents described below.
--aws-access-key-id-file KEYFILE
--aws-secret-access-key-file SECRETFILE
Files containing Amazon AWS access key and secret access key.
Defaults to environment variables or .awssecret file contents
described below.
--aws-credentials-file CREDENTIALSFILE
File containing both the Amazon AWS access key and secret access
key on seprate lines and in that order. Defaults to contents of
$AWS_CREDENTIALS environment variable or the value $HOME/.awssecret
--region REGION
Specify a different region like "eu-west-1". Defaults to
"us-east-1".
--exclude SNAPSHOT_ID
Specify the ID of a snapshot to exclude from processing. This parameter
may be used more than once to specify multiple snapshots. For example:
"--exclude snap-00000001 --exclude snap-00000002".
--hours HOURSTOKEEP
The number of hours of snapshots to keep, from the current date (integer)
--days DAYSTOKEEP
The number of days of snapshots to keep, from the current date (integer).
Days start after the period specified by HOURSTOKEEP. All snapshots taken on
HOUROFDAY, until the end of the specified time will be kept. If there is
no snapshot for HOUROFDAY (or more than one snapshot for HOUROFDAY),
FAVOURRECENT will be used to determine which to keep.
--weeks WEEKSTOKEEP
The number of weeks of snapshots to keep, from the current date (integer).
Weeks start after the period specified by DAYSTOKEEP. All snapshots taken on
DAYOFWEEK, until the end of the specified time will be kept. If there is
no snapshot for DAYOFWEEK (or more than one snapshot for DAYOFWEEK),
FAVOURRECENT will be used to determine which to keep.
--months MONTHSTOKEEP
The number of months of snapshots to keep, from the current date (integer).
Months start after the period specified by WEEKSTOKEEP. All snapshots taken on
DAYOFMONTH, until the end of the specified time will be kept. If there is
no snapshot for DAYOFMONTH (or more than one snapshot for DAYOFMONTH),
FAVOURRECENT will be used to determine which to keep.
--hourofday HOUROFDAY
The (numeric) hour (0-23) to keep for the daily snapshot.
If available this value will take precedence over FAVOURRECENT.
--dayofweek DAYOFWEEK
The (numeric) day (1-7, with 1=Mon and 7=Sun) to keep for the weekly snapshot.
If available this value will take precedence over FAVOURRECENT. Snapshots from
this day will not be deleted as they will be needed in the future for a weekly
snapshot.
--dayofmonth DAYOFMONTH
The (numeric) day (1-31) to keep for the monthly snapshot. If the DAYOFMONTH
doesn't exist, FAVOURRECENT will be used to determine which snapshot to keep.
If available this value will take precedence over FAVOURRECENT. Snapshots from
this day will not be deleted as they will be needed in the future for a monthly
snapshot.
--favour-recent FAVOURRECENT
If this is set to 1, in the event that either no HOUROFDAY/DAYOFWEEK/DAYOFMONTH
is set, or a snapshot for the time period is not available, or more than one
snapshot falls within the time period, the most recent snapshot from the time
period will be used. If this is set to 0, the oldest snapshot will be used in
the above scenario.
=head1 ARGUMENTS
VOLUMEID EBS volume id(s) for which to rotate snapshots.
Additionally, at least one of days, weeks, or months is required.
=head1 AUTHOR
EBS Pruning functions:
cyberx86 [ThatsGeeky.com]
AWS Key processing and other sections:
Eric Hammond
=head1 LICENSE
Copyright (C) cyberx86 [ThatsGeeky.com]
Some sections copyright (C) 2009-2011 Eric Hammond
Licensed under the Apache License, Version 2.0, see
http://www.apache.org/licenses/LICENSE-2.0
=cut
Download: ec2-prune-snapshots.gz (4.9kB, MD5: 8787BB6987F05B3946626EA7C2D40C0A)
If you find a bug, have an improvement or suggestion, or simply found it useful, please let me know.
Great info on backing up EC2, thanks!
However, can’t seem to get –aws-credentials-file working (works fine with ec2-consistent-snapshot) Any thoughts?
Thanks for pointing that out. You had me puzzled for a moment – the code for that section is exactly the same as ec2-consistent-snapshot, however I think I know the problem. I didn’t include the ‘File::Slurp’ module (line 19) – as I didn’t use it in my code anywhere, and since I always pass the credentials through the command line, it never came up in my testing. I have made the (one line) fix, and updated the scripts above – however, I won’t get a chance to test it until tonight/tomorrow. If you use it, do let me know if it works. (It will take a few minutes for the change to propagate through CloudFront, so if you download the file, check the MD5 sum.) Thanks.
Works like a charm now, thanks!
Works as advertised. Great work!
Thanks – glad it worked for you.
Pretty usefull script! Thnx..
Found one inconsistency though. In the usage section you describe ‘favor-recent’, but it should be ‘favour-recent’ otherwise it won’t work.
Grtz,
marcel
Thanks for pointing that out, I’ve updated the script with the change. (Such spelling mistakes are an unfortunate side effect of a language that has ‘localized’ spellings of certain words – I am used to using the ‘Canadian’ spelling of most words, but computers tend to use the ‘American’ spelling.) Good to know you found the script useful.
I’ve been trying to run this script, but I get “Not an ARRAY reference on line 131”. Any ideas?
I would guess that line 117 is failing for some reason – perhaps no snapshots on the account, or incorrect authentication, etc. The variable set there ($current_snapshots) is what is used for the loop on Line 131. Enable Debug (set line 25 to 1), and see if you get any more useful output (if you post the output, but sure to obscure any sensitive information.)
I found the problem. I was using us-east-1b for the region instead of just us-east-1. Now, the script runs flawlessly. Thanks!
On a side note, you may want to consider adding a few lines to handle the scenario where there are no snapshots on the account (although admittedly, this is a rare edge case and may not be worth the time).
Glad to know it works for you. Thanks for letting me know what the problem was. As soon as I read your first comment, it occurred to me that perhaps some additional error-checking wouldn’t be amiss. I will look into it over the weekend (hopefully – one of many things on my to do list, unfortunately), and see what I come up with.
Well, I have a quick bit of error checking that can be added in (right after line 117):
if (ref $current_snapshots eq 'Net::Amazon::EC2::Errors') {print $current_snapshots->errors->[0]->message;
}
Pardon the n00by question, but after extracting this script to the /usr/bin directory (where I have ec2-consistent-snapshot) and running chmod -x on it, “ec2-prune-snapshots” returns “unable to execute /usr/bin/ec2-prune-snapshots: No such file or directory”
Ideas?
You can disregard the last question – the problem was Windows-y CR/LFs that prevented the interpreter from starting correctly. Now, however, I’m getting this error:
Attribute (progress) does not pass the type constraint because: Validation failed for ‘Maybe[Str]’ failed with value HASH(0xa0ca218) at /usr/share/perl5/Net/Amazon/EC2.pm line 2280
Net::Amazon::EC2::describe_snapshots(undef, ‘Owner’, ‘self’) called at /usr/bin/ec2-prune-snapshots line 117
main::prune_snapshots(‘ARRAY(0x9b25c98)’, ‘https://ec2.us-east-1.amazonaws.com’, 0, 30, 10, 6, 0, 7, 1, 0) called at /usr/bin/ec2-prune-snapshots line 92
Thanks for any input, I would love to use this script!
Sorry for the comments – but it turns out that the following line:
ec2-prune-snapshots –aws-credentials-file /home/bitnami/.awssecret –region us-east-1 –months 6 –weeks 10 –days 30 vol-99aafbf3
works correctly at the command line, but fails in a shell script (giving the above error). ec2-consistent-snapshot works correctly in that same script.
Here’s some more debug info that you might find helpful – it shows that the variables appear to be correctly instantiated and loaded –
120101-19:31:15 snapshot.sh starting
snap-24bf5140
ec2-prune-snapshots: Using AWS access key: [removed for privacy, but correctly loaded from .awssecret]
ec2-prune-snapshots: Sun Jan 1 19:31:16 2012: begin snapshot pruning
ec2-prune-snapshots: Endpoint: https://ec2.us-east-1.amazonaws.com
ec2-prune-snapshots: Vols: vol-99aafbf3
Attribute (progress) does not pass the type constraint because: Validation failed for ‘Maybe[Str]’ failed with value HASH(0xb20b488) at /usr/share/perl5/Net/Amazon/EC2.pm line 2280
Net::Amazon::EC2::describe_snapshots(undef, ‘Owner’, ‘self’) called at /usr/bin/ec2-prune-snapshots line 117
main::prune_snapshots(‘ARRAY(0xac54cf8)’, ‘https://ec2.us-east-1.amazonaws.com’, 0, 30, 10, 6, 0, 7, 1, 0) called at /usr/bin/ec2-prune-snapshots line 92
When you say it doesn’t work from a script – do you mean a script run directly, or via cron? Cron initializes a different environment, which may be the problem if you don’t have absolute paths specified.
Also, it is possible that you have multiple version of the Net::Amazon::EC2 package installed and a different one is being used in each circumstance. (Try
find / -name EC2.pm– you should only get one result).You can try changing the version on line 20 to 0.14 and see if it notifies you that you are running an older version. Also, you can uncomment line 114.
This appears to be an old bug in the Net::Amazon::EC2 package, that was fixed in v0.13 (current version is 0.14). If you are, in fact, running an older version of this package, try to upgrade to the current version and see if the problem persists. (Perhaps I should change the version requirements)
You can find the version you have installed by running:
perl -MNet::Amazon::EC2 -e 'print "$Net::Amazon::EC2::VERSION\n"'There was, actually this exact same problem reported for EC2-consistent-snapshot: https://bugs.launchpad.net/ec2-consistent-snapshot/+bug/485692
Let me know if the problem is resolved. If it persists and I will look into it a bit more.
Thanks for the quick reply – here are brief answers to your questions:
1 – running script from command line as root (sudo)
2 – only one instance of EC2.pm
3 – version is 0.13. upgrading the package informed me that I have the latest version (?!?) but according to your post the problem was fixed in v0.13 anyway.
4 – uncommenting line 114 was interesting, it spat out an array of my snapshots which was identical whether running the script in bash or from the command line, which would appear to rule out some sort of syntax error in the shell script (I did not parameterize the command options). It also returned this which may help you –
QUERY TO SIGN: ActionDescribeSnapshotsAWSAccessKeyId[OBFUSCATED]OwnerselfSignatureVersion1Timestamp2012-01-02T01:08:05.000ZVersion2009-11-30
It seems I was mistaken, the problem was fixed in version 0.14. The relevant sections of the two versions are included below:
v0.13 (EC2.pm, lines 2276-2290):
unless ( grep { defined && length } $snap->{description} and ref $snap->{description} ne 'HASH') { $snap->{description} = undef; } my $snapshot = Net::Amazon::EC2::Snapshot->new( snapshot_id => $snap->{snapshotId}, status => $snap->{status}, volume_id => $snap->{volumeId}, start_time => $snap->{startTime}, progress => $snap->{progress}, owner_id => $snap->{ownerId}, volume_size => $snap->{volumeSize}, description => $snap->{description}, owner_alias => $snap->{ownerAlias}, );v0.14 (EC2.pm, lines 2276-2294):
unless ( grep { defined && length } $snap->{description} and ref $snap->{description} ne 'HASH') { $snap->{description} = undef; } unless ( grep { defined && length } $snap->{progress} and ref $snap->{progress} ne 'HASH') { $snap->{progress} = undef; } my $snapshot = Net::Amazon::EC2::Snapshot->new( snapshot_id => $snap->{snapshotId}, status => $snap->{status}, volume_id => $snap->{volumeId}, start_time => $snap->{startTime}, progress => $snap->{progress}, owner_id => $snap->{ownerId}, volume_size => $snap->{volumeSize}, description => $snap->{description}, owner_alias => $snap->{ownerAlias}, );Note the 3 lines added to v0.14 at line 2280:
unless ( grep { defined && length } $snap->{progress} and ref $snap->{progress} ne 'HASH') { $snap->{progress} = undef; }If you are unable to upgrade to version 0.14 (e.g. via cpan, or by downloading the source from here), you can just add in the 3 lines above to EC2.pm and everything should work.
Hopefully you get it working – do let me know if it succeeds or if you run into further difficulties.
After I download the file, and extract it using gunzip I got this error message when running the script.
How to fix this?
James
That is a line ending error. Possibly resulting from the fact that the file I uploaded was created on Windows. If you have dos2unix, you can run it on the script and it should resolve the problem. Alternatively, you can copy and paste the code from the page (and paste directly into a linux text editor). I’ll fix it up in a couple of minutes and update the script.
I try to copy and paste to mac terminal and this is the new error message when I run it to ec2 instance.
perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_CTYPE = "UTF-8", LANG = "en_US.UTF-8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C").James
The problem is likely caused by a missing locale for en_US. It seems to be a fairly common issue on Ubuntu. Running the following should fix it, although, I would suggest trying the downloaded version version (it has been updated) – there is a good chance that some UTF8 characters were added by the copy/paste – I don’t believe there should be any in the original script.
Fixed – the updated version has been uploaded and the MD5 sum has been updated (and I downloaded a copy and tested it, so all should be good). Thanks for pointing out that error – I have a tendency to update this site separately from the files I actually use – so it is fairly unlikely that I would have come across this problem myself.
Hi cyberx86,
I have some questions regarding to your script. Can I message you in email or even in chat? I have some private parts in AWS to ask you.
Thanks.
James
I have emailed you a link to a chat site, if you wish to use it. I can only stick around for about half an hour. If you have questions about the script though, they may be better posted here so other people can learn from them – or if they are about AWS post them on ServerFault. Hopefully none of your questions involve sensitive information (to understand how the script works doesn’t require any user-specific information, so you can obfuscate your IDs, etc. and the explanations should still be valid).
I haven’t received your email for the link. Anyway, I still don’t understand how the script works for my needs in EBS snapshot backup. I wanted to have retention period for all of my EBS snapshots and I came across here to see if it works for mine.
I made some test and this is the output of my 3 snapshot backups (http://pastebin.com/iBWqGRSy) based from ec2-describe-snapshots command. When I want to keep my snapshots in 4 days (– days 4), the result is this. I can’t understand how the counting starts…
Can you help me to explain?
Thanks.
James
Your 3 snapshots have the following time stamps:
The script will only keep 1 snapshot per timeframe – since, you have specified ‘days’ – only one snapshot will be kept per day, for the last 4 days.
Let’s say the current time is around 2012-03-08T04:40:00+0000, that means that 4 days ago is 2012-03-04T04:40:00+0000 – only snapshots after this date will be considered (of course, you can add weeks and months parameters as well). The only snapshots within the acceptable date range are the two on 2012-03-05. However, since both of those fall within the same day, only one will be retained (you can specify if you want the earlier or later one to be kept, but only one will ever be kept for a given timeframe. So, if you specify hours and days – one will be kept for each hour, and after the hours then the daily snapshots will be kept (however, always only one per timeframe).). The snapshot from 2012-02-24 is older than 4 days and will be deleted. The script is meant to keep snapshots based on time – not a number of snapshots (so you can’t ask it to keep the 4 most recent snapshots – although, that is considerably easier to code). For the basic overview of how the script works, see the ‘Basic Methodology’ section.
Essentially, by passing the script
--days 4, you are saying ‘keep at most one snapshot per day for the last 4 days’ – 4 days is calculated quite literally, but subtracting 4 days days from the current date and time (so a snapshot taken 4 days and 1 minute ago will not be kept). You can add ‘--debug‘ to the list of parameters you pass the script, and it will tell you what the earliest date(s) it will keep for each time frame are, and will show you a lot more detail to help you understand what is going on. For instance, part of the ‘debug’ output (for--days 4) may read :It won’t keep any hours (since it wasn’t specified), days is exactly 4 days ago, and weeks and months are set to days. It is important to note that each timeframe starts where the previous ends. So, if you specify
--hours 24 --days 4, it will be checking for hourly snapshots in the past 24 hours, and daily snapshots in the 4 days before that. The timeframes do not overlap. So, for--hours 24 --days 4, you get:Hope this helps, let me know if you have any more questions (sorry you didn’t get the email – I used the site Chatzy – a few of my friends use it and it has usually been reasonably good).
Thanks for your detailed explanation. To run your script, is it to be done through cron? How?
Thanks. James
It will execute the pruning operation whenever it is run. If you want to run it just once – you can do so; if you want to run it periodically – then cron would be the way to go.
The setup that I use is to take daily snapshots of each EBS volume I use, and then about 10 minutes later, to prune the old snapshots.
In cron (run as root) I have:
So every day, at 3:20am – I take my snapshots (
backup.shrunsec2-consistent-snapshoton each volume, with different parameters), and everyday at 3:30am the snapshot script gets rid of the old snapshots I no longer want. I keep 7 daily snapshots, 4 weekly backups, and 3 monthly backups. Setup your command to run (the various parameters available can be listed usingec2-prune-snapshots --help) and include the--noactionparameter initially to see what will be done, and once you are happy with it, remove that parameter to actually carry out the deletion.If you have automated your snapshot taking process with cron, then you probably want to add this to the same cron job – shouldn’t be any special setup – just use the full paths, since cron runs with a limited set of environment variables. If you only run snapshots manually, then you may prefer to run this script manually.
This would be great to run in different instance or even outside the network as long as the machine can connect to amazon.
Do you know any script like this for “RDS for MySQL” to apply in DB Snapshots?
Thanks.
James
Unfortunately, I can’t help with the RDS snapshots as I don’t use RDS. In theory the same basic idea should apply – but, of course, you would need to find a library that worked with RDS and replace all the EC2 parts with references to the equivalent RDS functions (which isn’t something I am likely to do at any rate, but anyone else is welcome to do so).
As you mention, this script can be run from any machine – it doesn’t need to be run on the instance with the EBS volumes. So, you may have 5 instances and only run this on one of them, providing the volume-ids of all the relevant EBS volumes – it will work just fine. You could even run it from your home/work computer instead of an instance and it will do the same task.
My daily schedule backup is from Sunday to Friday and Weekly backup is only Saturday.
ec2-create-snapshot vol-xxxxxxxx -d "Daily Backup"
ec2-create-snapshot vol-xxxxxxxx -d "Weekly Backup"
Is this the correct syntax below should I use in ec2-prune-snapshots?
Thanks. James
I might have done it slightly differently – but I see no reason why your approach won’t work.
--dayofweek 6toec2-prune-snapshotsso that it knows to keep ‘Saturday’ as your weekly backup (and change –days to 7, but that is a preference not a requirement).ec2-prune-snapshots -–aws-access-key-id XXXXXXXXXX –-aws-secret-access-key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX –-region ap-southeast-1 –-days 7 –-weeks 4 --dayofweek 6 vol-xxxxxxxxTaking only daily snapshots, and using the above, should result in:
Correct me if I’m wrong. That command I assumed that daily snapshot will be kept for 7 days? If I’d like to kept it 8 days, so the right parameter is
--days 8?Thanks.
James
You are correct:
--days 7will keep 7 days worth of snapshots (plus the weekly snapshots). While it really doesn’t matter, keeping 7 ensures that there is only one snapshot for each day of the week at any given time. That said, if you are looking to keep 8 snapshots,--days 8is the way to go.Let say if I missed the backup on Day 6 due to power failure, is the cleaning up will still properly removed? Thanks!
The easy way to test this is to run the command you usually use with
--noaction. That way, you can see what files will be deleted, and if they are to your liking, you can go ahead with the deletion (optionally, run with--debugas well). The script calculates deletion based on the time of your snapshots – not based on when the script was last run. Any files that fall between certain time points and meet other, additional criteria, will be kept, while the others will be deleted. (So, if you keep daily snapshots for the last 10 days, then instead of keeping 10, you would expect the script to only keep 9 until 10 days have passed from your missed backup).a) The backup was all started on April 1 and no backup on March. Only April 7 was deleted on April 23.
b) Give me your email address and will send you the log file.
c) Melbourne/Australia timezone
So you are GMT+10? What time of day do you run the script at (e.g. I run mine at 3:27am GMT-4) (I ask, because the script works extensively with time calculations and differences, and I am not yet sure what is causing it to behave the way you described. It hasn’t yet deleted anything it shouldn’t have for me, so I wouldn’t have stumbled across this problem ordinarily.)
I have sent you an email to the email address you used for your comments.
Yes, I run it 11:30 PM everyday GMT+10
Excellent, got your email (thanks!) will take a look at as I get a chance over the next couple of days, and will let you know what conclusions I come to.
Edit: I see you have found the problem. Just to mention it here, for future visitors – after a few snapshots (50), additional snapshots are shown on a different page in the AWS console. You had me a bit concerned for a while there – disappearing snapshots is definitely a bad thing. Glad to know that no snapshots were harmed by the running of this script 🙂
Hi cyberx86,
I am back again.
I am using your script now but just now I realized that running this script in weekly, I found strange behavior. Take a look of this outcome I have from http://pastebin.com/YibbS0yB.
I try to run the script today (13 Sept. 2012) and the snapshot ID “snap-XXXXXX99” supposedly be deleted as it is already more than 4 weeks past. My parameter is set correctly “–weeks 4” and I don’t know why the script still KEPT it. Can you please help me to correct what’s wrong in my parameters or on the script?
Looking forward to hear from you.
James
Hi James, that behaviour is actually by design. The script will first keep the days requests, and then counts weeks from the last day (so you have 8 days + 4 weeks = a bit over 5 weeks). Essentially, the parameters specified say to keep 8 daily snapshots and an additional 4 weekly snapshots (on top of the daily snapshots). The script does not count the daily snapshots as part of the weekly ones. At the top of the script output (when run with
--debug) the script displays the date ranges it has calculated for each snapshot type (daily, weekly, etc) – you can use those to adjust the values you use if you don’t wish to keep so many snapshots.Hi cyberx86 – I followed your syntax to keep the backup snapshot for 8 days, 4 weeks and day of week is 6 (sat). When on 23 April 2012, I noticed that the weekly snapshot for 7th was deleted. I don’t know why so I changed the
--daysto 10 to kept 10 days and retain 4 weeks and--dayofweekis 6. Today, all my 3 weekly backup snapshot for 7th, 14th and 21st were deleted. Is there any wrong in my syntax below?I am thinking to separate the daily and weekly clean up. What would be syntax will looks like?
Thanks.
It is quite likely you have found a bug – the syntax that you have used looks fine. If you can give me some more information, I can hopefully figure out what is going on and fix the bug.
a) On Apr 23, 2012, were the (weekly) snapshots from Apr 14, and Mar 31 kept (and only the one in the middle, Apr 7 deleted)?
b) Could you give me a copy of the debug output, please (‘X’ out your volume/snapshot IDs – for the moment, I only care about the times).:
c) Also, what time of day (GMT) do you run the script at?
I am thinking that there may be some issue with the time calculation in specific combinations (since today would be 10 days + 1 from the last snapshot), but I can’t seem to pinpoint it at the moment.
As for separating the daily and weekly cleanup, it isn’t really possible – one version would end up deleting the snapshots kept by the other version. The script calculates the time ranges using all the smaller time increments (so, the limit for weekly backups depends on the number of hourly and daily backups that are kept).
I’m trying to get your script to work on Ubuntu. When I run the script I see this:
As you can see, every time I run the script it says that it is deleting the same snapshot, but the snapshot is not actually getting deleted. I see no errors so I’m not sure how to troubleshoot this. Do you have any ideas why the snapshot would not be deleted even though the script says it is?
Only one reason springs to mind as to why a snapshot wouldn’t be deleted, which is that your account has permissions to list snapshots but not delete them (e.g. if you are using IAM) – but even that should generate an error. I presume you are actually passing credentials to the program (and just removed them from the command above); also no dashes are needed before the volume-id (and the script does require at least v0.14 of the Net::Amazon::EC2 module). Add
--debugto the parameters, and see if the output is more helpful – the script will show its process in much more detail, and may shed some light on the problem.The actual deletion occurs at line 192, and should generate an error if it fails (unless, of course, you use
--noaction, in which case it won’t actually perform the deletion, but will instead display ‘SKIPPED’).if ($ec2->delete_snapshot(SnapshotId => $snap_to_del)){ $Quiet or print "Snapshot [", $snap_to_del, "] deleted\n"; }else{ warn "$Prog: Snapshot [", $snap_to_del, "] deletion FAILED\n"; }New insight:
The description of the snapshot that I was trying to delete above was “Created by CreateImage(i-1fa49e58) for ami-89cc95cc from vol-78e5081a”.
That was an automatically created snapshot by the system when you create an AMI. I guess your script, or the API for that matter, does not have permission to delete those snapshots.
I was able to successfully delete snapshots created with ec2-consistent-snapshot with your script. Thanks
Glad it worked out for you. Thanks for the info and follow-up – I never thought to try deleting a snapshot created as part of an AMI – only ever used the script for those created by ec2-consistent-snapshot, so definitely handy to know for future reference.
I try to run the script outside from the aws network and the OS is CentOS 6. And this is the error I received.
I input the correct AWS key and the region is correct which is ‘ap-souteasth-1’ but still getting this error “
Not an ARRAY reference at ./ec2-prune-snapshots line 131.“… How can I fix this problem?By the way, when I run in ec2 instance, it really works without issue. I am not sure what is the cause of my problem.
Thanks. James
Uncomment line 114 (
# ($Debug ? (debug => 1) : ()),) and add the following right after line 117if (ref $current_snapshots eq 'Net::Amazon::EC2::Errors') { print $current_snapshots->errors->[0]->message; }Run the script with (with the changes above) with
--debugand see if you get any additional information. I will see if I can reproduce the problem – but may not get a chance to try it for another half day or so.I added that line and uncomment the debug code. Now, I found the error message.
I installed the following packages and everything is working now.
Thanks man! 🙂
James
Great – glad it worked out for you. Both of those modules are dependencies of
ec2-consistent-snapshot– so I don’t list them anywhere on this page (although, the fact that this script has the same dependencies asec2-consistent-snapshotis mentioned in the Installation section).What is your script looks like for backing up EBS? I am only using a simple bash script below and run it on cron.
I was thinking of how to make the script that will send through email notification if the backup is successful or not even applying to
ec2-prune-snapshots. I believe that would be useful feature to add on this script.James
I can’t say I can ever recall having a snapshot fail for me (after initial testing). You can get cron to send you an email – usually it will do so if there is any output from the commands that are run.
I use
ec2-consistent-snapshotto generate my backups. The advantages of it, compared toec2-create-snapshot, are numerous – but briefly, it uses the API directly (instead of having to load Java) and it flushes the disk caches and freezes the file system before taking the snapshot (I use an XFS file system on my data volume).My crontab contains:
/root/backup.sh contains:
If either script errors out – the output will result in cron sending an email to the root user.
I try to have a logs for my crontab scripts with a simple code below but it doesn’t piped out the output to the log.
Do you know how to exactly pass the output of that command to the log file?
Thanks.
James
That is a more interesting question than it seems on the surface. The
2>&1 >/tmp/testlog.txtpart redirects STDERR(2) to STDOUT(1) and anything originally going to STDOUT gets redirected to /tmp/testlog.txt. Despite what it may seem, STDERR does not end up going to /tmp/testlog.txt since it is the stream (and not the device) that is repointed.The relevance of this requires a look at the code. When you run with
--noaction, the following gets executed:The Perl command
warnsends output to STDERR – not to STDOUT.When run for real, the following gets executed:
Essentially saying that if you don’t have the
--quietoption turned on,printthe message. Perl’sprintcommand sends output to STDOUT.The slight difference in output behaviour between these two is conserved from
ec2-consistent-snapshot, but does make some sense – running with noaction is expected to be atypical.The resolution is quite simple – first redirect to your file, then redirect the streams:
>/tmp/testlog.txt 2>&1. This will send STDOUT to your file, and STDERR to STDOUT (which is now going to the file) – the end result being that both STDOUT and STDERR end up in your file. Your command, therefore becomes:(As a point of mention, you could have tested this without cron. Running the command directly from the shell should have revealed that the redirection wasn’t working.)
I understand that this script will get all snapshots of a particular volume and do pruning. My question is, Is it possible to skip or to exclude a specific snapshot on deleting? Is there like an option like ‘
--exclude‘ ?? My main reason is that I don’t want to include the snapshot I’ve created for an ec2 instance from running this script.Thanks. James
Seems like exclusions could come in handy on occasion. I have updated the script to add the functionality. You can now specify one or more snapshots to exclude by passing
--exclude SNAPSHOT_ID. If you want to specify multiple snapshots to exclude, you must pass the--excludeparameter multiple times, for example:As with any update, run it with –noaction and –debug before trying the real thing. With –debug, the script will identify those snapshots you excluded with:
Let me know if it works, or if you find any problems with it.
I get:
[root@ip-10-28-95-138 ~]# ./ec2-prune-snapshots -h Can't use an undefined value as an ARRAY reference at /usr/lib/perl5/Class/MOP/Class.pm line 785. Compilation failed in require at /usr/lib/perl5/Moose/Exporter.pm line 11. BEGIN failed--compilation aborted at /usr/lib/perl5/Moose/Exporter.pm line 11. Compilation failed in require at /usr/lib/perl5/Moose.pm line 15. BEGIN failed--compilation aborted at /usr/lib/perl5/Moose.pm line 15. Compilation failed in require at /usr/local/share/perl/5.10.1/Net/Amazon/EC2.pm line 2. BEGIN failed--compilation aborted at /usr/local/share/perl/5.10.1/Net/Amazon/EC2.pm line 2. Compilation failed in require at ./ec2-prune-snapshots line 20. BEGIN failed--compilation aborted at ./ec2-prune-snapshots line 20.I’m not familiar with perl/perl modules, and unable to find much on the particular situation using Google. Got any ideas?
It may be related to the specific versions of the modules you are using. Do you use
ec2-consistent-snapshotas well (if so, does it work)? For me, I am using the following:Amazon’s Linux (CentOS/RHEL 6 derived): v2012.03
Perl (
perl -v): v5.10.1Net::Amazon::EC2 (
perl -MNet::Amazon::EC2 -le 'print $Net::Amazon::EC2::VERSION'): v0.14Moose (
perl -MMoose -le 'print $Moose::VERSION'): v1.15Class::MOP (
perl -MClass::MOP -le 'print $Class::MOP::VERSION'): v1.12See if you can update your modules to similar versions if they are not already up to date. If you are using a RHEL/CentOS derived operating system, all of these are available either through the main repositories or through EPEL, and can be installed with
yum.Sorry I didn’t give more details — I had worked 18 hrs the night before with 4 hours of sleep before working again. I’m running this on Debian Squeeze, installing perl modules with cpanm.
Oddly, trying to grab the versions of any of these modules is returning the same set of errors as above, and cpanm fails on an attempt reinstallation of any of them (the build.log once again shows the same compilation failures). Inexplicably, everything looks broken — I’ve only installed modules required for ec2-consistent-snapshot and ec2-prune-snapshots using very standard methods, retrieving everything from cpan.org. This is quite odd. I’ll update once it’s fixed.
It has been a while since I tried to install the dependencies for this script using CPAN. While CPAN does have the latest versions, it just doesn’t compare to the efficiency of a package manager. You may want to give
apt-geta try (now, I am not a Debian/Ubuntu person, so bear with me). From the list I previously provided, the following appear to be the Debian equivalents:MooseandClass::MOPare slightly older versions than I am using, but hopefully they will work without issue.In addition,
ec2-consistent-snapshot(from whichec2-prune-snapshotsis derived) has the following dependencies (the version is what is available on Debian; if you are not installingec2-consistent-snapshot, you can omitlibdbi-perlandlibdbd-mysql-perl):Alternatively, ec2-consistent-snapshot does come as a .deb file, which you may be able to install (see the comments on Alestic’s site)
In addition to the above,
ec2-prune-snapshotshas the following dependencies:While this script does not depend on
ec2-consistent-snapshot, it was intended to share the same basic parameters and conserve dependencies.I suppose I did treat cpanm like a package manager — when I discovered the lack of a “remove” command, I found out it didn’t really act like one. I did what you suggested here, and after installing the final two dependencies there with apt-get (which I had originally done with cpan), not only does the script work, but I can now run the command to do things like grab my Net::Amazon::EC2 version, which failed before on the same errors as the script did. I still have no idea what caused any of that to happen… but it’s fixed.
Great to hear. Glad to know it worked out for you.
Hello,
I am doing snapshots every hour and use ec2-prune-snapshots script for rotating it. I use this:
ec2-prune-snapshots --aws-credentials-file /root/aws_secret --region ${aws_region} --months 3 --weeks 4 --days 2 --hours 2Every day in week it works perfect, but on sunday it keeps snapshots for everyhour (whole day), I cant figure out why it keeps snapshots on sunday for every hour, not for last 2 hours?
As a test, add
--debugand--noaction(the latter, so that it doesn’t actually delete any snapshots, and you can run it by hand). So:ec2-prune-snapshots --aws-credentials-file /root/aws_secret --region ${aws_region} --months 3 --weeks 4 --days 2 --hours 2 --debug --noaction vol-xxxxxxxxI suspect the problem comes down to the script trying to keep a weekly snapshot (i.e. it has decided to keep a snapshot for Sunday as the weekly snapshot, but it didn’t settle on just one for some reason). Obviously, that is not the intended behaviour. If you can post the output of the debug command above, I should be able to figure out what is wrong.
Here is last hour log, now it kept snaphots of whole monday:
It looks like it is keeping all the hourly snapshots until it deals with the daily snapshot since it doesn’t know which hourly snapshot it will need. Did it delete the old snapshots (i.e. is it just keeping the one day’s worth of hourly snapshots and getting rid of the extra ones the following day or is it not deleting the hourly snapshots at all?). I take it that the expected behaviour would be to keep just the two snapshots you want, and then choose from those one to keep for the daily snapshot. I will take a look and see if I can get the code to behave more in the expected manner.
This looks great. Exactly what I was looking for. On a more general level I’m trying to wrap my head around Amazon’s snapshot methodology. My understanding is that snapshots are incremental. Therefore the first snapshot taken with be a full capture, while subsequent snapshots will only contain the changes to the volume that occurred since the last snapshot was taken. So if I have a week of backups and I want to delete the 3 oldest versions, wouldn’t I be obliterating the original snapshot — which contains the great majority of data (potentially)! Can I someone explain why I wouldn’t be screwed in the above scenario. Much thanks.
EBS snapshots are differential and compressed. Basically, Amazon creates a map of the used blocks and then associates the necessary blocks with a given snapshot. Deleting a snapshot will only delete the associated snapshot blocks if they are not used by any other snapshot. This means you can delete snapshots in any order, since they are not incremental in the traditional sense and do not depend on each other. I explain it in a bit more detail here.
I’m having the same problem as Jonas. It kept all 24 hours worth of snapshots for last Sunday. I take a snapshot every hour and run this right after taking the snapshot every hour:
ec2-prune-snapshots –region us-west-1 –hours 24 –days 3 –weeks 2 –months 1 –favour-recent 0 — vol-d11289b5
Here is the output running this with debug:
[ec2-user@ip-10-0-0-39 bin]$ ec2-prune-snapshots –region us-west-1 –hours 24 –days 3 –weeks 2 –months 1 –favour-recent 0 –debug –noaction — vol-d11289b5
ec2-prune-snapshots: Using AWS access key:
ec2-prune-snapshots: Tue May 29 11:01:50 2012: begin snapshot pruning
ec2-prune-snapshots: Endpoint: https://ec2.us-west-1.amazonaws.com
ec2-prune-snapshots: Vols: vol-d11289b5
ec2-prune-snapshots: Earliest for ‘hours’: 2012-05-28T17:01:50
ec2-prune-snapshots: Earliest for ‘days’: 2012-05-25T17:01:50
ec2-prune-snapshots: Earliest for ‘weeks’: 2012-05-11T17:01:50
ec2-prune-snapshots: Earliest for ‘months’: 2012-04-11T17:01:50
ec2-prune-snapshots: snap-c9333fad [vol-d11289b5; 2012-05-26T00:40:11] to be KEPT [day #2] (1)
ec2-prune-snapshots: snap-c5c6d3a1 [vol-d11289b5; 2012-05-27T00:40:11] to be KEPT [day #1] (1)
ec2-prune-snapshots: snap-418d9825 [vol-d11289b5; 2012-05-27T01:40:12] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-395a4e5d [vol-d11289b5; 2012-05-27T02:40:12] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-590f1b3d [vol-d11289b5; 2012-05-27T03:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-c5c8dca1 [vol-d11289b5; 2012-05-27T04:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-1d879379 [vol-d11289b5; 2012-05-27T05:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-a56770c1 [vol-d11289b5; 2012-05-27T06:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-45f4e321 [vol-d11289b5; 2012-05-27T07:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-b5495fd1 [vol-d11289b5; 2012-05-27T08:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-a1d9cfc5 [vol-d11289b5; 2012-05-27T09:40:12] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-f95d4c9d [vol-d11289b5; 2012-05-27T10:40:12] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-853223e1 [vol-d11289b5; 2012-05-27T11:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-81e8f9e5 [vol-d11289b5; 2012-05-27T12:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-ed746489 [vol-d11289b5; 2012-05-27T13:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-4908182d [vol-d11289b5; 2012-05-27T14:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-11ddcd75 [vol-d11289b5; 2012-05-27T15:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-f999899d [vol-d11289b5; 2012-05-27T16:40:12] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-71acbc15 [vol-d11289b5; 2012-05-27T17:40:12] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-81697ae5 [vol-d11289b5; 2012-05-27T18:40:12] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-05d1c261 [vol-d11289b5; 2012-05-27T19:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-a59487c1 [vol-d11289b5; 2012-05-27T20:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-3d554759 [vol-d11289b5; 2012-05-27T21:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-311e0c55 [vol-d11289b5; 2012-05-27T22:40:11] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-8dd7c5e9 [vol-d11289b5; 2012-05-27T23:40:12] to be KEPT (for future) [day #1] (4)
ec2-prune-snapshots: snap-ed9e8c89 [vol-d11289b5; 2012-05-28T00:40:11] to be KEPT [day #0] (1)
ec2-prune-snapshots: snap-91f2eaf5 [vol-d11289b5; 2012-05-28T17:40:12] to be KEPT [hour #23] (1)
ec2-prune-snapshots: snap-bdbca4d9 [vol-d11289b5; 2012-05-28T18:40:12] to be KEPT [hour #22] (1)
ec2-prune-snapshots: snap-a17e65c5 [vol-d11289b5; 2012-05-28T19:40:12] to be KEPT [hour #21] (1)
ec2-prune-snapshots: snap-993e25fd [vol-d11289b5; 2012-05-28T20:40:11] to be KEPT [hour #20] (1)
ec2-prune-snapshots: snap-49c1da2d [vol-d11289b5; 2012-05-28T21:40:11] to be KEPT [hour #19] (1)
ec2-prune-snapshots: snap-89948fed [vol-d11289b5; 2012-05-28T22:40:12] to be KEPT [hour #18] (1)
ec2-prune-snapshots: snap-515a4035 [vol-d11289b5; 2012-05-28T23:40:12] to be KEPT [hour #17] (1)
ec2-prune-snapshots: snap-551a0031 [vol-d11289b5; 2012-05-29T00:40:12] to be KEPT [hour #16] (1)
ec2-prune-snapshots: snap-e9dec48d [vol-d11289b5; 2012-05-29T01:40:12] to be KEPT [hour #15] (1)
ec2-prune-snapshots: snap-71e1fb15 [vol-d11289b5; 2012-05-29T02:40:12] to be KEPT [hour #14] (1)
ec2-prune-snapshots: snap-36ae4b51 [vol-d11289b5; 2012-05-29T03:40:11] to be KEPT [hour #13] (1)
ec2-prune-snapshots: snap-76f01511 [vol-d11289b5; 2012-05-29T04:40:12] to be KEPT [hour #12] (1)
ec2-prune-snapshots: snap-8a29cced [vol-d11289b5; 2012-05-29T05:40:12] to be KEPT [hour #11] (1)
ec2-prune-snapshots: snap-9248adf5 [vol-d11289b5; 2012-05-29T06:40:12] to be KEPT [hour #10] (1)
ec2-prune-snapshots: snap-0ecb2f69 [vol-d11289b5; 2012-05-29T07:40:12] to be KEPT [hour #9] (1)
ec2-prune-snapshots: snap-964fabf1 [vol-d11289b5; 2012-05-29T08:40:12] to be KEPT [hour #8] (1)
ec2-prune-snapshots: snap-c2f81fa5 [vol-d11289b5; 2012-05-29T09:40:12] to be KEPT [hour #7] (1)
ec2-prune-snapshots: snap-5e49ae39 [vol-d11289b5; 2012-05-29T10:40:12] to be KEPT [hour #6] (1)
ec2-prune-snapshots: snap-66ef0901 [vol-d11289b5; 2012-05-29T11:40:12] to be KEPT [hour #5] (1)
ec2-prune-snapshots: snap-2602e441 [vol-d11289b5; 2012-05-29T12:40:11] to be KEPT [hour #4] (1)
ec2-prune-snapshots: snap-f6a94891 [vol-d11289b5; 2012-05-29T13:40:12] to be KEPT [hour #3] (1)
ec2-prune-snapshots: snap-869e7fe1 [vol-d11289b5; 2012-05-29T14:40:12] to be KEPT [hour #2] (1)
ec2-prune-snapshots: snap-7225c415 [vol-d11289b5; 2012-05-29T15:40:12] to be KEPT [hour #1] (1)
ec2-prune-snapshots: snap-de6786b9 [vol-d11289b5; 2012-05-29T16:40:12] to be KEPT [hour #0] (1)
ec2-prune-snapshots: Snapshot [snap-de6786b9] Tue May 29, 2012 16:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-7225c415] Tue May 29, 2012 15:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-9248adf5] Tue May 29, 2012 06:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-8a29cced] Tue May 29, 2012 05:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-76f01511] Tue May 29, 2012 04:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-36ae4b51] Tue May 29, 2012 03:40:11 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-71e1fb15] Tue May 29, 2012 02:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-e9dec48d] Tue May 29, 2012 01:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-551a0031] Tue May 29, 2012 00:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-515a4035] Mon May 28, 2012 23:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-89948fed] Mon May 28, 2012 22:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-49c1da2d] Mon May 28, 2012 21:40:11 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-869e7fe1] Tue May 29, 2012 14:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-993e25fd] Mon May 28, 2012 20:40:11 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-a17e65c5] Mon May 28, 2012 19:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-bdbca4d9] Mon May 28, 2012 18:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-91f2eaf5] Mon May 28, 2012 17:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-f6a94891] Tue May 29, 2012 13:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-2602e441] Tue May 29, 2012 12:40:11 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-66ef0901] Tue May 29, 2012 11:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-5e49ae39] Tue May 29, 2012 10:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-c2f81fa5] Tue May 29, 2012 09:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-964fabf1] Tue May 29, 2012 08:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-0ecb2f69] Tue May 29, 2012 07:40:12 to be KEPT (hour)
ec2-prune-snapshots: Snapshot [snap-ed9e8c89] Mon May 28, 2012 00:40:11 to be KEPT (day)
ec2-prune-snapshots: Snapshot [snap-c5c6d3a1] Sun May 27, 2012 00:40:11 to be KEPT (day)
ec2-prune-snapshots: Snapshot [snap-c9333fad] Sat May 26, 2012 00:40:11 to be KEPT (day)
ec2-prune-snapshots: Tue May 29 11:01:51 2012: done
You can see that for last Sunday it’s kept all 24 of the hourly snapshots “for future”. I’m assuming this is for the weekly snapshot that it is anticipating it needs, but there is no reason it should have kept all 24 snapshots for this. What I would expect would happen is that it would just keep the midnight snapshot for Sunday and would have deleted all the rest. Then once that midnight snapshot is older than 3 days, instead of being deleted it would be kept “for future” for the weekly backup.
Thank you so much for all your hard work in creating this script! I really love the methodology behind it and I would love to use this in a production environment. Please, please fix this bug (as I perceive it) and this script will be perfect. Let me know if I can do anything to help debug.
Glad you find the script useful. The behaviour you describe is definitely not the intended behaviour of the script. I will add it to my todo list – but it is going to be a couple of weeks at least, until I get a chance to spend time on this script.
Also experiencing the problem of keeping too many hourlies, look forward to a fix. Thanks very much for the super useful script.
Also, I added an option to check the description of the snapshot vs a regex – I tag all my automated snapshots with a certain description, and only want to automatically purge the ones that match the description. (in case I want to keep a particular snapshot around longer, for whatever reason). If you’re interested I can send you the diff.
I definitely haven’t forgotten about this script – just been really busy off late. Will hopefully be able to fix the bug in early/mid-July when my schedule frees up a bit. Glad you found it useful. I’d be happy to add in a the functionality you describe and credit you with its implementation.
Hi,
I am receiving this error if I run the script from the archive file:
If I am running the script from this page is working. Am I doing something wrong?
Thank you!
Line 128 reads:
The error would suggest that somehow there was a space added between
$minand_months. Looking at the file however, this isn’t true (actually, there are no instance of$minon its own in the file at all). Moreover, if I download the file from the site and try to run it, it functions without error.I’d suggest checking the MD5 checksum, and open the file after decompressing it and check that line 128 matches the one you see on the site.
Hi,
Seems that my version was wrong. I downloaded again and now is working.
Thank you!
Best regards.
Hi,
Have you used this script for snapshots that were created with EBS RAID option? If I have an array of snapshots it will know about the snapshots’ dependencies?
It is possible now to keep one snapshot for last 2 months and that snapshot to be the one from the first Sunday of the month?
Thank you!
Best regards.
I used to use RAID on some of the first EC2 instances I ran, but it has been a few years since I have done so. The script does NOT know anything about the dependencies or content of the snapshots – it only looks at the timestamps and volume-ids. If, however, you take your snapshots together, then simply based on time, matching snapshots should be kept. I am not sure exactly what you are looking to do, but if you take your snapshots on the first Sunday of the month, you can tell the script to keep the past 2 months worth of snapshots.
Hey, I just created a fork of ec2-rotate-volume-snapshot that supports raid grouping: https://github.com/bluestatedigital/ec2-rotate-volume-snapshots/tree/feature-raidgrouping
Check it out!
Looks nice. (Although, one of my objectives with this script was for it to be in Perl so that it shared dependencies with ec2-consistent-snapshot). Thanks for commenting.
Just wondering if there has been any progress on fixing that bug. Thanks!
Hi,
First off, this is an awesome script! Thank you for your hard work because it definitely makes life simpler. I am trying to deploy this on a 32 bit version of Amazon Linux and am running into this error.
Net::Amazon::EC2::Errors=HASH
Any chance you could lend hand in helping me figure this out? It worked flawlessly on the 64 bit version of Amazon Linux.
Thanks,
EDIT: The full error is this: Net::Amazon::EC2::Errors=HASH(0x99b9a50)ec2-prune-snapshots: Wed Sep 5 21:58:55 2012: done
Glad you find the script useful. I would, unfortunately, need a few additional bits of information to offer any suggestions beyond speculation. If you could run the script with –debug you might find a few more clues (if you end up posting some of that output here, please obscure any personal IDs that are included).
The error message unfortunately does not say much in its current form – basically that there are errors and they are stored in an array (hopefully –debug will shed some light on things).
I am guessing the problem lies in the version of perl or more likely the versions of certain modules being used.
The script should definitely work on a 32 bit version of Amazon Linux – I use it on one (I have since I wrote the script. The following versions are what I use:
Amazon’s Linux: v2012.03
Perl (
perl -v): v5.10.1Net::Amazon::EC2 (
perl -MNet::Amazon::EC2 -le 'print $Net::Amazon::EC2::VERSION'): v0.14Moose (
perl -MMoose -le 'print $Moose::VERSION'): v1.15Class::MOP (
perl -MClass::MOP -le 'print $Class::MOP::VERSION'): v1.12You definitely need at least version 0.14 of Net::Amazon::EC2.
If using –debug with the script, as is, doesn’t reveal anything, do two things:
Uncomment line 118 (
# ($Debug ? (debug => 1) : ()),) from the scriptAdd the following at line 120:
if (ref $current_snapshots eq 'Net::Amazon::EC2::Errors') { print $current_snapshots->errors->[0]->message; }Hope it works for you.
Thanks for the reply. Here’s what i got:
Amazon Linux AMI release 2010.11.1 (beta)
NET::AMAZON::EC2: v0.20
Perl: perl, v5.10.1 (*) built for i386-linux-thread-multi
Moose: 2.0603
Class::MOP: 2.0603
I tried your instructions and still got the same result. I actually got the error originally posted from –debug. This time when i ran it, i got the same thing even with the script changes. Any ideas?
Wow Im an idiot… i tried this again and rechecked everything. Turns out I’ve been using the wrong AWS Key File.. EPIC FAIL
No problem – glad you found the issue. I can reproduce it (using an incorrect key) – however, commenting out the debug line (line 116) does appear to display a relevant error message for me:
I really should see about updating this script with some error checking…
Interesting. I was able to run the script under Windows, with the following, without any issue:
Perl: 5.16.1
Net::Amazon::EC2: 0.20
Moose: 2.0603
Class::MOP: 2.0603
Therefore, it would seems that the versions you are running, may not be the issue. Is any other output shown when you run with –debug? (I ask because although the output itself is not relevant, the location in the script would be relevant for debugging).
I do have a different question though… is “–year” a valid function?
No – I never wrote a year function into the script. When I wrote it initially I didn’t really expect it to get much use (so it was largely based on my needs – I take daily snapshots, and prune them down to monthly snapshots which I keep for a few months). A few other bits of functionality were tacked on later (and I believe there are still some issues with the handling of hourly snapshots, which I just haven’t gotten around to looking into). No one has ever expressed an interest in yearly snapshots before (and admittedly, such a scenario would be rather difficult to test). Put another way, it probably isn’t too difficult to implement (although I can’t make any promises as to when I will get around to it), but I would have no way to test that such a function does what it is supposed to.
Awesome!! No worries at all! I was really more so curious to know if i was just messing up the syntax when i entered it in. The person im setting this up for was interested in keeping yearly snapshots so when i tried the function it failed…for the obvious reason that it wasnt meant to work that way in the first place :). Thank you though, and dont worry about modifying the script for yearly. I’m just glad i can give him an answer stating that it just isnt a possibility right now.
Thanks again for all your help!
Well Im back lol… This time a totally different issue! So everything works really well, except, when i try to parse snapshots after an AMI has been created for an instance. When the AMI is created it also creates snapshots which, unfortunately, use the same Volume ID’s as the machines im trying to parse. When the AMI creates a snapshot, the snapshot cannot be deleted until the AMI is de-registered. Do you now of any way around this?
Can’t get it to run. After getting past several errors by apt-get’ing 4 packages (
libdatetime-perl,libdatetime-format-dateparse-perl,libfile-slurp-perl,libnet-amazon-s3-tools-perl) it still has an error I don’t know how to get past:Sorry for the delay in replying, I seem to have missed the notification for your comment. By the looks of it, you are missing Net::Amazon::EC2 (you need at least v0.14 of it). You do not need the S3 tools. As far as I can recall, the complete list of dependencies for this script is:
DateTime,DateTime::Format::DateParse,Net::Amazon::EC2,File::Slurp,Net::SSLeay,IO::Socket::SSL,Time::HiRes,Params::Validate,Mozilla::CA(This is basically the list of dependencies for
ec2-consistent-snapshot, with the database specific requirements removed).Hi cyberx86,
Sorry, I know that my question is not related on this script but I just want to know what could you recommend a script to backup Windows server?
I tried to use the normal Amazon API command line stated below and it’s working fine. However, when I restore the snapshot file, it didn’t work correctly. The only thing it works is creating a AMI and restoring the instance is working well.
/opt/aws/bin/ec2-create-snapshot vol-xxxxxxxx -d "System Drive"I would appreciate any recommendation for backing up Windows instance.
James
I have no experience with Windows servers, so, unfortunately, cannot provide any meaningful suggestions on this front. The AMI though stores more information than the snapshot, including the type of virtualization and other ‘machine’ data that is necessary to successfully launch an instance from an image. You may be able to pass some of these parameters as part of the launch command (although, again, not having worked with Windows instances, I can’t really point you in the right direction here). If you don’t manage to find a solution, I would suggest asking on ServerFault (although, would advise against directly asking for a script recommendation, as ServerFault would consider that a ‘shopping’ question).
Thanks! I figured it out to use ec2-create-image tool to backup the Windows instance. However, another challenging part is on how to automate the process to cleanup all created AMI’s. It would be great too if you could create a similar script like this ec2-prune-snapshots that will do the same functions for AMI. The function would be like this:
(1) de-registering the AMI .
(2) snapshots file will be deleted.
Thanks
Glad it worked out for you. I appreciate your interest, however, I will say that it is unlikely I make such a script (at least in part, because it should already exist). I believe that hybridfox can do this (although I haven’t tried it), if you want to just go through the images manually (just select the AMI, and tell it to deregister the image and delete the snapshot). There is also a Ruby script that appears to do this on GitHub. Depending on your needs, I am sure there are many more similar scripts out there. Good luck.
Is there a possible way to get the script to ignore all snapshots that have an AMI attached to it?
You can use the
--excludeparameter to specify any snapshots that you want omitted from the pruning. (If you need to get a list of the snapshots tied to AMIs, you can get the list from the block device mappings of all your AMIs).Nice thanks! Ill give that a shot
Not to clutter up your comments, but just wanted to say: this is a great script, and it saved me a great deal of frustration and delay. Thanks for your work!
Always glad to know someone finds it useful. Thanks for commenting.
Hi – Love the script! I implemented it last fall and it was working fabulously, pruning well over 6000 snapshots to just a few hundred with the Months, Week, Days, and Hours I wanted to keep (saved us a decent chunk of $$ on snapshot storage 🙂
However, I just noticed that there has been some excess pruning. I’m not sure if it started with the new year or before and unfortunately I wasn’t logging any results (that’s a task for tomorrow). These are the parameters I was using (repeated for all servers and volumes):
all I have as of today (4/1) is 2/1, 3/1, and the correct Weekly and Daily snapshots for a total of 17 snapshots per volume. Any idea why the script might have pruned all the prior Months?
Thanks so much for sharing this script and thank you in advance for any advice you can offer.
Cheers!
Bryan
Glad you found it useful; sorry to hear it deleted snapshots it shouldn’t have (while there have been some instances of extra snapshots kept, I think all the issues so far with erroneously deleted snapshots were resolved – so hopefully that happens here too). Snapshot storage definitely adds up. I would really love it if snapshots could be moved to Glacier.
You can try running the script with
--debug(and add--noactionfor safety), which should display the computed time-frames. A quick look at the output suggests that the time calculations are correct. I get the following:I normally run the script with
--days 7 --weeks 4 --months 3(and to be honest, it has become more of a set it and forget it script for me – I don’t check my snapshots too often). Looking at the snapshots that have been kept, I current have the following months: 2012/12/1, 2013/01/1, 2013/02/1, 2013/03/1 (the extra snapshot was kept since it will be needed as a monthly snapshot) – the other time frames are as expected.I will certainly add looking over this script to my todo list, however – it might not happen for a week or two (time is a luxury I don’t seem to have at the moment). If you have some debug data you can share, I will certainly look it over and see if I can spot anything out of the ordinary.
Great script – I suspect an upgrade has broken it though:
Any ideas ?
Much appreciated
Not sure what update could have caused that. The script continues to run fine for me (running on a fully up to date Amazon Linux).
Some starting points would be to check the versions of (and the versions I have installed)
You can find the version of a Perl Module with:
perl -MMODULE::NAME -e 'print "$MODULE::NAME::VERSION\n"'For example:
perl -MHTTP::Request::Common -e 'print "$HTTP::Request::Common::VERSION\n"'Working – thanx for the quick response – HTTP::Message HTTP::Request both needed upgrading !
Great to hear you got it fixed.
Hi,
I am unable to setup as per the instruction. I am getting error all the time when I tried to troubleshoot.
Can any one give me the step by step installation guide to install in Amazon Linux AMI.
Sorry to hear that. I run it on Amazon Linux without issue. What type of errors are you getting? Usually, most of the errors people encounter with the installation are a result out outdated packages – ensure that your packages are installed from EPEL/CPAN and meet or exceed the version numbers discussed in the comments.
Thanks for the quick response… I have followed the below steps… but I get error as follows…
Step 1: Created Amazon AMI Linux 64Bit Micro Instance.
Step 2: Reset Password for root user
Step 3: yum –enablerepo=epel update
Step 4: yum install perl-DateTime perl-DateTime-Format-DateParse -y
Step 5: find / -name EC2.pm (NO EC2.pm found)
Step 6: Unable to follow the link http://www.thatsgeeky.com/2011/06/rotating-ebs-snapshots-ec2-prune-snapshots/#comment-695
Step 7: When executing “perl -MNet::Amazon::EC2 -e ‘print “$Net::Amazon::EC2::VERSION\n”‘” below is the error.
[root@ip-10-10-100-28 ec2-user]# perl -MNet::Amazon::EC2 -le ‘print $Net::Amazon::EC2::VERSION’
Can’t locate Net/Amazon/EC2.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).
I have struck here… Please help to proceed.
a) This script was intended for use with
ec2-consistent-snapshotand therefore shares the dependencies of that script. You need to install the pre-requisites for that script for this one to work:b) According to your steps, you haven’t installed the
Net::Amazon::EC2package – it either needs to be installed from EPEL (via yum) or via CPAN (as per the point above). The link listed above (step 6), requiresEC2.pmto exist (which it only does once installed)Hi, This worked great.. now I could able to execute the script… but now I am confused how this works in my environment.
I have mixed versions of OS like Linux & Windows. I need to take backup of each server snapshot with 7 days, 4 week & 12 months. Many Windows servers have additional EBS volumes… How could be this possible ? Please guide. I am new to linux & aws…
Can’t help you with Windows – try ServerFault. This script does NOT take snapshots, its purpose is to delete snapshots based on the specified patterns. It can run from any machine (doesn’t need to be an EC2 instance), and will delete snapshots based on an EBS volume ID. Essentially, you need to take daily snapshots, using a method of your choosing (I like ec2-consistent-snapshot on an XFS volume in Linux), and then pass the number of daily/weekly/monthly snapshots to keep to this script for it to remove the other snapshots.
I’m getting an error referenced earlier in the comments, though the solution there isn’t my issue exactly. Here’s the error message I’m getting (with some debug info included):
I have ensured that
Net::Amazon::EC2is up-to-date (version 0.23). Do you happen to have any ideas on how I can narrow this down? Thanks in advance!I don’t suppose there is a bit more to that error message (like a line number, etc.). Unfortunately, that isn’t much to go by, so I am guessing here – it can read the snapshot information, but not delete them – might be an IAM permissions issue. If you can tell me a bit more about the problem, I might be able to offer some additional suggestions.
If a snapshot is in use – then the script stops with an error . There is class for describing tags – but how do I add a subroutine to test the status of a snapshot to test its not in use ?
TAG snapshot snap-e433bxxx Name In Use
TAG snapshot snap-f132bxxx Name In Use
ec2-prune-snapshots: Deleting snaps for vol-xxxxxxxx:
Amazon EC2 Errors [Request 278e3831-0ced-43d6-b9df-2fbba8e26c52]:
[InvalidSnapshot.InUse] The snapshot snap-f132bxxx is currently in use by ami-a31xxxx
I don’t believe the net::Amazon::EC2 module supports that at this time. What could be done is to handle the error more gracefully.
Hi
it’s been a long time since this was published.
Is it still the method that you use ?
Have you considered upgrading it to use instance “role” making AWS keys unnecessary ?
I should definitely update it – only about 10 lines are needed to add in the IAM functionality.
As far as I can tell this script is doing nothing once installed:
(that should have definitely kicked off a delete). also i get no output no matter what i put in with –noaction ..e.g.
ubuntu@ip-172-31-41-23:~$ ./ec2-prune-snapshots --region us-east-1 --aws-access-key-id AKIAJBGEG2UJIRW7VURQ --aws-secret-access-key zSUSwRIktM1M1waZJvr0ovyf97gx0gCKH1bCbYP2 --days 1 --noaction lfkdjsalfkjasdlfkjasdubuntu@ip-172-31-41-23:~$
The script will balk if I remove the region or aws keys etc but it doesn’t seem to actually do anything.
Do I *have* to use ec2-consistent-snapshot for your script to work? Are there dependencies I’m missing? I’m getting no output…
I use the EC2 CLI tools often, not sure what I’m missing here with your script..
Thank You
This script expects a volume id, not a snapshot ID (it is looking any snapshots created for a given volume – it does not accept specific snapshots).
You do not need to have ec2-consistent-snapshot installed (it does share many of the same dependencies though)
a) –noaction should be used while testing – it simply will inform of you what would be done, but will not actually do it
b) use –debug to get more verbose output
c) use –help to get a full list of parameters and their meaning
thanks- a little more using of my brain actually concluded that 🙂 thank you, I ran it and it worked perfectly. you really saved me a lot of bash scripting so truly, thank you.
p.s. it there a way to get the output to a file? i tried redirecting it to a file e.g. 2>&1 foo with no luck…i tried wrapping it in a bash script as well, still no luck.
Glad it worked out for you. It should redirect to a file without much effort:
If it doesn’t, it isn’t immediately obvious why not.
Hey- Was wondering if it would be relatively easy to use this to rotate snapshots that don’t have attached volumes. E.G. Copying snapshots over to another region and then rotating them out in that region. I know little about Perl but if it’s a matter of changing a few lines I could totally do that..Suggestions, advice? Thank you, this script is fantastic.
This script essentially takes all your snapshots (as returned by describe-snapshots) and then filters the list to decide which ones to delete. The filtering is currently done on status (i.e. only completed snapshots), volume, and time (e.g. volume filtering is done around line 139:
!grep $_ eq $snapshot->volume_id, @$volume_ids). You could add another filter parameter fairly easily, the issue is that there is a limited set of parameters available through the underlying library (Net::Amazon::EC2). You can see a list of returned parameters on CPAN. Unfortunately, this list doesn’t include tags. You could implement a variation which entirely ignores the volume (and just prunes everything), or one that uses the description field. There is a more recent version of Net::Amazon::EC2 that has some support for tags, but I haven’t looked into it enough to give a definitive answer on that front.As for copying a snapshot to another region, Net::Amazon::EC2 doesn’t provide a copy_snapshot (or equivalent) function. You would need to script it using either the full Perl API or using the console tools (or do it manually). If the volume_id is preserved in such a copy (haven’t checked if it is), then you could use the script without modification (although, I don’t believe Net::Amazon::EC2 provides a region descriptor on the snapshots either).
Cool thanks…The volume is not moved over to another region (it shows the old volume in the new region but it’s empty when you click on it it’s empty; you only *asked* to copy the snapshot)…And hence that is why I am having a problem rotating snapshots after using the aws command ec2 copy snapshots- There’s no volume to grab snapshots from 🙁 I wrote a quick bash script that deletes every 8th day but it’s far from as elegant as yours- it just deletes everything that is dated 8 days ago… My scripting sucks and my available time to learn it better doesn’t exist lately so there you go 🙂
Without a VolumeID, there is little to filter on. If you just want to use this script as an enhancement to your bash script, you can remove the lines that filter on VolumeID. If the script runs (which will somewhat depend on whether the VolumeID is just blank (e.g. “”) or if the field doesn’t exist), it will probably group all the blank VolumeIDs together and then it will rotate the entire set together. If you only have a single volume that you are doing this for, it should be fine, if you have multiple volumes, then the behaviour will probably be less than ideal. The volume filtering occurs around line 139:
}elsif(!grep $_ eq $snapshot->volume_id, @$volume_ids){ #snapshots for volumes not specified $Debug and warn "$Prog: Snapshot [", $snapshot->snapshot_id, "] skipped; volume [",$snapshot->volume_id,"] not in list", "\n";Of course, if you make any changes, run the script with –noaction and –debug to see what it will do.
Hi. I am running into this error:
It’s preventing a whole bunch of snapshots from being deleted. I’m running this script fine in other near-identical environments fine, but I have one where this problem is prevalent. How can I forced it to delete snapshots ?
Thank you!
You can’t delete a snapshot that is in use. If you look at the EC2 documentation, it reads: “Note that you can’t delete a snapshot of the root device of an EBS volume used by a registered AMI. You must first deregister the AMI before you can delete the snapshot. For more information, see Deregistering Your AMI.
What you can do is to exclude that snapshot (
--exclude snap-e8efb65d) – after which everything should operate as normal.