Terminating a Staging Environment – Customizing AWS Elastic Beanstalk

Let us discuss the script staging-terminate.sh which terminates the staging environment. The important point about the script is that out of all the available AWS Elastic Beanstalk environments, it correctly identifies and terminates the intended environment and another main thing is that these scripts could be executed from any EC2 instance and not necessarily from the instance which we want to terminate. In other words we could be using an EC2 instance belonging to a production or development environment and still execute the command to terminate the staging environment.

I have got one bad news though. This script works only if you are making use of Route53. At hudku.com we make use of Route53 and if an environment has been created properly then we expect few entries to have been made correctly in the Route53 DNS records. Then for any action on that environment, we depend on its entries in Route53 record-set as the authority and expect those entries to be accurate so that they can be relied upon.

Even if you are not using Route53 or have not fully configured all the settings you are supposed to make, we still recommend you to open a AWS Route53 account as it is not going to cost you much and take advantage of all the automation advantage our scripts provide.

For querying Route53 records we make use of the command line tool dnscurl.pl. I found a script route53DynDNS.bash in one of the public git repositories that illustrates excellent usage of this tool which we have made use while writing our scripts.

The script obtains the resource record sets from the Route53 account and obtains the value of the AliasTarget record which is supposed to contain the Load Balancer URL of the environment. Using the ELB URL, the beanstalk environment name is obtained and the name is checked whether it adheres to the format used during the creation of a new environment.

We use elastic-beanstalk-terminate-environment, the AWS Elastic Beanstalk command line utility and provide it the environment name to terminate the environment. If not using Route53, then if you can ensure by whatever means, the environment name you want to terminate is indeed the right one then you could directly use the command line utility without depending on our script.

Next the script proceeds further to remove the Route53 entries pertaining to the environment that we just terminated. The Route53 entries could be for example, myAppName-staging.elasticbeanstalk.com and ec2.myAppName-staging.myDomainName.com. The script is also capable of terminating an entry with www prefix for example www.myAppName-staging.elasticbeanstalk.com. At hudku.com we do not have that entry and the code is just there for historical reasons. The important point about deleting the Route53 entries is that all the entries are deleted as part of a single transaction. That means we just make one request to delete all the Route53 entries.

Here is the source code of staging-terminate.sh.

#!/bin/bash


# Execute "export DEBUG=1" to debug this script.
# Set value to 2 to debug this script and the scripts called within this script.
# Set value to 3,4,5 and so on to increase the nesting level of the scripts to be debugged.
[[ $DEBUG -gt 0 ]] && set -x; export DEBUG=$(($DEBUG - 1))


#
# Terminates the Elastic Beanstalk Staging environment
#


# include all the utility scripts
source $ELASTICBEANSTALK_APP_SCRIPT_DIR/include/include.sh


display_usage()
{
    echo -e "\nUsage: $0 [route53_RR_Name]\n"
}

# Exactly one argument should be supplied
if ([ $# -gt 1 ]) then
    display_usage
    exit 2
fi


# Terminate staging entry unless another entry is supplied
stagingDNSEntryName=$ROUTE53_RR_STAGING_NAME
if ([ ! -z $1 ]) then
    stagingDNSEntryName=$1
fi


# Delete www entry also
#prefixName="www"

awsAccountKeyName=$AWS_ACCOUNT_KEY_NAME

dnsEntryName=$stagingDNSEntryName


# extract host name from the dns entry name
hostName=$(echo $dnsEntryName | grep -o "[^\.]*\.[^\.]*\.$")


# dnscurl common options
optionsDNSCurl="-- -s -H \"Content-Type: text/xml; charset=UTF-8\""

# Route53 API settings
urlRoute53API="https://route53.amazonaws.com/2012-02-29"


# Get Route53 HostedZoneId
allHostedZones=$(dnscurl.pl --keyname $awsAccountKeyName $optionsDNSCurl -X GET $urlRoute53API/hostedzone 2>/dev/null)
hostNameSearch="ListHostedZonesResponse/HostedZones/HostedZone[Name=\"$hostName\"]/Id"
hostedZoneId=$(echo $allHostedZones | xpath $hostNameSearch 2>/dev/null | awk -F'[<|>]' '/Id/{print $3}' | cut -d/ -f3)
if ([ -z $hostedZoneId ]) then
    echo "Error: Failed to obtain hosted zone id for '$hostName' in Route53"
    exit 1
fi

# Obtain all the resource record sets for the hosted zone id
allRecords=$(dnscurl.pl --keyname $awsAccountKeyName $optionsDNSCurl -X GET $urlRoute53API/hostedzone/$hostedZoneId/rrset 2>/dev/null)

# Obtain the DNS record
dnsRecordSearch="ListResourceRecordSetsResponse/ResourceRecordSets/ResourceRecordSet[Name=\"$dnsEntryName\"]"
dnsRecord=$(echo $allRecords | xpath $dnsRecordSearch 2>/dev/null)

# Get ec2 CNAME record
dnsEC2RecordSearch="ListResourceRecordSetsResponse/ResourceRecordSets/ResourceRecordSet[Name=\"ec2.$dnsEntryName\"]"
dnsEC2Record=$(echo $allRecords | xpath $dnsEC2RecordSearch 2>/dev/null)

# Obtain elb URL of the staging environment
dnsNameSearch="ListResourceRecordSetsResponse/ResourceRecordSets/ResourceRecordSet[Name=\"$dnsEntryName\"]/AliasTarget/DNSName"
dnsName=$(echo $allRecords | xpath $dnsNameSearch 2>/dev/null | awk -F'[<|>]' '/DNSName/{print $3}' | cut -d/ -f3 | sed 's/.$//')

# Using the elb URL obtain the beanstalk environment name
envName=$(elastic-beanstalk-describe-environments -j | grep -io "\"EndpointURL\":\"$dnsName\",\"EnvironmentId\":\"[^\"]*\",\"EnvironmentName\":\"[^\"]*\"" | cut -d, -f3 | cut -d: -f2 | sed s/\"//g)

# Check envName format
testEnvName=$(echo $envName | grep "^env-[0-9]\{8\}-[0-9]\{4\}$")
if ([ -z $testEnvName ]) then
    echo "Environment Name is '$envName' whose format is unknown. Not trying to terminate the environment."
    exit 1
fi


# Just check to ensure that we are not trying to terminate the production environment
ipProduction=$(getIPOfURL "$ROUTE53_RR_PRODUCTION_NAME")
ipCurELB=$(getIPOfURL "$dnsName")
if ([ ! -z "$ipProduction" ] && [ ! -z "$ipCurELB" ]) then
    if ([ "$ipProduction" == "$ipCurELB" ]) then
        echo "Error: Attempt to terminate the production environment running at $ROUTE53_RR_PRODUCTION_NAME."
        exit 1
    fi
fi  


echo "Terminating the environment $envName"
elastic-beanstalk-terminate-environment -e $envName


dnsRecord=$(echo "<Change><Action>DELETE</Action>$dnsRecord</Change>")
dnsEC2Record=$(echo "<Change><Action>DELETE</Action>$dnsEC2Record</Change>")

# check if www prefix also has to be processed
if ([ ! -z $prefixName ]) then
    dnsPrefixRecord=$(echo $dnsRecord | sed "s|<Name>$dnsEntryName</Name>|<Name>www.$dnsEntryName</Name>|g")
fi


# Generate a timestamp to mark the Route53 transaction
timestamp=$(date)

# Create a temporary XML file
xmlTmp=$(mktemp)

# Set up the xml to delete the DNS entries
echo "Deleting Alias records of $dnsEntryName"
cat <<ROUTE53-XML > $xmlTmp
<?xml version="1.0" encoding="UTF-8"?>
<ChangeResourceRecordSetsRequest xmlns="$urlRoute53APIDoc">
    <ChangeBatch>
        <Comment>Deleting Record for $dnsEntryName at $timestamp</Comment>
        <Changes>
            $dnsRecord
            $dnsPrefixRecord
            $dnsEC2Record
        </Changes>
    </ChangeBatch>
</ChangeResourceRecordSetsRequest>
ROUTE53-XML

# POST the XML containing Route53 actions
route53Response=$(dnscurl.pl --keyname $awsAccountKeyName $optionsDNSCurl -X POST --upload-file $xmlTmp $urlRoute53API/hostedzone/$hostedZoneId/rrset 2>/dev/null)

# Delete the temporary XML file
rm -f $xmlTmp

# Obtain the response and check its status
route53ResponseStatus=$(echo $route53Response | xpath 'ChangeResourceRecordSetsResponse/ChangeInfo/Status' 2>/dev/null | awk -F'[<|>]' '/Status/{print $3}' | cut -d/ -f3)
if ([ "$route53ResponseStatus" != "PENDING" ]) then
    echo "Error: Expected PENDING status from Route53, but received some other status."
    echo $route53Response
    exit 1
fi

# Obtain Route53 transaction ID
route53ResponseID=$(echo $route53Response | xpath 'ChangeResourceRecordSetsResponse/ChangeInfo/Id' 2>/dev/null | awk -F'[<|>]' '/Id/{print $3}' | cut -d/ -f3)
echo "Received response status ID $route53ResponseID with status $route53ResponseStatus"

echo "Successfully removed Route53 records."