Custom Tasks – Customizing AWS Elastic Beanstalk

Definitely every application needs to run few routine tasks daily or hourly as per the requirement. Under Linux, crontab can be used for that purpose.Let us have our directory structure. In this blog post we are covering three scripts. That leaves only one script in the “setup” directory.

Our script crontab-setup.sh sets up the routine jobs that need to be run. The jobs are listed in the data file crontab.txt in the data directory. Just for illustration purpose the data file contains only one job that runs once every day at 20:30 hours. Important point to note here is to check the time zone used by the EC2 instance (not its location or region). AWS Elastic Beanstalk by default uses UTC time zone. Please ensure what is the case with your EC2 instance and try scheduling daily jobs during the time when your website traffic is minimum.

30 20 * * * /myApp/.ebextensions/scripts/daily/myApp-daily-backup.sh

At hudku.com the names of all our routine scripts start with “hudku”, the name of the application. This is useful in identifying our jobs from the list of crontab jobs available in the system. The script crontab-setup.sh, first obtains the list of existing crontab jobs, removes the jobs that have the application name, reads the list of jobs from the data file crontab.txt, appends them to the list and schedules all the jobs.

#!/bin/bash

# Execute "export DEBUG=1" to debug this script.
# Set value to 2 to debug this script and the scripts called within this script.
# Set value to 3,4,5 and so on to increase the nesting level of the scripts to be debugged.
[[ $DEBUG -gt 0 ]] && set -x; export DEBUG=$(($DEBUG - 1))


# Setup crontab jobs
if ([ -f $ELASTICBEANSTALK_APP_DATA_DIR/crontab.txt ]) then

    # Remove any existing entries and add new ones
    sudo crontab -l | grep -v "/$ELASTICBEANSTALK_APP_NAME/" | { cat; cat $ELASTICBEANSTALK_APP_DATA_DIR/crontab.txt; } | crontab -

fi

The script logs-setup.sh configures the log rotation settings. Please alter it according to your requirement.

#!/bin/bash


# Execute "export DEBUG=1" to debug this script.
# Set value to 2 to debug this script and the scripts called within this script.
# Set value to 3,4,5 and so on to increase the nesting level of the scripts to be debugged.
[[ $DEBUG -gt 0 ]] && set -x; export DEBUG=$(($DEBUG - 1))


#
# Setup logging
#


# include all the utility scripts
source $ELASTICBEANSTALK_APP_SCRIPT_DIR/include/include.sh



# Make the log rotation happen daily instead of hourly
#if $(exists /etc/cron.hourly/logrotate-elasticbeanstalk*) ; then
    # mv /etc/cron.hourly/logrotate-elasticbeanstalk* /etc/cron.daily/
    # sed -i 's|/usr/sbin/logrotate /etc/logrotate.conf.elasticbeanstalk.httpd|/usr/sbin/logrotate -f /etc/logrotate.conf.elasticbeanstalk.httpd|g' /etc/cron.daily/logrotate-elasticbeanstalk-httpd
#fi

The script app-custom-setup.sh as the name indicates can be used to perform the tasks that are specific to a particular application. The important point to be noted in this script is it identifies three types of jobs that may be needed.

1. Jobs that need to be run only once
That means the script is running for the very first time on the EC2 instance. Jobs like installing packages, setting up ssh keys, network settings, etc. might fall under this category.

2. Jobs that need to be executed only by the leader EC2 instance of the Auto Scaling group
Definitely we do not need all the instances in the group to be performing a daily database backup. Similarly uploading new files to be served by AWS CloudFront is another example.

3. Jobs that need to run during every deployment
These are the changes that are part of the new application version and are supposed to come into effect. For example, it could be the changes to web server configuration files, settings that are required by the application, etc.

Here is the source code of app-custom-setup.sh.

#!/bin/bash

# Execute "export DEBUG=1" to debug this script.
# Set value to 2 to debug this script and the scripts called within this script.
# Set value to 3,4,5 and so on to increase the nesting level of the scripts to be debugged.
[[ $DEBUG -gt 0 ]] && set -x; export DEBUG=$(($DEBUG - 1))


appName=$ELASTICBEANSTALK_APP_NAME


#
# You can do all custom tasks here or you can call other scripts.
#


# Check if this is the very first time this script is running by checking the presence of a file
if ([ ! -f /root/.not-a-new-instance.txt ]) then
    newEC2Instance=true
fi



if ([ $newEC2Instance ]) then
    
    # If new instance then perform tasks such as installing a package or download files/folders from S3 private bucket
    if ([ -f $ELASTICBEANSTALK_APP_SCRIPT_DIR/$appName-setup/$appName-package-install.sh ]) then
        $ELASTICBEANSTALK_APP_SCRIPT_DIR/$appName-setup/$appName-package-install.sh
    fi

fi


# Commands to be executed only by the leader of AutoScaling group
if ([ $ELASTICBEANSTALK_CMD_LEADER ]) then
    
    # For example copy css, images and other static resources to S3 bucket serving static files
    if ([ -f $ELASTICBEANSTALK_APP_SCRIPT_DIR/$appName-setup/$appName-upload-static-resources.sh ]) then
        $ELASTICBEANSTALK_APP_SCRIPT_DIR/$appName-setup/$appName-upload-static-resources.sh
    fi
        
fi


# Do tasks to be performed by each instance on every deployment
if ([ -f $ELASTICBEANSTALK_APP_SCRIPT_DIR/$appName-setup/$appName-setup.sh ]) then
    $ELASTICBEANSTALK_APP_SCRIPT_DIR/$appName-setup/$appName-setup.sh
fi

Note:
I updated the post “Security Credentials – Customizing AWS Elastic Beanstalk” to add few more details about “.s3cfg“, the configuration file used by the tool s3cmd.

In our next post we shall discuss the only remaining script file in our setup directory.