Anything Worth Doing Is Worth Overdoing or: Extending an Amazon AWS CloudFormation Template

The need for stability

Meg’s sites, megbitton.com and soulsimagined.com have uptime requirements that far and away exceed those of mine. Almost the entirety of her business is predicated on a persistent Internet presence. As I’m sure you understand, a few minutes of downtime could lead to a belief that she’s no longer in business. This quote from The Social Network says it all:

Mark Zuckerberg: [speaking frantically, almost hysterical] Without money the site can’t function. Okay, let me tell you the difference between Facebook and everyone else, we don’t crash EVER! If those servers are down for even a day, our entire reputation is irreversibly destroyed! Users are fickle, Friendster has proved that. Even a few people leaving would reverberate through the entire user base. The users are interconnected, that is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go, don’t you get that? I am not going back to the Caribbean Night at AEPi!

While I certainly appreciate the job that Dreamhost does in keeping their servers and vis-à-vis their network available to the Internet; it’s not at a level of fault tolerance that you can obtain from an Amazon Web Services EC2 instance. EC2 stands for Elastic Cloud Compute and is, for all intents and purposes, a virtual private server capable of hosting an OS such as Windows or Linux. What makes an EC2 instance special is two-fold. First, you can dynamically expand the number of instances running based on demand. Second, and this is the real crux of the issue, you can host multiple EC2 instances in disparate data centers and load balance across all instances. That is where the reliability comes into play.

Amazon is making a big play in this space and has a myriad of options available within AWS. I’m not going to go into great detail, but I’ll have you know that you can host databases, load balancers, DNS, message queues, notification systems, and a whole host of other services. Here is a quick screenshot of the toolbar icons that are available in the AWS console.

AWS Console Toolbar Icons

As a quick aside, I’m using S3 and CloudFront as my CDN for this site.

CloudFormation, AWS and the free Micro tier

In order for users to become acquainted with AWS, Amazon makes an EC2 micro instance free for one year. Beside the obvious which is a free VPS for one year, what it really does is allow you to drive AWS like a rented mule. Create the instance, muck up the OS “real good” and then kill it and create a new one. This is where I started to have fun with another component of AWS titled CloudFormation. From the CloudFormation site:

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.

The templates are JSON, and have a defined structure to them. The CloudFormation documentation has a lot of detaills and a must for getting around. I’m not going to go into those details because I want to jump to the gist of this post, which is taking the Single EC2 Instance web server with Amazon RDS database instance sample template and adding additional CloudFormation functionality.

AWS::CloudFormation::Init

A key feature in templates is the ability to perform actions against the EC2 instance after it’s created. You can do the following:

  • Install packages such as httpd using yum
  • Unpack one or more zip or tarballs into a defined location
  • Perform command line operations
  • Create files
  • Start services such as httpd
  • Add users
  • Manage user groups

AWS::CloudFormation::Init is a sub-resource created within the template under the EC2 instance. AWS::CloudFormation::Init is passed as metadata to the EC2 resource telling it what to do when when its fired up. AWS::CloudFormation::Init is started by a call to an AWS specific script that is installed within the default Linux instance, cfn-init. This script is called by an Bash script defined in the template. The Bash script, pushed in the UserData field of the EC2 instance properties, also contains a number of functions in the sample template. The original EC2 resource Properties looked like this:

UserData

  "Properties": {
    "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "InstanceType"   : { "Ref" : "InstanceType" },
    "SecurityGroups" : [ {"Ref" : "WebServerSecurityGroup"} ],
    "KeyName"        : { "Ref" : "KeyName" },
    "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
      "#!/bin/bash\n",
      "yum update -y aws-cfn-bootstrap\n",

      "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" }, " -r WebServer ",
      "         --access-key ", { "Ref" : "HostKeys" },
      "         --secret-key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},
      "         --region ", { "Ref" : "AWS::Region" }, "\n",

      "/opt/aws/bin/cfn-signal -e $? '", { "Ref" : "WaitHandle" }, "'\n",

      "# Setup correct file ownership\n",
      "chown -R apache:apache /var/www/html/wordpress\n"

    ]]}} 

This script is updating the AWS scripts to the latest version, starting AWS::CloudFormation::Init, and then tweaking the WordPress install. WordPress was installed from a tarball into the web root of the server in this section of the template:

Metadata

"AWS::CloudFormation::Init" : {
      "config" : {
        "packages" : {
          "yum" : {
            "httpd"     : [],
            "php"       : [],
            "php-mysql" : []
          }
        },
        "sources" : {
          "/var/www/html" : "http://wordpress.org/latest.tar.gz"
        },
        "files" : {
          "/var/www/html/wordpress/wp-config.php" : {
            "content" : { "Fn::Join" : ["", [
              "<?php\n",
              "define('DB_NAME',          '", {"Ref" : "DBName"}, "');\n",
              "define('DB_USER',          '", {"Ref" : "DBUsername"}, "');\n",
              "define('DB_PASSWORD',      '", {"Ref" : "DBPassword" }, "');\n",
              "define('DB_HOST',          '", {"Fn::GetAtt" : ["DBInstance", "Endpoint.Address"]},"');\n",
              "define('DB_CHARSET',       'utf8');\n",
              "define('DB_COLLATE',       '');\n",
              "define('AUTH_KEY',         'f@A17vs{ mO0}:&I,6SB.QzV`E?!`/tN5:~GZX%=@ZA%!_T0-]9>g]4ll6~,6G|R');\n",
              "define('SECURE_AUTH_KEY',  'gTFTI|~rYHY)|mlu:Cv7RN]GQ^3ngyUbw;L0o!12]0c-ispR<-yt3qj]xjquz^&9');\n",
              "define('LOGGED_IN_KEY',    'Jd:HG9M)1p5t2<v~+R-vd{p-Q*|*RB^&PUI{vIrydAEEiV!{HS{jN:nErCmLv`p}');\n",
              "define('NONCE_KEY',        '4aMj4KZV;,Gu7(B|qOCve[c5?*J5x1+x93i:Ey6hh/6jXh+V_{V4+hw!qE^d*U,-');\n",
              "define('AUTH_SALT',        '_Y_&8m)FH)Cns)8}Yb8b88KDSn:p1#p(qBa<~VW&Y1v}P.*9/8S8@P`{mkNxV lC');\n",
              "define('SECURE_AUTH_SALT', '%nG3Ag41^Lew5c86,#zbN:yPFs.GA5a)z5*:Oce1>v6uF~D`,.o1pzS)F8[bM9i[');\n",
              "define('LOGGED_IN_SALT',   '~K<y+Ly+_Ww1~dtq>;rSQ^+{P5/k|=!]k%RXAF-Y@XMY6GSp+wJ5{(|rCzaWjZ%/');\n",
              "define('NONCE_SALT',       ',Bs_*Y9:b/1Z:apVLHtz35uim|okkA,b|Jt[-&Nla=T{<l_#D?~6Tj-.2.]FonI~');\n",
              "define('WPLANG'            , '');\n",
              "define('WP_DEBUG'          , false);\n",
              "$table_prefix  = 'wp_';\n",
              "if ( !defined('ABSPATH') )\n",
              "    define('ABSPATH', dirname(__FILE__) . '/');\n",
              "require_once(ABSPATH . 'wp-settings.php');\n"
            ]] },
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          }
        },
        "services" : {
          "sysvinit" : {
            "httpd"    : { "enabled" : "true", "ensureRunning" : "true" },
            "sendmail" : { "enabled" : "false", "ensureRunning" : "false" }
          }
        }
      }

This segment of the original template creates the WordPress wp-config.php file with references to the RDS resource that is also generated by the template, installs the necessary packages on the server, and sets up the proper services to run as daemons. The first component of the template that I took issue with was that every install was going to have the same random keys. Fortunately, WordPress has a service that will generate these values for you. All you need to do is call https://api.wordpress.org/secret-key/1.1/salt/ and WordPress will return to you the necessary segments to add to your wp-config.php. First problem to solve, how do I get these values into the middle of the file. Why the middle? I tried a simple append but the WordPress install didn’t see them. After quite a bit of trial and error, I wrote a Perl script that is called from the command line. I did this because I didn’t want to leave my custom script as a file on the server.

Perl file mangling

              "command" : { "Fn::Join" : ["", [
                "perl - /var/www/html/wp-config.php <<'EOF'\n",
                "use strict;\n",
                "use warnings;\n",
                "\n",
                "$^I = '.bak';\n",
                "my $keys = `curl -sS https://api.wordpress.org/secret-key/1.1/salt/`;\n",
                "while(<>) {\n",
                "  if (\/\\{KEYS_TOKEN\\}\/){ \n",
                "    print $keys; \n",
                "  } else { \n",
                "    print; \n",
                "  }\n",
                "}\n",
                "EOF\n"
              ]]}

This script looks for a token that I inserted into the wp-config.php file. This snippet is part of a larger section of the template. Since things have to happen in order, CloudFormation templates allow you to have sets of config options, run them in specific order, and then run commands in order. Here is the larger snippet.

configSets

      "configSets" : {
          "default" : [ "first", "second" ]
        },
      "second" : {
          "commands" : {
            "a" : {
              "command" : "mv /var/www/html/wordpress/* /var/www/html/wordpress/.??* /var/www/html/ ; rmdir /var/www/html/wordpress"
            },
            "b" : {
              "command" : "chown -R apache:apache /var/www/html ; find /var/www/html -type d -exec chmod 770 {} + ; find /var/www/html -type f -exec chmod 660 {} +"
            },
            "c" : {
              "command" : { "Fn::Join" : ["", [
                "perl - /var/www/html/wp-config.php <<'EOF'\n",
                "use strict;\n",
                "use warnings;\n",
                "\n",
                "$^I = '.bak';\n",
                "my $keys = `curl -sS https://api.wordpress.org/secret-key/1.1/salt/`;\n",
                "while(<>) {\n",
                "  if (\/\\{KEYS_TOKEN\\}\/){ \n",
                "    print $keys; \n",
                "  } else { \n",
                "    print; \n",
                "  }\n",
                "}\n",
                "EOF\n"
              ]]}
            }
          }
        },
      "first" : {
        "packages" : {
          "yum" : {
            "httpd"     : [],
            "php"       : [],
            "php-mysql" : [],
            "mysql"     : []
          }
        },
        "sources" : {
          "/var/www/html" : "http://wordpress.org/latest.tar.gz"
        },
        "files" : {
          "/var/www/html/wordpress/wp-config.php" : {
            "content" : { "Fn::Join" : ["", [
              "<?php\n",
              "define('DB_NAME',          '", {"Ref" : "DBName"}, "');\n",
              "define('DB_USER',          '", {"Ref" : "DBUsername"}, "');\n",
              "define('DB_PASSWORD',      '", {"Ref" : "DBPassword" }, "');\n",
              "define('DB_HOST',          '", {"Fn::GetAtt" : ["DBInstance", "Endpoint.Address"]},"');\n",
              "define('DB_CHARSET',       'utf8');\n",
              "define('DB_COLLATE',       '');\n",
              "define('WPLANG'            , '');\n",
              "define('WP_DEBUG'          , false);\n",
              "{KEYS_TOKEN}\n",
              "$table_prefix  = 'wp_';\n",
              "if ( !defined('ABSPATH') )\n",
              "    define('ABSPATH', dirname(__FILE__) . '/');\n",
              "require_once(ABSPATH . 'wp-settings.php');\n"
            ]] },
            "mode" : "000644",
            "owner" : "root",
            "group" : "root"
          },
          "/etc/cfn/cfn-credentials" : {
            "content" : { "Fn::Join" : ["", [
              "AWSAccessKeyId=", { "Ref" : "WebServerKeys" }, "\n",
              "AWSSecretKey=", {"Fn::GetAtt": ["WebServerKeys", "SecretAccessKey"]}, "\n"
            ]]},
            "mode"    : "000400",
            "owner"   : "root",
            "group"   : "root"
          },

          "/etc/cfn/cfn-hup.conf" : {
            "content" : { "Fn::Join" : ["", [
              "[main]\n",
              "stack=", { "Ref" : "AWS::StackName" }, "\n",
              "credential-file=/etc/cfn/cfn-credentials\n",
              "region=", { "Ref" : "AWS::Region" }, "\n"
            ]]},
            "mode"    : "000400",
            "owner"   : "root",
            "group"   : "root"
          },

          "/etc/cfn/hooks.d/cfn-auto-reloader.conf" : {
            "content": { "Fn::Join" : ["", [
              "[cfn-auto-reloader-hook]\n",
              "triggers=post.update\n",
              "path=Resources.WebServerEc2Instance.Metadata.AWS::CloudFormation::Init\n",
              "action=/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" },
              "              -r WebServerEc2Instance ",
              "              -c default",
              "              --credential-file /etc/cfn/cfn-credentials ",
              "              --region     ", { "Ref" : "AWS::Region" }, "\n",
              "runas=root\n"
            ]]}
          }              
        },

You’ll notice now there are first and a second config sections. They appear backwards in the JSON, however if you look at the configSets (documentation), the sections are called in the correct order. Also commands are called in alphabetical order, so I just named them a, b, c, and so on. The other commands I perform is moving WordPress out of a folder under the webroot, setting the correct file and folder permissions, and of course the mangling of the wp-config.php file.

cfn-hup

The last added item to the template is the use of cfn-hup. cfn-hup is an optional script that can run as a daemon and look for template updates that are pushed to the server. When changes occur, cfn-hup reads its respective config files and performs the prescribed actions defined within. I added some boilerplate code into the template from the CloudFormation docs.

userData Redux

    "UserData"       : 
    { 
      "Fn::Base64" : 
      { 
        "Fn::Join" : 
        [
          "", 
          [
            "#!/bin/bash\n",
            "yum update -y aws-cfn-bootstrap\n",

            "# Helper function\n",
            "function error_exit\n",
            "{\n",
            "  /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '", { "Ref" : "WaitHandle" }, "'\n",
            "  exit 1\n",
            "}\n",

            "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" }, " -r WebServerEc2Instance -c default",
            "         --access-key ", { "Ref" : "HostKeys" },
            "         --secret-key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},
            "         --region ", { "Ref" : "AWS::Region" },
            " || error_exit 'Failed to run cfn-init.'\n",

            "# Start up the cfn-hup daemon to listen for changes\n",
            "/opt/aws/bin/cfn-hup || error_exit 'Failed to start cfn-hup'\n",

            "# Now that WordPress is complete, perform a full yum update\n",
            "#yum -y update",
            "# || error_exit 'Failed to yum update.'\n",

            "# All is well so signal success\n",
            "/opt/aws/bin/cfn-signal -e 0 -r \"Wordpress installation complete.\" '", { "Ref" : "WaitHandle" }, "'\n"
          ]
        ]
      }
    }        

Here is the update Bash script that is run after install. Besides starting init, it also starts up cfn-hup. There’s also a full yum -y update in there, however for testing, it’s commented out.

Complete Template

You’re welcome to a copy of the script, and it’s in a ZIP file below. I also started a GitHub repo with this template and any others I create in the future. Thanks!

Download Here

5 comments

  1. Would you recommend someone moving from wordpress hosted on Dreamhost to hosting on the Amazon AWS/CloudFormation setup? Or would it be more trouble than its worth, if the user (me) wasn’t totally familiar with servers and setting them up? My site doesn’t get much traffic, but I’ve been looking for a solution for speed – and amazon is tempting, but I’m just not sure if I would be getting in over my head…

    1. Nathan,
      It all comes down to your needs. This site for example is on Dreamhost. I’m using AWS for my wife’s sites which have a higher up-time requirement. The key with AWS is the ease of setting up replicated servers in multiple availability zones as well as replicated RDS (database) servers. The flip-side is that you’re pretty much on your own when it comes to server maintenance. If you have a small amount of non-ecommerce traffic, I would stick with Dreamhost.

  2. I have hosted many websites but all of them through cPanel / Plesk. Is there a way to install such services on AWS so that the task is reduced to coding the main logic for the website rather than worrying about the SSH and setting up of resources which I do not know how to do.

    1. You can essentially instal whatever you like. If you’re looking for a no-cost solution, checkout Webmin.

Comments are closed.