Exporting Lightsail instance snapshots to Amazon EC2

I’ve been using Amazon Lightsail for the past six months to host my WordPress blog site. I’m happy with Lightsail – I have full control over both my Linux instance OS and my WordPress blog site administration but I don’t have to deal with the headaches (or the price) of a full-blown EC2 instance.

A few months back I wrote a blog post listing the pros and cons of Amazon Lightsail.  Why would one use Lightsail versus Amazon EC2?  Within the Amazon ecosystem, Lightsail is the cheapest and easiest to use platform if the goal is to get a Linux instance and simple application up and running in AWS.

Think of Lightsail as a lightweight version of AWS EC2.  There really is no learning curve to get started on Lightsail – just pick some basic options and launch the instance.  The Lightsail pricing structure is straightforward and predictable.  Both the pricing and ease of use make Lightsail as a hosting platform  attractive for bloggers and developers to get started on Amazon and the experience is quick, cheap, and easy.

With this simplicity comes limitations.  A lightweight version of an Amazon EC2 instance doesn’t have the full compliment of every EC2 feature.  This makes sense – Lightsail is geared towards bloggers and developers and is kept separate from EC2.  A lightweight version works fine when starting a project but what if we hit a point where the advanced networking features of a VPC are required?  Or autoscaling groups?  Or spot instances?

Fortunately this has changed as of AWS re:Invent 2018 – we can now convert a Lightsail instance into an EC2 instance using Lightsail instance snapshots!   I’ll walk through the process in this blog and show some of the strategies and considerations needed to move a Lightsail instance out of the limited Lightsail environment into the feature-rich EC2 environment.

Why should I move my instance out of Lightsail?

Let’s face it – we aren’t going to move our instance out of Lightsail unless we have to.  Lightsail is the cheapest option for running an instance in AWS.  Instance monthly plans start as low as $3.50/month as long as the instance stays within the resource bounds of the pricing plan.

Adding more horsepower to a Lightsail instance is also not a reason to move to EC2.  Lightsail allows upgrading an instance to use more resources (for a price) – just use an instance snapshot to clone the existing instance to a new plan with more resources (the application inside the instance will persist using instance snapshots).

But what if we want to to take our application out of the Lightsail environment and place it into more of a production environment?  What if we have found limitations in Lightsail and found that our applications needs to reside in EC2 to take advantage of all the available EC2 features?  What if we need web-scale features, advanced networking, and access to services that are limited in the Lightsail environment?  In that use case we need to move to EC2.

AWS terminology and concepts

Before we go through the conversion process, let’s clear up some terminology and concepts that will help illustrate the task.

Instance – this is just Amazon’s fancy word for a virtual machine.  The term “virtual machine” probably sounded too much like VMware so Amazon calls virtual machines “instances”.  An instance is a virtual machine but runs on Amazon’s proprietary hypervisor environment.

Lightsail instance – An AWS image running in the Lightsail environment.  Lightsail is separate from other AWS environments and runs on a dedicated  hardware stack in selected data centers.

EC2 instance – An AWS image running in the EC2 environment.  EC2 is completely separate from Lightsail and runs on its own hardware and has many more management and configuration options.

Instance snapshot  – A snapshot of either a Lightsail or EC2 instance that not only includes the state of the virtual machine but also a snapshot of the attached OS disk.

Disk snapshot – Unlike an instance snapshot, a disk snapshot only tracks changes to the physical disk attached to the virtual machine.  No virtual machine state is captured so this type of snapshot is only good for keeping track of block disk data, not for the virtual machine state.

CloudFormation Stack – a template that tracks AWS resources associated with an AWS service through the use of a template.  CloudFormation can be used to define the configuration details of an AWS instance.

Key pair – quite simply, a .PEM file used to connect to an instance.  We can’t connect to an instance remotely (SSH, Putty) without a .PEM key pair.

AMI – a preconfigured template that is also associated with an EBS snapshot and is used to launch an instance with a specified configuration

How exporting an instance snapshot works

Instance snapshots are portable.  When an instance snapshot is taken of a Lightsail instance, the OS and disk state are stored to flat files and kept in some form of S3 object storage.  The flat files can be shared within Lightsail to create a new instance from the same snapshot – we can even assign more CPU or RAM to an instance snapshot to upgrade the instance to something faster.

With the latest AWS Lightsail EC2 instance snapshot export feature we can now take the same Lightsail snapshot and share it with AWS EC2 infrastructure.  This allows EC2 to import the Lightsail snapshot  to clone our Lightsail instance and spin it up in EC2.  Lightsail snapshots were always portable but the recent EC2 export feature enabled sharing of snapshots outside of Lightsail.

Behind the scenes Amazon uses CloudFormation to enable this conversion process.  Which simply means we automatically get a CloudFormation stack (or template) created when we move our Lightsail instance snapshot out to EC2.  But we don’t have to worry at all about CloudFormation, this is transparent to the user and doesn’t even require the user to open the CloudFormation console or interact at all with CloudFormation.

Walkthrough of moving a Lightsail instance to EC2

Exporting a Lightsail instance is very easy.  First log into your AWS account and open the Lightsail console.  Select your existing Lightsail instance to see the options below.

lightsail Console

Open the snapshots tab and create an instance snapshot (if you don’t already have one).  Use any name you like and click “create snapshot”.  The process will most likely take a few minutes and the progress is shown in the console.

Create Instance Snapshot in Lightsail console

After the snapshot is completed it will display in the list of “recent snapshots”.  We can then select this snapshot (three orange dots on the right) and select “Export to Amazon EC2”.  This will move the snapshot out of Lightsail object storage over to EC2.

Recent Snapshots in Lightsail Console

An informational pop-up will appear explaining the outcome (EBS snapshot and AMI) along with a warning that EC2 billing rates will apply, click “Yes, continue”.

Export Lightsail Snapshot to EC2 in Lightsail console

One last warning will appear alerting that the existing default Lightsail key pairs (.PEM file) should be replaced with an individual key pair.  Click “Acknowledged”.

Export Security Warning in Lightsail Console

The progress can be monitored using the top “gear” in the Lightsail console which will “spin” during the export process.  When completed, the gear will stop “spinning” and the task will show as completed and “Exported to EC2” in the task history.

Running Snapshot export task in Lightsail Console

Completed Snapshot export in lightsail console

Using the exported EC2 Lightsail instance

Now that we have exported our Lightsail instance snapshot to EC2, what happens next?  By default, nothing.  Looking at the completed task screenshot (above), we can open the EC2 console to see what we have or we can use the Lightsail console to launch in EC2 instance from our exported snapshot.

Lightsail is great and I love the simplified console but we really should use the EC2 console at this point.  Lightsail has all new wizard screens to manage our instance in EC2 but my feeling is that if we are going to export to EC2, we should switch over and learn to use the EC2 console.

See the screenshot below, open the EC2 console and make sure region is the same that was used for Lightsail (in my case I’m US East (Ohio)).

EC2 in the AWS Console

Open the “Elastic Block Storage” link on the left side of the EC2 console.  On the top right side of the console we can see the instance snapshot that was created by Lightsail.  Since this snapshot is an instance snapshot we will also find an AMI associated with this EBS snapshot.

elastic block storage snapshots in EC2 Console

EBS snapshots in ec2 console

Open the “Images” – “AMI” link in the EC2 console.  We can see in the upper right frame that the Lightsail snapshot has been converted to an EC2 AMI.  This was done with AWS CloudFormation but that doesn’t really matter, the AMI is ready for us to use without worrying about how it was created.

AMIs in ec2 console

AMis in ec2 console

Now the fun part, we can launch our Lightsail instance in EC2 using the AMI.  Just click “launch”!  A few things to consider when launching the AMI (the details are out of scope for this post):

Instance size – pick an EC2 image size for your Lightsail instance, the EC2 options have different pricing plans and different levels of CPU, RAM, GPU, storage, etc.

Security (key pair, security groups, VPC info) –  design a strategy (if you don’t already have one) to secure your new image with a VPC, security group, key pair, etc.

Additional storage – additional disk storage can be attached during the EC2 launch

Tags – optionally you can tag your instance for easier AWS billing.

Pick your options and you are ready to go, your Lightsail instance is now running in EC2!  Thanks for reading!

Backing up WordPress on Lightsail

I work for a data protection company.  I spend a decent amount of time (and a small amount of money) writing blogs posts on my WordPress site that runs on an AWS Lightsail instance.  So after publishing a few posts to my blog I naturally started thinking about how I was going to backup my site and protect it against any unforeseen glitches or hacks.

An AWS Lightsail instance runs on a single physical server.  That server could have hardware problems and/or reboot unexpectedly.  Which means your instance could experience an unexpected outage, similar to pulling the power plug when you were least expecting it.  In a worst case scenario it could lead to data corruption to your WordPress config files, your operating system, or your mySQL database.

Think about it from a security perspective as well.  WordPress sites get hacked all the time, it would be nice to have the protection of a point in time copy of your instance plus your WordPress data that you could roll back to in case things your site gets compromised.  A website is public facing, it is very likely someone could scan your site for vulnerabilities and hack it.  For me it would be more a nuisance than anything else but I still don’t want to lose hours of time spent that I could have easily avoided against with backups.

UPDATED DEC 7th, 2018 – See my Github site for sample code to backup WordPress on Lightsail!

WordPress – what to backup?

WordPress is pretty simple, we only have to protect a few things:

  • PHP – WordPress is written using PHP as the scripting language, we’ll need a good copy of the PHP config
  • Apache –  Apache is the web server that runs WordPress and serves up web pages, we’ll need a copy of the Apache config
  • mySQL – mySQL is the database used by WordPress for data management, this is an important component of our backup because it contains our website data
  • Lightsail instance / operating system – Our website won’t run without an operating system, we need a way to keep a point in time copy of our Linux OS

If you are running WordPress on a hosted site you probably don’t need to worry as much about backups.  Your service provider is probably running your WordPress backups and charging you for the service.  However, if you are running a Lightsail instance with a preconfigured WordPress application you do need to worry about protecting your site and OS.  Why?  Because AWS is simply giving you a Linux instance running in their cloud with a preconfigured Bitnami WordPress app installed.  AWS doesn’t take care of your backups, you have to do this yourself and this is part of the AWS shared responsibility model.

What about still using a WordPress backup plugin?  You certainly could use a free or paid WordPress plugin to backup your WordPress site.  These usually take care of the all the config files and the mysql database that makes up WordPress.  Personally, I’m not terribly enthused with WordPress plugins, they expose your site to vulnerabilities, they constantly need patching, they get abandoned and deprecated over time by the developers, and they just seem clunky to me.

Also, what if my Linux operating system is corrupt or hacked?  With a WordPress backup plugin, I would have to redeploy my instance from scratch with a blank WordPress installation and then try restore using the plugin.  I want a copy of my Linux instance so I have less work to do in the case of a disaster.  If my instance vanished tomorrow, I just want to bring everything back in a few minutes without reinstalling a bunch of packages, plugins, then manually restoring.

Getting a clean copy of your WordPress site in a backup

You might be tempted to just take a Lightsail snapshot of your running WordPress Linux instance and call it a day. Snapshots are a point in time clone of your Lightsail instance disk that can be used to spin up a copy of your instance in case of disaster or to launch a second identical instance.  You can create snapshots from the Lightsail console, the AWS API, or the AWS CLI installed on your instance.  I might sound paranoid for this, but an operating system snapshot isn’t adequate for my WordPress backup.  Just look at the Bitnami documentation for creating a full backup of your WordPress site.

PHP, Apache, and mySQL work together to run your WordPress site.  The operating system coordinates the compute, memory management, networking, and disk I/O subsystem.  If you try to just take a LightSail snapshot of the operating system, you may have a transaction in flight that is sitting in RAM but has not yet been committed to the mySQL database.  Or Linux may have filesystem I/O that has not yet been flushed to disk.  You can’t be sure you are getting a good backup unless you stop your WordPress services and back up the database and config files while the services are stopped.

Would a Lightsail snapshot work as a backup?  Sure, you can take an existing Lightsail snapshot and create a new instance from that snapshot.  And most likely your PHP, Apache, and mySQL would start up just fine and your website would work just like it did when you took the snapshot. But there is a slim chance that you could experience some type of problem where your PHP, Apache, or mySQL doesn’t start correctly.  The industry lingo for this concept is a crash consistent versus application consistent snapshot.  Lightsail snapshots are crash consistent since the operating system will boot up but not application consistent (the app may not start correctly).  We want application consistency for WordPress and crash consistency for our Linux image.

Application consistency with WordPress

Luckily you can easily get a backup of your WordPress site and use a Lightsail snapshot of the instance to keep everything in an AWS snapshot.  This involves a quick site outage but will guarantee your mySQL database and WordPress files are all quiesced with no pending writes.

The WordPress app bundled in (Linux) Lightsail is nicely contained in the directory /opt/bitnami.  Everything related to WordPress resides in that directory.  So all you have to do to backup WordPress is to stop all the services, create a backup tar image of the directory, and then start the services back up.

Once you have a tar image of /opt/bitnami you can then take a snapshot of your Lightsail instance.  This will create an AWS snapshot (stored in S3) of your instance that contains a tar file of your entire WordPress site.  If your instance got hacked or corrupted, all you’d have to do would be to create a new instance from the AWS Lightsail snapshot, stop the WordPress services, clear our the contents of the /opt/bitnami directory, untar the  backup file to /opt/bitnami, then restart the WordPress services.

How to backup WordPress on Lightsail

We are going to do this with the CLI so first you’ll need to download your SSH key and SSH into your instance.  Unlike EC2 instances, your SSH user  is ‘bitnami’ instead of ‘ec2-user’.  Make sure you turn on SSH access on your instance firewall in the Lightsail console.

 

ssh -i YourLightsailKey-region.pem bitnami@<yourPublicIPaddress>

Install the AWS CLI tools so you can take snapshots of your instance from the command line.  That way you can script the backup process in ‘cron’ later if you want to automate this process.  Don’t install the AWS CLI with ‘apt-get’ since you’ll get an old version that doesn’t include Lightsail tools.  Install install with the Python tool ‘pip’.  You’ll know if you have the right AWS CLI version if you see the option to run something like ‘aws lightsail help’.

$ aws --version

aws-cli/1.15.80 Python/2.7.12 Linux/4.4.0-1060-aws botocore/1.10.79

Now run ‘aws configure’ to connect the AWS CLI to your account.  Use your access key and secret key to connect and then pick the region where you have your Lightsail instance hosted.  You should now be able to list your instance names (save the name for later).

$ aws lightsail get-instances | grep name

            "username": "bitnami", 

            "name": "pebblesandweeds-512MB-myregion", 

                "name": "running"

Create a directory where you’ll store your WordPress backup tar file, I’m using /home/bitnami/backup.  Now stop all WordPress services (php, Apache, and mySQL).

sudo /opt/bitnami/ctlscript.sh stop

Now that everything is stopped, tar up everything in /opt/bitnami into a tar file in /home/bitnami/backup (or whatever backup directory you created).

$ pwd
/home/bitnami/backup
sudo tar -pczvf application-backup.tar.gz /opt/bitnami

Now start start WordPress again to get your website back online.

sudo /opt/bitnami/ctlscript.sh start

Now create an instance snapshot for a crash consistent AWS snapshot of your running instance that contains a full WordPress site backup file embedded in the snapshot.  You’ll need your instance name as an argument (and the region if you are working with a region different than the one you gave in ‘aws configure’).

aws lightsail create-instance-snapshot --instance-snapshot-name my.latest.snapshot --instance-name pebblesandweeds-512MB-myregion

As an extra measure of safety, I’m going to also move my backup file to another S3 bucket in my account so I have a second copy.

$ aws s3 cp /home/bitnami/backup/application-backup.tar.gz s3://mybucketname

You probably want to only store the latest Lightsail instance snapshot to avoid getting charged for storing many snapshots, you can easily remove old snapshots with the CLI or the LightSail console.  The entire process can be scripted and scheduled as an automated process as well.

Thanks for reading!