Memory backed filesystem for the MySQL tmpdir

If your queries are constantly writing to disk, and you cannot fix your query (which is strongly recommended) or properly adjust your buffer sizes for one reason or another, you can create a memory backed file system for the tmpdir, which should allow for incredibly faster writes and reads.

This should only be viewed as a temporary fix until you can get your query to stop writing its temp tables to disk.

Create the mount point:

mkdir -p /tmp/mysqltmp

Now give MySQL access to it:

chown mysql:mysql /tmp/mysqltmp

Set up the entry in /etc/fstab so it will mount on reboot:
* NOTE: Change the uid and gid to whatever MySQL runs as in /etc/passwd and /etc/group

vi /etc/fstab
# Add
tmpfs /tmp/mysqltmp tmpfs
rw,gid=27,uid=27,size=256M,nr_inodes=10k,mode=0700 0 0

Mount the memory backed file system:

mount -a

Update /etc/my.cnf and add the variable:

tmpdir = /tmp/mysqltmp

Now, restart MySQL:

service mysql restart

Finally, confirm MySQL is now utilizing the new memory backed filesystem:

mysql
mysql> show variables like '%tmpdir%';
+-------------------+----------------+
| Variable_name     | Value          |
+-------------------+----------------+
| slave_load_tmpdir | /tmp           |
| tmpdir            | /tmp/mysqltmp  |
+-------------------+----------------+
2 rows in set (0.00 sec)

Remote backups with rsnapshot

What type of backup strategy do you employ for your solution? Do you have backups within your datacenter, or are you utilizing your hosting providers backup infrastructure if one is available? These are both good starting points for preparing your solution for disaster.

Now, what do you have in place for remote backups? Remote backups are critical in the event something where to happen to your primary datacenter. What if there was a fire, or there was a major natural disaster that took out the datacenter?

Perhaps as a more common scenario, maybe your existing backup solution was having problems and you weren’t aware of it. When the time comes for needing to restore your backups, you find that they are corrupted and unusable. This happens more often then people think.

When you deploy a new solution, you make sure its redundant and highly available. It is important to also do the same with your backup architecture. Having an on-site backup allows you to perform a speedy recovery should something go wrong. Including an off-site backup solution allows you to plan for the worst case scenario, and also gives you the piece of mind that your data is stored outside of that datacenter, under your control.

When having solution architecture discussions with clients, I strongly encourage:
– Use all available backup solutions offered by the hosting provider
– Have an off-site backup solution that is managed by yourself or a different provider

You can never have enough backups. Your data took weeks, months, or sometimes even years to develop and fine tune. If there are concerns about how much will it cost to have a remote backup solution, here is a more important cost consideration: How much will it cost your business and reputation to rebuild all your website and database content from scratch?

As you can probably tell, I am very paranoid about my clients data. So now that I hopefully gave you some food for thought, I’ll show you one inexpensive way I like to perform remote backups for smaller solutions (Under 500G). Please keep in mind that there are many backup solutions available, this is just one of many different types of solutions I present as an option to my clients.

Welcome rsnapshot. Taken from their website, http://www.rsnapshot.org:

"rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.  Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals. 

Depending on your configuration, it is quite possible to set up in just a few minutes. Files can be restored by the users who own them, without the root user getting involved. 

There are no tapes to change, so once it's set up, your backups can happen automatically untouched by human hands. And because rsnapshot only keeps a fixed (but configurable) number of snapshots, the amount of disk space used will not continuously grow.

Many of the more common questions such as, “How do I restore a backup?” are answered in their FAQ which is located here:
http://www.rsnapshot.org/faq.html

I strongly encourage you to review their documentation so you can decide if this software is good for your solution. I like this solution cause it essentially allows you to simply rsync or SCP the needed information from your remote backup server back to your production servers when you need it. There are no complicated tools required to get your critical data back on your solution.

So, what do you need to set this up? You simply need a Linux/UNIX based computer that is running offsite, maybe even at your office if it is in a secure location, and enough hard drive space to store your backups. Installation is quick and easy as I’ll outline below. For this example, I am using a Rackspace Cloud, CentOS 6 server with 2x 200G Cloud Block Storage volumes setup in a Raid 1, encrypted using LUKS, mounted under /opt/storage01. I outlined how to set this up in an older article: http://www.stephenlang.net/2012/12/encryption-block-storage-in-the-cloud/.

My setup is a bit more elaborate, but again, I am just paranoid about data. A simple server with enough free hard drive space will work just as well. Just make sure it is in a secured location.

Procedure

Without further ado, here is how I personally setup rsnapshot. Please note that you have to enable the EPEL repos on your server to yum install rsnapshot. You can enable the EPEL repo by:

CentOS 5

wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
sudo rpm -Uvh epel-release-5*.rpm

CentOS 6

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo rpm -Uvh epel-release-6*.rpm

Now, install rsnapshot:

yum install rsnapshot

The rest of our setup will take place in /etc/rsnapshot.conf. Make a quick backup of the configuration:

cp /etc/rsnapshot.conf /etc/rsnapshot.conf.orig

Modify the configuration to meet our needs:

vi /etc/rsnapshot.conf

Set the following to specify where you want your backups to be stored. I put in my preference, but you can change this to anything you like. Just be sure its in a location that is only accessible to root:

snapshot_root	/opt/storage02/snapshots/

Now uncomment cmd_ssh as we’ll be rsyncing over SSH:

cmd_ssh	/usr/bin/ssh

Define the backup intervals. Here is what I use:

interval        hourly  6
interval        daily   7
interval        weekly  4
interval        monthly 3

All that is left configure which remote servers you will be backing up. You will have to be sure that you setup SSH keys so rsnapshot can SSH into the remote servers without a passphrase.

As a side note, when backup up your databases, be sure to backup your MySQL dumps, (Or the dumps from whatever database software you are using). If you try to backup the live database, you will likely have severe corruption if you ever need to restore it.

For our example, I am backing up 2 servers:
– db01.example.com (192.168.2.2) : /etc, /var/lib/mysqlbackup
– web01.example.com (192.168.2.3) : /etc, /var/www, and excluding /var/www/example.com/file/big_log_file.log

# db01.example.com (192.168.2.2)
backup  [email protected]:/etc/  db01.example.com/
backup  [email protected]:/var/lib/mysqlbackup/  db01.example.com/

# web01.example.com (192.168.2.3)
backup  [email protected]:/etc/  web01.example.com/
backup  [email protected]:/var/www  web01.example.com/ exclude=file/big_log_file.log

Finally, setup the cron jobs:

crontab -e
0 */4 * * * /usr/bin/rsnapshot hourly
30 8 * * * /usr/bin/rsnapshot daily
55 8 * * 1 /usr/bin/rsnapshot weekly
15 9 1 * * /usr/bin/rsnapshot monthly

Test to ensure everything works accordingly:

/usr/bin/rsnapshot hourly

– Check the directory to ensure your content was saved:

ls /opt/storage02/snapshots/

– Check the log file to ensure there are no errors:

less /var/log/rsnapshot

Most importantly, you must check to ensure that your backup system is functionality properly pretty often. You will want to periodically test your backups, at least every 90 days, to ensure that your team is familiar with the process, and to ensure that everything is okay with your backups. Backups are not ‘set it and forget it’. Always verify your data’s integrity, otherwise you may have a really bad time the day to find you need to restore from backups!

How to dynamically enable general log in MySQL

Ever wonder what the exact queries are that cause MySQL to spike your CPU constantly? Here is an quick and easy way to enable the general log within MySQL without restarting MySQL.

First, confirm your initial variables before changing anything:

mysql> show variables like '%general%';
+------------------+--------------------------+
| Variable_name    | Value                    |
+------------------+--------------------------+
| general_log      | OFF                      |
| general_log_file | /var/lib/mysql/web01.log |
+------------------+--------------------------+
2 rows in set (0.00 sec)

So in this case, the logs will be recorded to /var/lib/mysql/web01.log. If you have nothing set for ‘general_log_file’, you can set it as shown below, but just make sure its in a directory that MySQL can write to:

mysql> set global general_log_file='/var/lib/mysql/web01.log';

With the log file ready, enable the general log by:

SET global general_log = 1;

Verify it is enabled by running:

mysql> show variables like '%general%';
+------------------+--------------------------+
| Variable_name    | Value                    |
+------------------+--------------------------+
| general_log      | On                       |
| general_log_file | /var/lib/mysql/web01.log |
+------------------+--------------------------+
2 rows in set (0.00 sec)

You can now view the queries being ran by looking at your general_log_file. Please note, your log will be in a different location then the one I have below:

tail -f /var/lib/mysql/web01.log

On busy MySQL servers, leaving the general_log setting enabled will quickly cause your hard drive to fill up. So after a few seconds or minutes, disable it by running:

SET global general_log = 0;

Then verify the setting is now off by:

mysql> show variables like '%general%';
+------------------+--------------------------+
| Variable_name    | Value                    |
+------------------+--------------------------+
| general_log      | OFF                      |
| general_log_file | /var/lib/mysql/web01.log |
+------------------+--------------------------+
2 rows in set (0.00 sec)

You can now go through your log file to see exactly what queries have been running.

Fixing invalid system activity with sar

Sometimes sar can return errors complaining about an invalid system activity file such as:

sar
Invalid system activity file: /var/log/sysstat/sa03

It can be resolved by:

rm /var/log/sysstat/sa03
/etc/init.d/sysstat start
* Starting the system activity data collector sadc [ OK ]

Now if you rerun sar, you should see the stats output starting to populate:

sar
Linux 3.2.0-83-virtual (web01) 04/03/2014 _x86_64_ (8 CPU)

Logrotate for MySQL slow query logs

Enabling the MySQL slow query log is extremely useful for finding out which queries are taking too long to execute, and either needs a table index created, or perhaps the query itself may need to be rewritten a bit more efficiently.

But oftentimes, it is forgotten that this log file can grow very large, and it needs to be rotated out. Below is a very quick and easy way to setup log rotation for the MySQL slow query logs:

vim /etc/logrotate.d/mysqllogs
/var/lib/mysql/slow-log {
        missingok
        rotate 2
        size 125M
        create 640 mysql mysql
}

That last line is critical. The file must be recreated with the owner and group set to ‘mysql’, otherwise, MySQL will not have permissions to write to that file.

How to add a 2G swap file to a server

Some cloud servers do not come with swap. There are many arguments about why this is a good thing, and other arguments as to why it is bad. However sometimes you just want that extra buffer for one reason or another. So here is a quick way to add a 2G swap file to an existing cloud server:

Create the initial 2G file

mkdir /opt/swapfiles
touch /opt/swapfiles/2G-swap
dd if=/dev/zero of=/opt/swapfiles/2G-swap bs=1024 count=2097152

Now set it up to be swap, and enable it:

mkswap /opt/swapfiles/2G-swap
chmod 600 /opt/swapfiles/2G-swap
swapon /opt/swapfiles/2G-swap

Set it in /etc/fstab so it will persist across reboots

vi /etc/fstab
# Add
/opt/swapfiles/2G-swap none swap sw 0 0

Confirm your vm.swappiness is not set to 0:

vi /etc/sysctl.conf
# Change
vm.swappiness = 0
# To
vm.swappiness = 10

Finally, update the sysctl without having to reboot:

sysctl vm.swappiness=10

You can confirm the swap space is now active by:

free -h
             total       used       free     shared    buffers     cached
Mem:          990M       905M        85M        43M       124M       527M
-/+ buffers/cache:       253M       736M 
Swap:         2.0G         0B       2.0G 

SSH brute force prevention with fail2ban

Ever take a look at your server’s auth logs and do a quick count of how many failed SSH login attempts you had on your server last week? Its very common to see hundreds, if not thousands of attempts in a very short period of time. Assuming you cannot use a firewall to restrict SSH access to only authorized IP addresses, how do you mitigate these brute force attacks?

There are many tools out there to help with this. One I like is fail2ban. This program scans through log files and takes action against events such as repeated failed login attempts, and blocks the offending IP address for a set period of time.

Procedure

On CentOS systems, fail2ban can be installed from the EPEL repositories. If you do not have EPEL installed, you can get it setup by:

CentOS 5

rpm -ivh http://archives.fedoraproject.org/pub/archive/epel/5/x86_64/epel-release-5-4.noarch.rpm

CentOS 6

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

CentOS 7

rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm

Now install fail2ban:

yum install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Now customize /etc/fail2ban/jail.local accordingly for your server. Posted below are some more commonly configured options for CentOS 6 servers. The defaults (at the time of this writing) should protect SSH by banning any 3 or greater failed login attempts for 5 minutes via iptables. So the defaults should be okay, but you may want to consider adding your workstations IP address to the ignore ip list below so you don’t lock yourself out by accident!

vi /etc/fail2ban/jail.local

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8

# "bantime" is the number of seconds that a host is banned.
bantime  = 600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 3

[sshd]

# To use more aggressive sshd filter (inclusive sshd-ddos failregex):
#filter = sshd-aggressive
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

Finally, set fail2ban to start at boot, and start service:

chkconfig fail2ban on
service fail2ban start

As a quick side note, sometimes hosting providers will automatically install fail2ban for you. And depending on the host, they may configure it in such a way that it sends an email each time an IP address gets banned from SSH. This can quickly create a flood of email or email failures, especially if its not configured for a real email address.

If you have having issues like this and your fail2ban configuration was set to email you, you can prevent fail2ban from sending you emails for SSH bans by removing the line from the ssh-iptables block:

sendmail-whois[name=SSH, dest=root, [email protected]]

As a live example assuming it was previously configured, here is what it would look like before you make the change:

vim /etc/fail2ban/jail.conf
...
[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
           sendmail-whois[name=SSH, dest=root, [email protected]]
logpath  = /var/log/secure
maxretry = 5

And here is what it would look like after you make the change:

vim /etc/fail2ban/jail.conf
...
[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
logpath  = /var/log/secure
maxretry = 5

And be sure to restart fail2ban after the configuration update has been completed:

service fail2ban restart

Rackspace Cloud API – Enable SSL Termination on existing cloud load balancer

The purpose of this post is to show how you can enable SSL termination on an existing cloud load balancer through the API. Using the API will allow you to script deployments so you can avoid having to use the control panel. This also provides you the building blocks for understanding deployment automation.

This guide will show you how to enable SSL termination on an existing cloud load balancer as described in my previous post: Rackspace Cloud API – Create cloud load balancers

Feel free to review http://docs.rackspace.com for learning about all the possible operations that can be done through the API.

In this example, we are going to enable SSL termination on an existing cloud load balancer called lb.example.com. The domain we are load balancing is www.example.com. Please note that enabling SSL termination on a Rackspace cloud load balancer costs more then a regular cloud load balancer!

SPECIAL NOTE: I advise against using SSL termination if you are passing any PII (personally identifiable information) or other sensitive data through the cloud load balancer to the Cloud server. The transmission will only be encrypted from the clients browser to the load balancer. From there, the cloud load balancer will send the request in clear text through the rackspace network to your cloud server.

When working with the API, I like to use a tool called httpie to simplify things a bit. You can install this by:

yum install httpie

Now that we have httpie installed, lets get an auth token from the API:

echo '{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "YOUR_USERNAME","apiKey":"YOUR_API_KEY"}}}' | http post https://identity.api.rackspacecloud.com/v2.0/tokens

The token you need will be listed next to “id” field as shown below

        "token": {
            "expires": "2013-07-09T23:17:08.634-05:00", 
            "id": "2334aasdf5555j3hfhd22245dhsr", 
            "tenant": {
                "id": "123456", 
                "name": "123456"

To simplify things moving forward, we will set some local variables that we’ll use when communicating with the API:

export token="YOUR_API_TOKEN_RECEIVED_ABOVE"
export account="YOUR_RACKSPACE_CLOUD_ACCOUNT_NUMBER"
export lb_endpoint="https://ord.loadbalancers.api.rackspacecloud.com/v1.0"

NOTE: Change the endpoints region accordingly (ord or dfw).

For the purposes of this guide, we will be creating a self signed SSL certificate. In production environments, you will want to purchase a SSL certificate through a CA.

We will create our self signed SSL certificate by using the openssl command:

openssl req -x509 -nodes -newkey rsa:2048 -keyout example.com.tmp.key -out example.com.cert -days 1825
Generating a 2048 bit RSA private key
........+++
.....................................+++
writing new private key to 'example.com.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:California
Locality Name (eg, city) []: Los Angeles
Organization Name (eg, company) [Internet Widgits Pty Ltd]:My Company LLC
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:*.example.com
Email Address []:

Now we must covert the key so the API will be able to use it:

openssl rsa -in example.com.tmp.key -out example.com.key
rm -f example.com.tmp.key

Format the key and cert so it is more friendly for the API. Be sure to save the output as we will need it for the next step:

while read line; do echo -n "$line\n"; done < example.com.key
while read line; do echo -n "$line\n"; done < example.com.cert

Now, lets setup the json file that contains our certificate information:

cat << EOF > example.com.cert.json
{
   "certificate":"-----BEGIN CERTIFICATE-----\nMIIblahblahblahblah\n-----END CERTIFICATE-----",
   "enabled":true,
   "secureTrafficOnly":false,
   "privatekey":"-----BEGIN RSA PRIVATE KEY-----\nMIICWblahblahblahblah\n-----END RSA PRIVATE KEY-----",
   "intermediateCertificate":"",
   "securePort":443}
EOF

Finally, lets execute the json file to enable SSL on your pre-existing cloud load balancer:

http put $lb_endpoint/$account/loadbalancers/YOUR_CLOUD_LOAD_BALANCER_ID/ssltermination X-Auth-Token:$token @example.com.cert.json

SSL termination on your existing load balancer has now been enabled.

Rackspace Cloud API – Create cloud load balancers

The purpose of this post is to show how you can build Rackspace cloud load balancers using the API. Building via the API will allow you to script deployments so you can avoid having to use the control panel. This also provides you the building blocks for understanding deployment automation.

This guide will only show you how to create a cloud load balancer. Feel free to review http://docs.rackspace.com for learning about all the possible operations that can be done through the API.

In this example, we are going to deploy a single cloud load balancer called lb.example.com that will be directing HTTP traffic to two web servers:

test01.example.com
test02.example.com

When working with the API, I like to use a tool called httpie to simplify things a bit. You can install this by:

yum install httpie

Now that we have httpie installed, lets get an auth token from the API:

echo '{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "YOUR_USERNAME","apiKey":"YOUR_API_KEY"}}}' | http post https://identity.api.rackspacecloud.com/v2.0/tokens

The token you need will be listed next to “id” field as shown below

        "token": {
            "expires": "2013-07-09T23:17:08.634-05:00", 
            "id": "2334aasdf5555j3hfhd22245dhsr", 
            "tenant": {
                "id": "123456", 
                "name": "123456"

To simplify things moving forward, we will set some local variables that we’ll use when communicating with the API:

export token="YOUR_API_TOKEN_RECEIVED_ABOVE"
export account="YOUR_RACKSPACE_CLOUD_ACCOUNT_NUMBER"
export lb_endpoint="https://ord.loadbalancers.api.rackspacecloud.com/v1.0"

NOTE: Change the endpoints region accordingly (ord or dfw).

Lets prep a json file that we’ll be using to build the cloud load balancer:

cat << EOF > lb.example.com.json
{
    "loadBalancer": {
        "name": "lb.example.com",
        "port": 80,
        "protocol": "HTTP",
        "virtualIps": [
            {
                "type": "PUBLIC"
            }
         ],
        "nodes": [
            {
                "address": "10.123.123.121",
                "port": 80,
                "condition": "ENABLED"
            }
        ]
    }
}
EOF

Now we execute the build by:

http post $lb_endpoint/$account/loadbalancers X-Auth-Token:$token @lb.example.com.json

This will return the VIP and id of the new load balancer as shown below:

                "address": "123.123.123.123", 
        "id": 123456, 
        "name": "lb.example.com", 

For the purposes of this guide, I left out adding the second server initially. So here is how we can add it to the load balancer so it will route traffic between test01.example.com and test02.example.com. First create a json file that contains test02.example.com:

cat << EOF > nodes.json
{"nodes": [
{
"address": "10.123.123.123",
"port": 80,
"condition": "ENABLED",
"type":"PRIMARY"
}
]
}
EOF

Now add this node to the load balancer:

http post $lb_endpoint/$account/loadbalancers/149275/nodes X-Auth-Token:$token @nodes.json

And your done! You can verify everything looks correct by running:

http get $lb_endpoint/$account/loadbalancers/YOUR_LOAD_BALANCER_ID X-Auth-Token:$token

Rackspace Cloud API – Create NextGen cloud servers

The purpose of this post is to show how you can build Rackspace Next Generation cloud servers using the API. Building via the API will allow you to script server builds so you can avoid having to use the control panel. This also provides you the building blocks for understanding deployment automation.

This guide will only show you how to create a cloud server. Feel free to review http://docs.rackspace.com for learning about all the possible operations that can be done through the API.

In this example, we are going to build 2 512M CentOS 6.4 Cloud Servers via the Rackspace Cloud API. The servers will be named:

test01.example.com
test02.example.com

When working with the API, I like to use a tool called httpie to simplify things a bit. You can install this by:

yum install httpie

Now that we have httpie installed, lets get an auth token from the API:

echo '{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "YOUR_USERNAME","apiKey":"YOUR_API_KEY"}}}' | http post https://identity.api.rackspacecloud.com/v2.0/tokens

The token you need will be listed next to “id” field as shown below

        "token": {
            "expires": "2013-07-09T23:17:08.634-05:00", 
            "id": "2334aasdf5555j3hfhd22245dhsr", 
            "tenant": {
                "id": "123456", 
                "name": "123456"

To simplify things moving forward, we will set some local variables that we’ll use when communicating with the API:

export token="YOUR_API_TOKEN_RECEIVED_ABOVE"
export account="YOUR_RACKSPACE_CLOUD_ACCOUNT_NUMBER"
export endpoint="https://ord.servers.api.rackspacecloud.com/v2/"

NOTE: Change the endpoints region accordingly (ord or dfw).

Now, lets see what images are available. I’m looking for a CentOS 6.4 image:

http get $endpoint/$account/images/detail X-Auth-Token:$token

The id of the CentOS 6.4 image in this case is:

            "id": "e0ed4adb-3a00-433e-a0ac-a51f1bc1ea3d", 

We wanted a 512M server, so we must find the flavors id:

http get $endpoint/$account/flavors X-Auth-Token:$token

This shows that the 512M flavor has the id of:

            "id": "2", 

All the information has been collected. Time to prep 2 json files that we’ll be using to build the 2 servers:

cat << EOF > test01.example.com.json
{
    "server" : {
        "name" : "test01.example.com",
        "imageRef" : "e0ed4adb-3a00-433e-a0ac-a51f1bc1ea3d",
        "flavorRef" : "2"
    }
}
EOF

cat << EOF > test02.example.com.json
{
    "server" : {
        "name" : "test02.example.com",
        "imageRef" : "e0ed4adb-3a00-433e-a0ac-a51f1bc1ea3d",
        "flavorRef" : "2"
    }
}
EOF

Finally, we have everything we need to begin the builds. Execute the build by:

http post $endpoint/$account/servers @test01.example.com.json X-Auth-Token:$token
http post $endpoint/$account/servers @test02.example.com.json X-Auth-Token:$token

When you run each POST statement above, 2 fields will be returned by the API. Be sure to record these somewhere:
– adminPass : This is your servers root password
– id : This is your servers id number that will be referenced next.

A new server is not useful without knowing its IP address. After a few minutes pass, you can retrieve the IP address by running:

http get $endpoint/$account/servers/YOUR_SERVER_ID X-Auth-Token:$token
http get $endpoint/$account/servers/YOUR_OTHER_SERVER_ID X-Auth-Token:$token

The 2 relevant fields you will need for this example are posted below:

        "accessIPv4": "123.123.123.123", 
                    "addr": "10.123.123.123", 

Now you can SSH into your server using the IP and admin password that have been returned by the API.