What’s new in Ansible 1.1 for AWS and Eucalyptus users?

I thought the Ansible 1.0 development cycle was busy but 1.1 is crammed full of orchestration goodness.  On Tuesday, 1.1 was released and you can read more about it here: http://blog.ansibleworks.com/2013/04/02/ansible-1-1-released/

For those working on AWS and Eucalyptus, 1.1 brings some nice module improvements as well as a new cloudformation and s3 module.  It’s great to see the AWS-related modules becoming so popular so quickly.  Here are some more details about the changes but you can find info in the changelog here: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md

Security group ID support

It’s now possible to specify the security group by its ID.  This is quite typically behaviour in EC2 and Eucalyptus will support this with the pending 3.3 release.  The parameter is optional.

VPC subnet ID

VPC users can now specify a subnet ID associated with their instance.

Instance state wait timeout

In 1.0 there was no way to specify how long to wait for instances to move to the running state (from pending).  This adds a wait timeout parameter which allows the user to specify how long to wait for the instances to move to running before failing.  Useful to stop a playbook blocking for a longer than desired amount of time.  Thanks to milan for this one 🙂

EC2 idempotency via client token

EC2 implements idempotency for run instance requests via a client token parameter.  This client token should be a unique ASCII string, as detailed here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html. It ensures that another request to launch instances with the same client token would result in an evaluation for idempotency of the request.  Rob Parrott added this feature.

Additional response elements in EC2 returns

The EC2 module returns some response elements which are attributes of the instance(s) you may have launched.  Harold Spencer added all of the attributes, so these can be accessed in plays for later use.

CloudFormation module

James Martin added a CloudFormation module which takes a JSON template to deploy a collection of resources in AWS.

S3 module

This basic S3 module allows a user to put an object into a bucket and then returns a download url.  Useful for stashing particular system info into S3 for archive, or as a different way to transfer files between hosts and groups.  This module aims to be idempotent with a state parameter.

EC2 EBS module (ec2_vol)

This isn’t in the changelog but is a new module for volume creation and attachment to instances.  Useful for supplementing the EC2 module in provisioning tasks but very basic.  It’s not idempotent (yet) and takes the instance ID, volume size, availability zone and device name as parameters.

A big thanks goes out to all the other contributors to the AWS modules who I’ve not explicitly mentioned.

Writing my first Ansible module: ec2_vol

As is probably quite evident, I’ve recently been using Ansible to deploy workloads into EC2 and Eucalyptus.  One of the ideas behind this is the convenience of being able to leverage the common API to achieve a hybrid deployment scenario.  Thanks to various folk (names mentioned in previous posts) we have a solid ec2 boto-based Python module for instance launching.   One thing I wanted to do when spinning up instances and configuring a workload was to add some persistent storage for the application.  To do this I had to create and attach a volume as a manual step or run a local_action against euca2ools.  I figured I could try to write my own module to practice a bit more python (specifically boto).  The result is something terribly (perhaps in more ways that one?) simple but I think this is a testament to just how easy it is to write modules for Ansible (p.s it doesn’t need to be Python) 😉

The resulting doc bits are here: http://ansible.cc/docs/modules.html#ec2-vol whilst the code lives here: https://github.com/ansible/ansible/blob/devel/library/ec2_vol

In a future post I’ll write a little bit about how it works, hopefully this can inspire other folks to try writing some additional EC2 modules for Ansible 🙂

It’s really very rudimentary but it does what it says on the tin.  Having spoken to Seth (and others) about this, some improvements for future versions would be:

– Make it idempotent.  In practical terms make it possible to run the module without changes to the system.  This would be possible via a state parameter (see other modules): present/absent.  The code could then check for a volume attached to a particular instance which is tagged with the specified tag.  If it doesn’t exist, add it.  If it does, pass over it.  I’d call this the ec2 volume passover module 😉

– Detect existing device mappings via instance.block_device_mapping and then adjust the attachment device appropriately.

I’ll probably attempt this in a couple of months. This would work fine under EC2 but tagging and *full* block device mapping support won’t be in Eucalyptus until version 3.3 (due end of Q2 this year).  See here for more details: https://eucalyptus.atlassian.net/browse/EUCA-4786 and https://eucalyptus.atlassian.net/browse/EUCA-2117.

As we continue to bring better feature compatibility with AWS, it makes these kind of hybrid things much easier 🙂  We’re also going to need to address some of the other AWS services with Ansible modules I think, like ELB, CloudWatch etc.

EBS Architecture (EDBP)

So, the second post in this series and now a look at Eucalyptus Elastic Block Storage (EBS) and the Eucalyptus Storage Controller (SC) component which handles this.

What does it do?

The Storage Controller sits at the same layer as the Cluster Controller (CC), each Eucalyptus Availability Zone (AZ) or Cluster will have its own CC and SC.  Within that AZ, the SC will provide EBS (Elastic Block Store) functionality via iSCSI (AoE is no longer used in 3.0 onwards) using the Linux Target Framework (tgt).  If you’re a subscription-paying customer you can also use the SAN adapter to have the Eucalyptus SC talk directly to your NetApp, Dell (and EMC with 3.2) array.  In turn the Node Controllers (NC) will also talk directly to your storage array.

EBS is core to EC2, it’s a feature which the vast majority compute users will use. It provides the capability for users to store persistent data (and snapshot that data to Walrus/S3 for long-term storage).  With Eucalyptus 3.0 users can now utilise EBS-backed instances, which are essentially boot-from-iSCSI virtual machines.  These virtual machines use an EBS volume for their root filesystem.

This post is a more in-depth look at best practices around storage controller configuration.

How does it work?

This is pretty well explained on the wiki page here but I’ll summarise in short for the benefit of readers.

An EBS volume is created by a number of steps, starting with messaging sent from the client to the CLC (Cloud Controller).  To then create the EBS volume, the SC performs the following steps:

  1. SC creates a volume file in /var/lib/eucalyptus/volumes named per the volume ID
  2. This file is then attached to a free loopback device
  3. A logical volume is created on top of this loopback
  4. An iSCSI target is created along with a new LUN with the backing store of this logical volume

Then, when a user wishes to attach this EBS volume to a running instance, the NC on which the instance resides will attempt to login to this iSCSI target and pass the block device from the LUN through to the virtual machine based on an XML definition file with a “virsh attach-device”.

The SC also facilitates point-in-time snapshots of EBS volumes.  This involves the SC copying the EBS volume to S3 for long-term persistence. From this snapshot, users can register boot-from-EBS  (bfEBS) images and create new volumes.

During the snapshot process the SC does the following:

  1. Creates a new raw disk image file in /var/lib/eucalyptus/volumes
  2. Adds this as a physical volume to the volume group of the EBS volume
  3. Extends the volume group over this new physical volume
  4. Create a new Logical Volume and dd the contents of the volume into this LV

After the copy is complete, the SC will then transfer the contents of the EBS volume up to S3/Walrus.

How should I architect EBS storage?

It’s quite common that the storage controller (without SAN adapter) can be the primary factor in deciding whether to scale out with multi-cluster in deployments.  In part this is down to the usage profile of EBS and the architectural design most users follow.  When designing the storage aspect of the cloud there are a number of key areas on which to focus.

Usage Profile

What is the usage profile for the cloud?  Are EBS volumes used at all?  Are EBS volumes a key infrastructure component of workloads? How many users are on the cloud? How many concurrent volume attachments can you expect?

These are all valid questions when designing the storage architecture.  Concurrent and heavy use of EBS volumes may dictate very different backend requirements to a cloud where EBS volumes are only lightly used.  Make sure you test at the planned or envisaged scale.

Component Topology

Always keep the SC on a separate host from the Cluster Controller (CC) or any other Eucalyptus component if you can.  This has the potential to dramatically improve performance for even the smallest deployments.

Disk Subsystem & Filesystem

With the overlay storage backend, volumes and snapshots are stored in a flat directory structure in /var/lib/eucalyptus/volumes.

You need to make sure you choose a fast disk subsystem for the storage controller.  If you’re using local disks, consider RAID levels with some form of striping, such as RAID1+0 on as many spindles as possible.  If you’re backing /var/lib/eucalyptus/volumes with some form of networked storage, avoid NFS and CIFS.  Choose iSCSI or Fibre Channel where possible for best performance under high utilization across high numbers of active EBS volumes.  For ultimate performance, consider SSD’s and SSD arrays.  If you’re planning on using a remote backing store, like iSCSI, consider optimising with jumbo frames and iSCSI TCP-offloading on the NIC, if supported.

With 3.2 the DASManager storage backend is now open sourced.  Unlike the typical overlay backend the DASManager directly carves up presented block storage with LVM, circumventing the requirement for loopbacks and thus removing the limit of 255 volumes+snapshots*.  When not using the SAN adapter for NetApp, Dell or EMC arrays, the DASManager should be the preferred choice for performance and additional scalability.

As noted in the wiki referred to previously the SC has been tested primarily with ext4, although any POSIX compliant filesystem should be fine.

* In Eucalyptus 3.0 and 3.1 the SC would loopback mount all inactive volumes and thus the limit of 256 volumes would be imposed (256 loopbacks in Linux).  With 3.2+ the overlay storage backend ensures that only active volumes are loopback mounted, so now users can have up to 256 in-use volumes or snapshots.


Don’t cross the streams!

Best possible scenario is to move all EBS traffic onto its own network segment, adding additional interfaces to both your NC’s and the SC and then registering the SC on the interface you wish to use for the storage traffic.  This will ensure that storage and data are segregated.  This should be a necessity if you really must have the CC and SC sharing a host.

Host System Tuning

The box on which the SC is running should have as much memory as possible, plenty for pagecache usage (write-caching).  If the inbound I/O from initiators cannot be written to disk fast enough, the pagecache is going to be important. Monitor the virtual memory subsystem at all times, using something like Ganglia, Nagios, collectl or collectd.

For RHEL hosts use tuned profiles to apply some generic tweaks.  For the SC, enterprise-storage is probably the most effective; adjusting vm.dirty_ratio upwards (the point at which processes start asynchronous writes to disk with pdflushd), setting deadline I/O scheduler and enabling transparent hugepage support.

Consider cache layers in the chain from initiator to the SC.  These can give misleading results during testing.  For example, writes from the instance will (by default unless cache=none) hit the host pagecache, followed by the tgt cache on the SC as well its the SC pagecache, followed by any cache layer for the backing of /var/lib/eucalyptus/volumes. So the instance itself may see very misleading performance figures for disk writes particularly.  Test the chain from initiators to SC under stress conditions.

iSCSI Tuning

By default iscsid may be considered to be quite aggressive in its timeouts.  On a congested network the last thing a user wants is the initiator logging out of a session.  If bfEBS is being used, it’s probably a good idea to back off on some of the timeouts, consider changing the following:

node.conn[0].timeo.noop_out_interval = 0 <- this stops the "ping" between initator and target
node.conn[0].timeo.noop_out_timeout = 0 <- this disables the action of timing out an operation from the above "ping"
node.session.timeo.replacement_timeout = 7200 <-  this sets the connection timeout high, if running the entire root of the OS on the iscsi volume (bfEBS), be lazy.

Expectations & Remediation

Maybe this section should come first but expectations are key here. If you are going to install Eucalyptus on low-end server hardware, with slow disks and network, then don’t expect miracles in terms of concurrent EBS and bfEBS usage.  Performance may suck, YMMV.

On the other hand, perhaps you have optimised as best you can but short of getting new hardware and a new network, there is nothing more you can do to improve EBS performance in your current architecture.  At this point, consider using the following cloud properties to limit EBS volume sizes:

<partition_name>.storage.maxtotalvolumesizeingb <- sets the maximum total EBS volume size in the partition / cluster
<partition_name>.storage.maxvolumesizeingb <- sets the max per-volume size in the partition / cluster

Futhermore, utilise EIAM (Eucalyptus Identitiy and Access Management) quotas to mask over weak points in your platform architecture by placing restrictions on volume sizes and numbers of volumes users can have.  This is an easy way to limit “abuse” of the storage platform.  You can find some sample quotas here.


Following on from my first post on design, the key here is to nail the requirements and usage profile before designing the EBS architecture and always, always monitor.

If you have any comments or requests, please reply to this post and I’ll try to address them 🙂

Creating a Boot From EBS (bfEBS) image

Hi all,

I recently spent some time investigating this with my good colleague Andy Grimm. I had some initial issues in creating a RHEL6 image for KVM which would boot from EBS. Boot from EBS (sometimes referred to as bfEBS) is a new feature in Eucalyptus 3.X which allows a cloud administrator to back an instance with an EBS volume, thus giving persistence in the VM. This feature is present in AWS.

Below are some of my steps I used to create a RHEL6 guest image as a bfEBS volume. Note that I was using RHEL6 + KVM as the hypervisor:

1) Stop eucalyptus-nc on a designated host (or use another KVM host somewhere)
2) On the designated host, use qemu-img create to create a disk image:

qemu-img create -f raw rhel6_bfebs.img 2G

3) Set the disk label using parted: parted rhel6_bfebs.img mklabel msdos
4) Use virt-install to kick off a from-scratch installation:

virt-install --name rhel6-bfebs2 --ram 1024 --os-type linux --os-variant rhel6 \
-l \
path=/tmp/rhel6_bfebs.img,bus=virtio --vnc -x "lang=en_US keymap=us"

In your kickstart .cfg :

clearpart --all --drives=vda  --initlabel
 part / --size=512 --grow --fstype ext3 --ondisk=vda

Tweak as required 🙂

5) Complete the install then using virt-manager or other libvirt tool, start up the guest again (since finishing the install process is sometimes likely to shut it down, rather than reboot due to power mgmt).
6) Inside the guest:

– Remove the HWADDR line from /etc/sysconfig/network-scripts/ifcfg-eth0 <- this strips out the MAC address
– In /boot/grub/menu.lst for kernel option pass: console=ttyS0 at the end <- this will give you a serial console
– In /boot/grub/menu.lst also remove quiet option from kernel params and the grub menu splash image
– Edit /etc/sysconfig/network and add NOZEROCONF=yes  (disable the zeroconf route, which can mess up access to metadata service)
– Edit /etc/udev/rules.d/70-persistent-net.rules and remove the entry for existing NIC completely
Then, don’t forget to use our rc.local from here:  https://github.com/EucalyptusSystems/Eucalyptus-Scripts/blob/master/rc.local

7) On the instance, you can always mount the volume and check stuff:

hdparm -z /dev/vdX (re-read partition table)
 mount /dev/vdX /mnt/test

8) Make sure you unmount the EBS volume, if attached. Then detach the volume from the instance:

euca-detach-volume <volumeID>

9) Now snapshot the volume:


10) Register the snapshot (after it has completed):

euca-register -n "MyRHEL6bfEBS" -s <volumeID>

Done! Run your EBS-backed instance. Make sure that when you want to stop/start that you use the following commands:


Otherwise, unless you’re using image attributes you’ll end up with the underlying volume being destroyed when you use euca-terminate-instances.  There goes your persistence 😉

Creating & Managing EBS Volumes

I started making some notes on how EBS volume creation can be tracked and troubleshot in Eucalyptus (Eucalyptus 3.X + btw).  I figured this would make a good blog post, so I’ve just dumped all the information into here and I’ll let Google do the rest 😉  I’ll update it as I go along and provide more hints in the troubleshooting section, this will also make it into the Eucalyptus Knowledge Base, albeit in different form and smaller chunks.  I hope this is useful to some.

Creating & Managing Elastic Block Storage (EBS) Volumes

This guide illustrates for cloud administrators how the process of volume creation works, how volumes can be managed and how volumes can be troubleshooted.
The Volume Creation Process
Volume creation is fairly straightforward, there are a number of steps in the process between issuing a euca-create-volume command and the configuration of the iscsi target.
To create a volume, one would issue a euca-create-volume command:

# euca-create-volume -z eucalyptus -s 5

This tells Eucalyptus to create a volume in the specified availability zone (use euca-describe-availability-zone to show zones) and with a size of 5GB.
At this point, Eucalyptus will communicate with the storage controller (SC) component.  The SC process creates a sparse image file in /var/lib/eucalyptus/volumes:

# ls -lsrth /var/lib/eucalyptus/volumes/
total 1.1M
1.1M -rw-r–r– 1 root root 5.1G Jan 23 08:57 vol-391D44BE

Next, the SC will create a loop device and loopback mount this image file.  Use the following command to view the loop device configuration:

# losetup -a
/dev/loop0: [fd00]:394108 (//var/lib/eucalyptus/volumes/vol-391D44BE)

Then, this device is controlled by the Logical Volume Manager (LVM); it’s added as a physical volume, a volume group is created and a logical volume from that.  Once again, this can be seen using the LVM commands:

# pvdisplay
— Physical volume —
PV Name               /dev/loop0
VG Name               vg-1lS3pg..
PV Size               5.00 GB / not usable 4.00 MB
Allocatable           yes (but full)
PE Size (KByte)       4096
Free PE               0
Allocated PE          1280
PV UUID               u8vTQh-pEhU-9ID2-ewEO-kOIh-iwQx-Kb7BrZ
# vgdisplay
— Volume group —
VG Name               vg-1lS3pg..
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  2
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               1
Max PV                0
Cur PV                1
Act PV                1
VG Size               5.00 GB
PE Size               4.00 MB
Total PE              1280
Alloc PE / Size       1280 / 5.00 GB
Free  PE / Size       0 / 0
VG UUID               eMYWz3-DkhN-IM9I-8T6j-cbnI-vAJe-H9OJgh

  Total PE              1280

and finally:

# lvdisplay
— Logical volume —
LV Name                /dev/vg-1lS3pg../lv-28NzTQ..
VG Name                vg-1lS3pg..
LV UUID                Goyv51-3WRJ-GOiq-yrjU-osw8-efqF-UGQt60
LV Write Access        read/write
LV Status              available
# open                 1
LV Size                5.00 GB
Current LE             1280
Segments               1
Allocation             inherit
Read ahead sectors     auto
– currently set to     256
Block device           253:3

With the logical volume created on the loopback mounted image, this then needs to be made available to our node controller which is running the instance on which we want to attach the EBS volume.  In Eucalyptus 2 and 3, the default storage networking method is iSCSI.  To provide iSCSI storage networking, Eucalyptus uses the Linux SCSI target framework (tgt).   Available in the base repositories of all major distributions, it is installed as a dependency for the Eucalyptus SC.
Eucalyptus configures the SCSI target daemon to run on the SC, configuring targets for each EBS volume which is configured.
Using the tgt-admin utility it’s possible to view the currently configured iSCSI targets:

# tgt-admin -s
Target 4: iqn.2009-06.com.eucalyptus.eucalyptus:store4
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET     00040000
SCSI SN: beaf40
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET     00040001
SCSI SN: beaf41
Size: 5369 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/vg-1lS3pg../lv-28NzTQ..
Backing store flags:
Account information:
ACL information:

Note the target name, which is a useful identifier.  Also note the backing store path, which happens to be the logical volume the SC has configured.  Visible for each target is the account information, Eucalyptus uses a non-root account for security reasons.

Attaching Volumes to Instances

At this point, the user may attach the volume to a running instance.  Use the following command to view your available volumes:

# euca-describe-volumes

Then, to attach the desired volume:

# euca-attach-volume -i i-133F3E53 -d /dev/sdb1 vol-391D44BE

This will attach the designated volume (vol-391D44BE) to the instance with ID i-133F3E53, as the device /dev/sdb1.
To cover what happened at this point, transition onto the Node Controller (NC) hosting the instance to which the EBS volume was attached.
Use the open ISCSI administration utility to query the target daemon on the SC and view the published targets:

# iscsiadm -m discovery -t sendtargets -p,1 iqn.2009-06.com.eucalyptus.eucalyptus:store4

Note the target LUN is visible to the NC.
Eucalyptus instructs the NC to connect to the target.  This is visible in /var/log/messages on the NC:

Jan 23 11:03:31 Pod-04 kernel: scsi3 : iSCSI Initiator over TCP/IP
Jan 23 11:03:32 Pod-04 kernel:   Vendor: IET       Model: Controller        Rev: 0001
Jan 23 11:03:32 Pod-04 kernel:   Type:   RAID                               ANSI SCSI revision: 05
Jan 23 11:03:32 Pod-04 kernel: scsi 3:0:0:0: Attached scsi generic sg5 type 12
Jan 23 11:03:32 Pod-04 kernel:   Vendor: IET       Model: VIRTUAL-DISK      Rev: 0001
Jan 23 11:03:32 Pod-04 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jan 23 11:03:32 Pod-04 kernel: SCSI device sdd: 10485760 512-byte hdwr sectors (5369 MB)
Jan 23 11:03:32 Pod-04 kernel: sdd: Write Protect is off
Jan 23 11:03:32 Pod-04 kernel: SCSI device sdd: drive cache: write back
Jan 23 11:03:32 Pod-04 kernel: SCSI device sdd: 10485760 512-byte hdwr sectors (5369 MB)
Jan 23 11:03:32 Pod-04 kernel: sdd: Write Protect is off
Jan 23 11:03:32 Pod-04 kernel: SCSI device sdd: drive cache: write back
Jan 23 11:03:32 Pod-04 kernel:  sdd: unknown partition table
Jan 23 11:03:32 Pod-04 kernel: sd 3:0:0:1: Attached scsi disk sdd
Jan 23 11:03:32 Pod-04 kernel: sd 3:0:0:1: Attached scsi generic sg6 type 0
Jan 23 11:03:33 Pod-04 kernel: peth0: received packet with  own address as source address
Jan 23 11:03:33 Pod-04 kernel: peth0: received packet with  own address as source address
Jan 23 11:03:33 Pod-04 iscsid: Connection2:0 to [target: iqn.2009-06.com.eucalyptus.eucalyptus:store4, portal:,3260] through [iface: default] is operational now

The resultant device is available as /dev/sdd on the NC, visible with “fdisk -l”:

Disk /dev/sdd: 5368 MB, 5368709120 bytes
166 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 10292 * 512 = 5269504 bytes
Disk /dev/sdd doesn’t contain a valid partition table

Then, Eucalyptus generates an XML file for the volume in the working directory of the instance it will attach to.  Below is the instance working directory, as the ID suggests:

[root@Pod-04 i-133F3E53]# pwd

Below is the XML file for the volume:

[root@Pod-04 i-133F3E53]# ll vol*
-rw-rw—- 1 eucalyptus eucalyptus 535 Jan 23 11:03 vol-391D44BE.xml

This XML generated by Eucalpytus looks like:

<?xml version=”1.0″ encoding=”UTF-8″?>
<hypervisor type=”xen” capability=”xen+hw” bitness=”64″/>
<os platform=”linux” virtioRoot=”false” virtioDisk=”false” virtioNetwork=”false”/>
<root type=”image”/>
<diskPath targetDeviceType=”disk” targetDeviceName=”sdb2″ targetDeviceBus=”scsi” sourceType=”block”>/dev/sdd</diskPath>

See the volume ID and target device name as specified with the euca-attach-volume command.
This is then added with a virsh command behind the scenes by Eucalyptus.  Note a manual method would be something like:

virsh attach-device <domain> <file>

Running the following command on the node controller will dump the current xml definition for the virtual machine:

virsh dumpxml <domain_id>

Use virsh list to obtain the domain ID for the instance.  The block device should be present in the <device> section:

<disk type=’block’ device=’disk’>
<driver name=’phy’/>
<source dev=’/dev/sdd’/>
<target dev=’sdb2′ bus=’scsi’/>

Thus, looking on the instance, the new volume is available as /dev/sdb2:

Disk /dev/sdb2: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Disk /dev/sdb2 doesn’t contain a valid partition table

At this point, the volume can be formatted and used by the instance:

# mkfs.ext3 /dev/sdb2
# mkdir /media/data
# mount /dev/sdb2 /media/data

Detaching Volumes

Detaching a volume follows much the same process but in reverse order.
WARNING:  You must unmount the device within the instance before detaching the volume, otherwise you risk data loss and being unable to properly detach the volume whilst the instance is running.
A volume can be detached with the following command:

euca-detach-volume -i i133F3E53 vol-391D44BE

This detaches the previously created volume from the running instance, i133F3E53.
Much the same as adding a block device to a running guest, this calls virsh commands, as below:

virsh detach-device <domain> <file>

Whereby domain is the ID of the running domain (which can be found by using ‘virsh list’) and file is the XML device definition file that Eucalyptus created at attach time, in this continued example it was vol-391D44BE.xml
This device defition file is then removed from the working directory of the instance on the node controller.

Snapshotting Volumes

An EBS volume snapshot acts as a point-in-time backup of your persistent storage into Walrus (S3), note that it is incremental in nature, only backing-up the changed data.  It can also be a method for duplicating EBS volumes; it is possible to snapshot a volume and recreate a new volume from the snapshot.
WARNING: As when detaching a volume, it is recommended that you unmount the volume within the instance before taking a snapshot.
To create a snapshot, use the euca-create-snapshot command:

euca-create-snapshot <volume_id>

Providing the volume ID, as shown with euca-describe-volumes as the arguement.  Next, view the status of the snapshot process with:

[root@Pod-03 lester]# euca-describe-snapshots
SNAPSHOT        snap-C2DF3CF2   vol-D02E43C2    pending 2012-01-30T15:44:04.853Z        0%      None

In this example, the snapshot is of a 15G EBS volume.
Eucalyptus EBS snapshots use Logical Volume Manager (LVM) copy-on-write (COW) snapshots to create the snapshot disk image.  Firstly the storage controller creates an image file large enough to add to the volume group for the current logical volume on which the current EBS volume resides.   This additional image is created in the volumes directory on the storage controller:

[root@Pod-03 lester]# ll /var/lib/eucalyptus/volumes/
total 17015932
-rw-r–r– 1 root root  8589934592 Jan 30 07:32 snap-26EE3E3A
-rw-r–r– 1 root root  3224145920 Jan 30 07:44 snap-C2DF3CF2
-rw-r–r– 1 root root  5368709120 Jan 30 06:49 snap-D7E53AC0
-rw-r–r– 1 root root 10741612544 Jan 20 09:56 vol-178C3E9C
-rw-r–r– 1 root root  5372903424 Jan 23 08:57 vol-391D44BE
-rw-r–r– 1 root root 16110321664 Jan 30 07:43 vol-D02E43C2
-rw-r–r– 1 root root  8053063680 Jan 30 07:44 vol-D02E43C2MsmGAIzx

This is then loopback mounted:

[root@Pod-03 volumes]# losetup -a
/dev/loop0: [fd00]:393652 (//var/lib/eucalyptus/volumes/vol-178C3E9C)
/dev/loop1: [fd00]:394108 (//var/lib/eucalyptus/volumes/vol-391D44BE)
/dev/loop2: [fd00]:393631 (//var/lib/eucalyptus/volumes/vol-D02E43C2)
/dev/loop3: [fd00]:394121 (//var/lib/eucalyptus/volumes/vol-D02E43C2MsmGAIzx)

Marked as a physical volume for LVM and added to the volume group, which now shows increased size:

[root@Pod-03 volumes]# vgdisplay
— Volume group —
VG Name               vg-WsNskQ..
System ID
Format                lvm2
Metadata Areas        2
Metadata Sequence No  5
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                2
Open LV               2
Max PV                0
Cur PV                2
Act PV                2
VG Size               22.50 GB
PE Size               4.00 MB
Total PE              5759
Alloc PE / Size       5759 / 22.50 GB
Free  PE / Size       0 / 0
VG UUID               Dh0Xs7-Pvlr-vUbm-d1mA-7pCf-I94z-dS3r0l

lvdisplay shows the status of the logical volumes:

[root@Pod-03 volumes]# lvdisplay
— Logical volume —
LV Name                /dev/vg-WsNskQ../lv-OWzOIQ..
VG Name                vg-WsNskQ..
LV UUID                JFyFII-F1K8-QGYD-0s77-1dZf-ALQm-33AOr9
LV Write Access        read/write
LV snapshot status     source of
/dev/vg-WsNskQ../lv-snap-CfzmRA.. [active]
LV Status              available
# open                 1
LV Size                15.00 GB
Current LE             3840
Segments               1
Allocation             inherit
Read ahead sectors     auto
– currently set to     256
Block device           253:4

— Logical volume —
LV Name                /dev/vg-WsNskQ../lv-snap-CfzmRA..
VG Name                vg-WsNskQ..
LV UUID                0S4AAC-g4ip-0Zjs-0dQV-Yfym-tTNA-tPV1rA
LV Write Access        read/write
LV snapshot status     active destination for /dev/vg-WsNskQ../lv-OWzOIQ..
LV Status              available
# open                 1
LV Size                15.00 GB
Current LE             3840
COW-table size         7.50 GB
COW-table LE           1919
Allocated to snapshot  0.00%
Snapshot chunk size    4.00 KB
Segments               1
Allocation             inherit
Read ahead sectors     auto
– currently set to     256
Block device           253:5

Note the COW-table size and logical volume size equals that of the underlying physical volume.  This additional physical volume in the volume group is needed to hold the snapshot logical volume.
For more information on how LVM snapshotting works, see the TLDP guide: http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html
From this snapshot the data is then copied into a snapshot disk image in /var/lib/eucalyptus/volumes/.  The process can be viewed using ps -aux on the storage controller:

dd if /dev/vg-WsNskQ../lv-snap-CfzmRA.. of //var/lib/eucalyptus/volumes/snap-C2DF3CF2 bs 1M

Once complete, the storage controller cleans up the snapshot device, removes the physical volume and unmounts the loopback device and removes the temporary disk image.  The volumes directory now just shows the snapshot image:

[root@Pod-03 ~]# ll /var/lib/eucalyptus/volumes/
total 29607236
-rw-r–r– 1 root root  8589934592 Jan 30 07:32 snap-26EE3E3A
-rw-r–r– 1 root root 16106127360 Jan 30 07:48 snap-C2DF3CF2
-rw-r–r– 1 root root  5368709120 Jan 30 06:49 snap-D7E53AC0
-rw-r–r– 1 root root 10741612544 Jan 20 09:56 vol-178C3E9C
-rw-r–r– 1 root root  5372903424 Jan 23 08:57 vol-391D44BE
-rw-r–r– 1 root root 16110321664 Jan 30 07:43 vol-D02E43C2

Next, the snapshot is stored in Walrus (S3) as a snapset.  Check /var/log/eucalyptus/cloud-output.log for the transfer messages:

| <euca:StoreSnapshotType xmlns:euca=”http://msgs.eucalyptus.com”&gt;
|   <euca:WalrusDataRequestType>
|     <euca:WalrusRequestType>
|       <euca:correlationId>9105e2c8-b228-4dfa-b622-e384a232852f</euca:correlationId>
|       <euca:_return>true</euca:_return>
|       <euca:_services/>
|       <euca:_disabledServices/>
|       <euca:_notreadyServices/>
|       <euca:accessKeyID>KGPY0PMLTKX4XUORAC8IK</euca:accessKeyID>
|       <euca:timeStamp>2012-01-30T15:48:34.688Z</euca:timeStamp>
|       <euca:bucket>snapset-ccc92f77-ab62-4554-9151-973e45fcc974</euca:bucket>
|       <euca:key>snap-C2DF3CF2</euca:key>
|     </euca:WalrusRequestType>
|     <euca:randomKey>snapset-ccc92f77-ab62-4554-9151-973e45fcc974.snap-C2DF3CF2.JAoH7pTIt7gbzQ..</euca:randomKey>
|   </euca:WalrusDataRequestType>
|   <euca:snapshotSize>16106127360</euca:snapshotSize>
| </euca:StoreSnapshotType>

Note the snapshot reference in the key field.  This will be followed in the log (if successful) with:

INFO WalrusManager.putObject(WalrusManager.java):1020 | Transfer complete: snapset-ccc92f77-ab62-4554-9151-973e45fcc974.snap-C2DF3CF2

The snapset should appear in the Walrus (S3) bukkits directory:

[root@Pod-03 eucalyptus]# ll /var/lib/eucalyptus/bukkits/
total 24
drwxr-xr-x 2 eucalyptus eucalyptus 4096 Jan 30 05:05 euca-centos
drwxr-xr-x 2 eucalyptus eucalyptus 4096 Jan 16 06:50 euca-ubuntu
drwxr-xr-x 2 eucalyptus eucalyptus 4096 Jan 30 08:35 snapset-80efb05e-f0d7-444d-946d-9cfdad3845ad
drwxr-xr-x 2 eucalyptus eucalyptus 4096 Jan 30 06:51 snapset-ad471308-c3f9-4908-a4f0-1e8790d8b99f
drwxr-xr-x 2 eucalyptus eucalyptus 4096 Jan 30 09:08 snapset-ccc92f77-ab62-4554-9151-973e45fcc974
drwxr-xr-x 2 eucalyptus eucalyptus 4096 Jan 30 07:35 snapset-f394c343-2580-4406-a29b-b5b62b2b9fd1

WARNING:  Snapshots count towards a users quota in S3, keep this in mind when configuring the storage system.


Q1:  I’ve tried to detach a volume from my instance but it stays in the “detaching” state.
A1:  Ensure you unmount your volume from within the instance before detaching it, in this case you would need to terminate the instance to free your volume.
Q2:  My EBS snapshot is created on the storage controller but it’s not uploaded to Walrus.
A2:  Check /var/log/eucalyptus/cloud-error.log and cloud-output.log for errors.  You may be hitting your users quota, use the GUI or euca-describe-properties to check the quota limits for snapshots.