Synnefo » snf-image

h2. snf-image

!snf-image-logo.png!

h1. Warning: This wiki page is obsolete. The up-to-date documentation of snf-image is hosted "here":http://www.synnefo.org/docs/snf-image/latest/

h2. Introduction - Features

snf-image is a "Ganeti":http://code.google.com/p/ganeti/ OS Definition, primarily used by "Synnefo":https://code.grnet.gr/projects/synnefo .

It is rewritten from scratch and allows Ganeti to launch instances from predefined or untrusted custom Images. The whole process of deploying an Image onto the block device, as provided by Ganeti, is done in complete isolation from the physical host, enhancing robustness and security.

snf-image supports KVM and Xen based ganeti clusters.

There are also additional hooks that can be enabled at image deployment. They allow for:

 * changing the password of root or arbitrary users

 * injecting files at arbitrary locations inside the filesystem, e.g., SSH keys

 * setting a custom hostname

 * re-creating SSH host keys to ensure the image uses unique keys

snf-image has been used successfully to deploy many major Linux distributions (Debian, Ubuntu/Kubuntu, CentOS, Fedora), Windows 2008 R2 & Windows Server 2012, as well as FreeBSD 9.1

The snf-image Ganeti OS Definition is released under [[Licence|GNU Generic Public Lisence version 2]].

h2. Ganeti OS Interface

snf-image requires ganeti-os-interface v20 to operate.

It introduces the following OS parameters:

 * @img_id@ (_required if @config_url@ is missing_): the URI used to identify the image (see [[Image IDs and Storage Back Ends|here]])

 * @img_format@ (_required if @config_url@ is missing_): the image format type (see [[Image Format|here]])

 * @img_passwd@ (_required if @config_url@ is missing_): the passwd to be injected inside the image

 * @img_properties@ (_optional_): additional image properties used to customize the image ([[Image_Format#Image-Properties|details]])

 * @img_personality@ (_optional_): files to be injected into the image filesystem ([[Personality OS Parameter|details]]).

 * @config_url@ (_optional_): the url to download configuration data from

h2. Architecture

snf-image is split in two components: The main program running on the Ganeti host, with full root privilege (@snf-image@ previously @snf-image-host@), and a part running inside an unprivileged, helper VM (@snf-image-helper@).

h3. snf-image

This part implements the Ganeti OS interface. It extracts the Image onto the Ganeti-provided block device, using streaming block I/O (@dd@ with @oflag=direct@), then passes control to @snf-image-helper@ running inside a helper VM. The helper VM is created using KVM, runs as an unprivileged user, @nobody@ by default.

There is no restriction on the distribution running inside the helper VM, as long as it executes the @snf-image-helper@ component automatically upon bootup. The @snf-image-update-helper@ script is provided with @snf-image@ to automate the creation of a helper VM image based on Debian Stable, using @multistrap@.

The @snf-image-helper@ component is spawned inside a specific hardware environment:

* The VM features a virtual floppy, containing an @ext2@ filesystem with all parameters needed for image customization.

* The hard disk of the VM being deployed is accessible as the first @virtio@ hard disk.

* All kernel/console output is redirected to the first virtual serial console, and eventually finds its way into the OS definition log files that Ganeti maintains.

* The helper VM is expected to output "SUCCESS" to its second serial port if image customization was successful inside the VM.

  In any other case, execution of the helper VM or @snf-image-helper@ has failed.

* The helper VM is expected to shutdown automatically once it is done. Its execution is time-limited; if it has not terminated after a number of seconds, configurable via @/etc/default/snf-image@, it is sent a @SIGTERM@ and/or a @SIGKILL@.

KVM is currently a dependency for @snf-image@, meaning it is needed to spawn the helper VM. There is no restriction on the hypervisor used for the actual Ganeti instances. This is not a strict requirement; KVM could be replaced by @qemu@, doing full CPU virtualization without any kernel support for spawning the helper VM.

h3. snf-image-helper

This part runs inside the helper VM and undertakes customization of the VM being deployed using a number of hooks, or _[[Configuration Tasks|tasks]]_. The tasks run in an environment, specified by rules found in a virtual floppy, placed there by @snf-image@. @snf-image-helper@ uses @runparts@ to run tasks found under @/usr/lib/snf-image-helper/tasks@ by default.

The architecture is presented below:

h3. Graphical Representation

!snf-image_arch.png!

h3. Image Names

snf-image assumes all image files have the following format:

<pre><image_id>.<image_format></pre>

In order to create an instance with ganeti using an image with filename @debian_base-6.0-5-x86_64.extdump@, you need to specify the following OS parameters:

<pre>img_format=extdump

img_id=debian_base-6.0-5-x86_64</pre>

h2. Download

h3. Source

The latest source packages for snf-image (v0.9.1) can be found here:

* "@snf-image_0.9.1.orig.tar.gz@":https://code.grnet.gr/attachments/download/2609/snf-image_0.9.1.orig.tar.gz

* "@snf-image-helper_0.9.1.orig.tar.gz@":https://code.grnet.gr/attachments/download/2606/snf-image-helper_0.9.1.orig.tar.gz

h3. Packages (for debian squeeze)

The latest debian package for snf-image (v0.9.1) can be found here:

* "@snf-image@":https://code.grnet.gr/attachments/download/2612/snf-image_0.9.1-1_all.deb

Previous versions and source tarballs can be found here:

https://code.grnet.gr/projects/snf-image/files

You can also download the package from our @apt@ repo:

http://apt2.dev.grnet.gr

h3. Fixed snf-image-helper image

If you installed snf-image using the provided package, you can create the helper VM by launching `snf-image-update-helper' command. People who installed the program from source may use the fixed helper VM image provided below:

* "@snf_image_helper_image-0.9.1-1.tar.xz@":https://pithos.okeanos.grnet.gr/public/euharYruWl7s0C3KfdC2I4

h3. Sample Images

While developing snf-image, we created and tested a number of images. The following images are basic installations of some popular Linux distributions, that have been tested with snf-image and provided here for testing purposes:

* Debian Squeeze Base System ["diskdump":https://pithos.okeanos.grnet.gr/public/RDISy7sNVIJ9KIm4JkmbX4] ["md5sum":https://pithos.okeanos.grnet.gr/public/Gal05XiqgxZKXJcLhkvUy5] ["metadata":https://pithos.okeanos.grnet.gr/public/alJm9TMz5dBwzOfrgyIZY2]

* Debian Wheezy Base System ["diskdump":https://pithos.okeanos.grnet.gr/public/q9Gxb9htMMfAVI3i4lN2V1] ["md5sum":https://pithos.okeanos.grnet.gr/public/OZofrTvsyKVw0ca1uVcqa6] ["metadata":https://pithos.okeanos.grnet.gr/public/5gMeRCqtMxl5hpGvfqE4E7]

* Debian Desktop ["diskdump":https://pithos.okeanos.grnet.gr/public/HI69qEBrLuCTKuNzF3ENi3] ["md5sum":https://pithos.okeanos.grnet.gr/public/I10MuN7BD9SqLGhaPVwT07] ["metadata":https://pithos.okeanos.grnet.gr/public/2cKXF26sIkWzBf4g12ds2]

* CentOS 6.0 ["diskdump":https://pithos.okeanos.grnet.gr/public/dSGXshb1uLx7vPkkHmyFp2] ["md5sum":https://pithos.okeanos.grnet.gr/public/5xsChidebUFSnPWB6N9rF2] ["metadata":https://pithos.okeanos.grnet.gr/public/C1kmDkcxoorkkTORlLf3n3]

* Fedora Desktop 18 ["diskdump":https://pithos.okeanos.grnet.gr/public/R3R6LGH2A2AgAaK5yFqug2] ["md5sum":https://pithos.okeanos.grnet.gr/public/wY6TfJzyUiTBLXyI2mnAU5] ["metadata":https://pithos.okeanos.grnet.gr/public/bsHhw6PpR4IfGZESNGcU06]

* Ubuntu Desktop LTS 12.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/PDfxRVFDQBb3fYnzi5B3h2] ["md5sum":https://pithos.okeanos.grnet.gr/public/qY3FWSlrLBnywolEXSLpw4] ["metadata":https://pithos.okeanos.grnet.gr/public/RJ3V0uF5b6KW8IBq88ygs5]

* Kubuntu LTS 12.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/bLC84nLirVxBSDSqZFvs94] ["md5sum":https://pithos.okeanos.grnet.gr/public/ePLgfbNqfh4qo5lVC300E6] ["metadata":https://pithos.okeanos.grnet.gr/public/43c88oKyBUjzRpG9X0OFN5]

* Ubuntu Desktop 13.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/n1XJ61tOX0VFvng4syVPt4] ["md5sum":https://pithos.okeanos.grnet.gr/public/hWlABUDQIXDBFaqHu3ett2] ["metadata":https://pithos.okeanos.grnet.gr/public/4BdfbnjeuGKDHGnyjuN9v2]

* Kubuntu 13.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/GjsC1FHYX4qaBdkS0aiNm4] ["md5sum":https://pithos.okeanos.grnet.gr/public/JT0kdUGb3RS84FX3tzBfm4] ["metadata":https://pithos.okeanos.grnet.gr/public/wXE4aI9tfFPMq2FZ64pe3]

* Ubuntu Server 12.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/pvyxBcYA6MwSRYtMKGDZa5] ["md5sum":https://pithos.okeanos.grnet.gr/public/JLg9Zagnas6plmjlacZA54] ["metadata":https://pithos.okeanos.grnet.gr/public/HUoa4AJyQaR45ObAz67Uu1]

* OpenSUSE Desktop 12.3 ["diskdump":https://pithos.okeanos.grnet.gr/public/Xvc55lG8wbSPXOkQOKVvW1] ["md5sum":https://pithos.okeanos.grnet.gr/public/PNc2itzHxpW4cicveGEiL4] ["metadata":https://pithos.okeanos.grnet.gr/public/VFRwOkuloIZQgshgAQwkk3]

* FreeBSD 9.1 ["diskdump":https://pithos.okeanos.grnet.gr/public/Ol6XlgO9BoJmByO7EeJl33] ["md5sum":https://pithos.okeanos.grnet.gr/public/pFxjb9WrfOLOELvWaR0bx] ["metadata":https://pithos.okeanos.grnet.gr/public/9bHZmQcAnND9gY9vW4x1z6]

After getting familiar with snf-image you can build your own Images ([[Image Format|more...]]), or build upon the ones listed above.

h2. Installation

Before installing snf-image be sure to have a working Ganeti installation in your cluster. The installation process should take place in *all* ganeti nodes. Here we will describe the installation in a single node. The process is identical for all nodes and should be repeated manually or automatically, e.g., with puppet.

h3. Installing snf-image using @apt@ (Debian-based distributions)

Add the following lines in your @sources.list@ file:

<pre>

deb http://apt2.dev.grnet.gr stable/

deb-src http://apt2.dev.grnet.gr stable/

</pre>

and then run:

<pre>

# apt-get update

# apt-get install snf-image

# snf-image-update-helper

</pre>

The last command may take a while. If it exits without errors, you have installed snf-image successfully and you can move on to the configuration section.

h3. Installing snf-image from source

# Download the provided source package:<pre>$ wget https://code.grnet.gr/attachments/download/2609/snf-image_0.9.1.orig.tar.gz</pre>

# Untar, configure and compile the source:<pre>$ tar -xvf snf-image_0.9.orig.tar.gz

$ cd snf-image-0.9.1

$ ./autoget.sh

$ ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var

$ make</pre>

# Install snf-image:<pre>$ make install

$ install -Dm600 defaults /etc/default/snf-image</pre>

# Download the fixed [[wiki#Fixed-snf-image-helper-image|snf-image-helper appliance]]

# Extract it's content under @/var/lib/snf-image/helper/@:<pre>$ mkdir -p /var/lib/snf-image/helper

$ cd /var/lib/snf-image/helper

$ tar -xvf /place/where/you/stored/snf_image_helper_image-0.9.1-1.tar.xz</pre>

You have now installed successfully snf-image and the helper VM. You should now configure snf-image before it is ready for use.

h2. Configuration

In this section, we assume that the image file you want to deploy resides under a local directory (e.g.: @/srv/myimages/@)

Once you have installed @snf-image@ in the Ganeti node, you need to overwrite the value of the corresponding option in @/etc/default/snf-image@:

<pre>

IMAGE_DIR = /srv/myimages/

</pre>

@IMAGE_DIR@ should point to the location where the image files are stored [default: @/var/lib/snf-image@].

h2. Sample Usage

h3. Download Image

Download a [[Wiki#Sample-Images|Sample Image]] and store it under @IMAGE_DIR@. Make sure you also have its corresponding metadata file.

h3. Spawn a @diskdump@ image

If you want to deploy an image of type @diskdump@, you need to provide the corresponding @img_properties@ as described in the [[Image_Format#OS-parameter-img_properties|Image Format]] section. If using a diskdump found in the [[Wiki#Sample-Images|Sample Images]] list, use the @img_properties@ described in the image's metadata file. For example:

<pre>gnt-instance add -o snf-image+default --os-parameters img_passwd=SamplePassw0rd,img_format=diskdump,img_id=debian_base-6.0-7-x86_64,img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"}' -t plain --disk=0:size=10G --no-name-check --no-ip-check --no-nics test1</pre>

h3. Spawn an @extdump@/@ntfsdump@ image

To create a ganeti instance with snf-image using @debian_base-6.0-5-x86_64.extdump@ hosted under @IMAGE_DIR@, use a command like the one below:

<pre>gnt-instance add -o snf-image+default --os-parameters img_passwd=SamplePassw0rd,img_format=extdump,img_id=debian_base-6.0-5-x86_64 -t plain --disk=0:size=10G --no-name-check --no-ip-check --no-nics test1</pre>

h2. Advanced Usage

If you want to learn more and use the advanced features of snf-image please refer to following sections:

# [[Configuration Tasks|Image Configuration Tasks]]

# [[Image Format|Supported Image formats]]

# [[Image Store|Supported Image store locations]]

# [[PMI|Progress Monitor Interface]]

h2. Developers

If you are a developer or want to use the latest source code you can download the latest development version by cloning the snf-image git repository:

<pre>$ git clone https://code.grnet.gr/git/snf-image</pre>

The _master_ branch contains the latest development version

The _debian_ branch contains the _master_ plus the debian packaging directories.

For more information on how to configure, install or package from source, see the [[Developer|Developer's page]].

h2. Community & Support

Your help is very important. Any contributions and bug reports will be highly appreciated. You can contact the team at:

* Bug reports - feedback - support: synnefo@googlegroups.com