Wiki

Version 132 (Nikos Skalkotos, 07/16/2013 12:37 pm)

1 21 Constantinos Venetsanopoulos
h2. snf-image
2 21 Constantinos Venetsanopoulos
3 21 Constantinos Venetsanopoulos
!snf-image-logo.png!
4 1 Constantinos Venetsanopoulos
5 10 Vangelis Koukis
h2. Introduction - Features
6 10 Vangelis Koukis
7 54 Constantinos Venetsanopoulos
snf-image is a "Ganeti":http://code.google.com/p/ganeti/ OS Definition, primarily used by "Synnefo":https://code.grnet.gr/projects/synnefo .
8 2 Constantinos Venetsanopoulos
9 101 Nikos Skalkotos
It is rewritten from scratch and allows Ganeti to launch instances from predefined or untrusted custom Images. The whole process of deploying an Image onto the block device, as provided by Ganeti, is done in complete isolation from the physical host, enhancing robustness and security.
10 101 Nikos Skalkotos
11 101 Nikos Skalkotos
snf-image supports KVM and Xen based ganeti clusters.
12 1 Constantinos Venetsanopoulos
13 7 Vangelis Koukis
There are also additional hooks that can be enabled at image deployment. They allow for:
14 1 Constantinos Venetsanopoulos
15 7 Vangelis Koukis
 * changing the password of root or arbitrary users
16 7 Vangelis Koukis
 * injecting files at arbitrary locations inside the filesystem, e.g., SSH keys
17 7 Vangelis Koukis
 * setting a custom hostname
18 7 Vangelis Koukis
 * re-creating SSH host keys to ensure the image uses unique keys
19 1 Constantinos Venetsanopoulos
20 120 Nikos Skalkotos
snf-image has been used successfully to deploy many major Linux distributions (Debian, Ubuntu/Kubuntu, CentOS, Fedora), Windows 2008 R2 & Windows Server 2012, as well as FreeBSD 9.1
21 7 Vangelis Koukis
22 40 Constantinos Venetsanopoulos
The snf-image Ganeti OS Definition is released under [[Licence|GNU Generic Public Lisence version 2]].
23 35 Nikos Skalkotos
24 35 Nikos Skalkotos
h2. Ganeti OS Interface
25 35 Nikos Skalkotos
26 7 Vangelis Koukis
snf-image requires ganeti-os-interface v20 to operate.
27 7 Vangelis Koukis
It introduces the following OS parameters:
28 7 Vangelis Koukis
29 69 Nikos Skalkotos
 * @img_id@ (_required if @config_url@ is missing_): the URI used to identify the image (see [[Image IDs and Storage Back Ends|here]])
30 63 Constantinos Venetsanopoulos
 * @img_format@ (_required if @config_url@ is missing_): the image format type (see [[Image Format|here]])
31 63 Constantinos Venetsanopoulos
 * @img_passwd@ (_required if @config_url@ is missing_): the passwd to be injected inside the image
32 8 Vangelis Koukis
 * @img_properties@ (_optional_): additional image properties used to customize the image ([[Image_Format#Image-Properties|details]])
33 1 Constantinos Venetsanopoulos
 * @img_personality@ (_optional_): files to be injected into the image filesystem ([[Personality OS Parameter|details]]).
34 62 Constantinos Venetsanopoulos
 * @config_url@ (_optional_): the url to download configuration data from
35 1 Constantinos Venetsanopoulos
36 1 Constantinos Venetsanopoulos
h2. Architecture
37 8 Vangelis Koukis
38 99 Nikos Skalkotos
snf-image is split in two components: The main program running on the Ganeti host, with full root privilege (@snf-image@ previously @snf-image-host@), and a part running inside an unprivileged, helper VM (@snf-image-helper@).
39 8 Vangelis Koukis
40 95 Nikos Skalkotos
h3. snf-image
41 8 Vangelis Koukis
42 8 Vangelis Koukis
This part implements the Ganeti OS interface. It extracts the Image onto the Ganeti-provided block device, using streaming block I/O (@dd@ with @oflag=direct@), then passes control to @snf-image-helper@ running inside a helper VM. The helper VM is created using KVM, runs as an unprivileged user, @nobody@ by default.
43 8 Vangelis Koukis
44 102 Nikos Skalkotos
There is no restriction on the distribution running inside the helper VM, as long as it executes the @snf-image-helper@ component automatically upon bootup. The @snf-image-update-helper@ script is provided with @snf-image@ to automate the creation of a helper VM image based on Debian Stable, using @multistrap@.
45 8 Vangelis Koukis
46 8 Vangelis Koukis
The @snf-image-helper@ component is spawned inside a specific hardware environment:
47 8 Vangelis Koukis
48 8 Vangelis Koukis
* The VM features a virtual floppy, containing an @ext2@ filesystem with all parameters needed for image customization.
49 8 Vangelis Koukis
* The hard disk of the VM being deployed is accessible as the first @virtio@ hard disk.
50 40 Constantinos Venetsanopoulos
* All kernel/console output is redirected to the first virtual serial console, and eventually finds its way into the OS definition log files that Ganeti maintains.
51 8 Vangelis Koukis
* The helper VM is expected to output "SUCCESS" to its second serial port if image customization was successful inside the VM.
52 8 Vangelis Koukis
  In any other case, execution of the helper VM or @snf-image-helper@ has failed.
53 8 Vangelis Koukis
* The helper VM is expected to shutdown automatically once it is done. Its execution is time-limited; if it has not terminated after a number of seconds, configurable via @/etc/default/snf-image@, it is sent a @SIGTERM@ and/or a @SIGKILL@.
54 8 Vangelis Koukis
55 8 Vangelis Koukis
KVM is currently a dependency for @snf-image@, meaning it is needed to spawn the helper VM. There is no restriction on the hypervisor used for the actual Ganeti instances. This is not a strict requirement; KVM could be replaced by @qemu@, doing full CPU virtualization without any kernel support for spawning the helper VM.
56 8 Vangelis Koukis
57 8 Vangelis Koukis
h3. snf-image-helper
58 8 Vangelis Koukis
59 102 Nikos Skalkotos
This part runs inside the helper VM and undertakes customization of the VM being deployed using a number of hooks, or _[[Configuration Tasks|tasks]]_. The tasks run in an environment, specified by rules found in a virtual floppy, placed there by @snf-image@. @snf-image-helper@ uses @runparts@ to run tasks found under @/usr/lib/snf-image-helper/tasks@ by default.
60 3 Constantinos Venetsanopoulos
61 14 Vangelis Koukis
The architecture is presented below:
62 11 Constantinos Venetsanopoulos
63 12 Constantinos Venetsanopoulos
h3. Graphical Representation
64 11 Constantinos Venetsanopoulos
65 11 Constantinos Venetsanopoulos
!snf-image_arch.png!
66 11 Constantinos Venetsanopoulos
67 47 Nikos Skalkotos
h3. Image Names
68 47 Nikos Skalkotos
69 47 Nikos Skalkotos
snf-image assumes all image files have the following format:
70 47 Nikos Skalkotos
<pre><image_id>.<image_format></pre>
71 47 Nikos Skalkotos
72 47 Nikos Skalkotos
In order to create an instance with ganeti using an image with filename @debian_base-6.0-5-x86_64.extdump@, you need to specify the following OS parameters:
73 47 Nikos Skalkotos
<pre>img_format=extdump
74 47 Nikos Skalkotos
img_id=debian_base-6.0-5-x86_64</pre>
75 47 Nikos Skalkotos
76 1 Constantinos Venetsanopoulos
h2. Download
77 1 Constantinos Venetsanopoulos
78 79 Nikos Skalkotos
h3. Source
79 1 Constantinos Venetsanopoulos
80 113 Nikos Skalkotos
The latest source packages for snf-image (v0.9.1) can be found here:
81 113 Nikos Skalkotos
* "@snf-image_0.9.1.orig.tar.gz@":https://code.grnet.gr/attachments/download/2609/snf-image_0.9.1.orig.tar.gz
82 113 Nikos Skalkotos
* "@snf-image-helper_0.9.1.orig.tar.gz@":https://code.grnet.gr/attachments/download/2606/snf-image-helper_0.9.1.orig.tar.gz
83 79 Nikos Skalkotos
84 79 Nikos Skalkotos
h3. Packages (for debian squeeze)
85 79 Nikos Skalkotos
86 113 Nikos Skalkotos
The latest debian package for snf-image (v0.9.1) can be found here:
87 113 Nikos Skalkotos
* "@snf-image@":https://code.grnet.gr/attachments/download/2612/snf-image_0.9.1-1_all.deb
88 10 Vangelis Koukis
89 1 Constantinos Venetsanopoulos
Previous versions and source tarballs can be found here:
90 10 Vangelis Koukis
https://code.grnet.gr/projects/snf-image/files
91 1 Constantinos Venetsanopoulos
92 73 Nikos Skalkotos
You can also download the package from our @apt@ repo:
93 64 Constantinos Venetsanopoulos
94 83 Nikos Skalkotos
http://apt2.dev.grnet.gr
95 64 Constantinos Venetsanopoulos
96 80 Nikos Skalkotos
h3. Fixed snf-image-helper image
97 80 Nikos Skalkotos
98 102 Nikos Skalkotos
If you installed snf-image using the provided package, you can create the helper VM by launching `snf-image-update-helper' command. People who installed the program from source may use the fixed helper VM image provided below:
99 80 Nikos Skalkotos
100 113 Nikos Skalkotos
* "@snf_image_helper_image-0.9.1-1.tar.xz@":https://pithos.okeanos.grnet.gr/public/euharYruWl7s0C3KfdC2I4
101 80 Nikos Skalkotos
102 41 Constantinos Venetsanopoulos
h3. Sample Images
103 41 Constantinos Venetsanopoulos
104 41 Constantinos Venetsanopoulos
While developing snf-image, we created and tested a number of images. The following images are basic installations of some popular Linux distributions, that have been tested with snf-image and provided here for testing purposes:
105 41 Constantinos Venetsanopoulos
106 122 Nikos Skalkotos
* Debian Squeeze Base System ["diskdump":https://pithos.okeanos.grnet.gr/public/RDISy7sNVIJ9KIm4JkmbX4] ["md5sum":https://pithos.okeanos.grnet.gr/public/Gal05XiqgxZKXJcLhkvUy5] ["metadata":https://pithos.okeanos.grnet.gr/public/alJm9TMz5dBwzOfrgyIZY2]
107 123 Nikos Skalkotos
* Debian Wheezy Base System ["diskdump":https://pithos.okeanos.grnet.gr/public/q9Gxb9htMMfAVI3i4lN2V1] ["md5sum":https://pithos.okeanos.grnet.gr/public/OZofrTvsyKVw0ca1uVcqa6] ["metadata":https://pithos.okeanos.grnet.gr/public/5gMeRCqtMxl5hpGvfqE4E7]
108 123 Nikos Skalkotos
* Debian Desktop ["diskdump":https://pithos.okeanos.grnet.gr/public/HI69qEBrLuCTKuNzF3ENi3] ["md5sum":https://pithos.okeanos.grnet.gr/public/I10MuN7BD9SqLGhaPVwT07] ["metadata":https://pithos.okeanos.grnet.gr/public/2cKXF26sIkWzBf4g12ds2]
109 124 Nikos Skalkotos
* CentOS 6.0 ["diskdump":https://pithos.okeanos.grnet.gr/public/dSGXshb1uLx7vPkkHmyFp2] ["md5sum":https://pithos.okeanos.grnet.gr/public/5xsChidebUFSnPWB6N9rF2] ["metadata":https://pithos.okeanos.grnet.gr/public/C1kmDkcxoorkkTORlLf3n3]
110 124 Nikos Skalkotos
* Fedora Desktop 18 ["diskdump":https://pithos.okeanos.grnet.gr/public/R3R6LGH2A2AgAaK5yFqug2] ["md5sum":https://pithos.okeanos.grnet.gr/public/wY6TfJzyUiTBLXyI2mnAU5] ["metadata":https://pithos.okeanos.grnet.gr/public/bsHhw6PpR4IfGZESNGcU06]
111 126 Nikos Skalkotos
* Ubuntu Desktop LTS 12.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/PDfxRVFDQBb3fYnzi5B3h2] ["md5sum":https://pithos.okeanos.grnet.gr/public/qY3FWSlrLBnywolEXSLpw4] ["metadata":https://pithos.okeanos.grnet.gr/public/RJ3V0uF5b6KW8IBq88ygs5]
112 126 Nikos Skalkotos
* Kubuntu LTS 12.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/bLC84nLirVxBSDSqZFvs94] ["md5sum":https://pithos.okeanos.grnet.gr/public/ePLgfbNqfh4qo5lVC300E6] ["metadata":https://pithos.okeanos.grnet.gr/public/43c88oKyBUjzRpG9X0OFN5]
113 127 Nikos Skalkotos
* Ubuntu Desktop 13.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/n1XJ61tOX0VFvng4syVPt4] ["md5sum":https://pithos.okeanos.grnet.gr/public/hWlABUDQIXDBFaqHu3ett2] ["metadata":https://pithos.okeanos.grnet.gr/public/4BdfbnjeuGKDHGnyjuN9v2]
114 128 Nikos Skalkotos
* Kubuntu 13.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/GjsC1FHYX4qaBdkS0aiNm4] ["md5sum":https://pithos.okeanos.grnet.gr/public/JT0kdUGb3RS84FX3tzBfm4] ["metadata":https://pithos.okeanos.grnet.gr/public/wXE4aI9tfFPMq2FZ64pe3]
115 129 Nikos Skalkotos
* Ubuntu Server 12.04 ["diskdump":https://pithos.okeanos.grnet.gr/public/pvyxBcYA6MwSRYtMKGDZa5] ["md5sum":https://pithos.okeanos.grnet.gr/public/JLg9Zagnas6plmjlacZA54] ["metadata":https://pithos.okeanos.grnet.gr/public/HUoa4AJyQaR45ObAz67Uu1]
116 132 Nikos Skalkotos
* OpenSUSE Desktop 12.3 ["diskdump":https://pithos.okeanos.grnet.gr/public/Xvc55lG8wbSPXOkQOKVvW1] ["md5sum":https://pithos.okeanos.grnet.gr/public/PNc2itzHxpW4cicveGEiL4] ["metadata":https://pithos.okeanos.grnet.gr/public/VFRwOkuloIZQgshgAQwkk3]
117 119 Nikos Skalkotos
* FreeBSD 9.1 ["diskdump":https://pithos.okeanos.grnet.gr/public/Ol6XlgO9BoJmByO7EeJl33] ["md5sum":https://pithos.okeanos.grnet.gr/public/pFxjb9WrfOLOELvWaR0bx] ["metadata":https://pithos.okeanos.grnet.gr/public/9bHZmQcAnND9gY9vW4x1z6]
118 41 Constantinos Venetsanopoulos
119 41 Constantinos Venetsanopoulos
After getting familiar with snf-image you can build your own Images ([[Image Format|more...]]), or build upon the ones listed above.
120 33 Vangelis Koukis
121 1 Constantinos Venetsanopoulos
h2. Installation
122 1 Constantinos Venetsanopoulos
123 5 Constantinos Venetsanopoulos
Before installing snf-image be sure to have a working Ganeti installation in your cluster. The installation process should take place in *all* ganeti nodes. Here we will describe the installation in a single node. The process is identical for all nodes and should be repeated manually or automatically, e.g., with puppet.
124 5 Constantinos Venetsanopoulos
125 130 Constantinos Venetsanopoulos
h3. Installing snf-image using @apt@ (Debian-based distributions)
126 103 Constantinos Venetsanopoulos
127 104 Constantinos Venetsanopoulos
Add the following lines in your @sources.list@ file:
128 103 Constantinos Venetsanopoulos
129 104 Constantinos Venetsanopoulos
<pre>
130 1 Constantinos Venetsanopoulos
deb http://apt2.dev.grnet.gr stable/
131 103 Constantinos Venetsanopoulos
deb-src http://apt2.dev.grnet.gr stable/
132 104 Constantinos Venetsanopoulos
</pre>
133 103 Constantinos Venetsanopoulos
134 103 Constantinos Venetsanopoulos
and then run:
135 103 Constantinos Venetsanopoulos
136 105 Constantinos Venetsanopoulos
<pre>
137 103 Constantinos Venetsanopoulos
# apt-get update
138 103 Constantinos Venetsanopoulos
# apt-get install snf-image
139 103 Constantinos Venetsanopoulos
# snf-image-update-helper
140 105 Constantinos Venetsanopoulos
</pre>
141 103 Constantinos Venetsanopoulos
142 103 Constantinos Venetsanopoulos
The last command may take a while. If it exits without errors, you have installed snf-image successfully and you can move on to the configuration section.
143 81 Nikos Skalkotos
144 81 Nikos Skalkotos
h3. Installing snf-image from source
145 81 Nikos Skalkotos
146 113 Nikos Skalkotos
# Download the provided source package:<pre>$ wget https://code.grnet.gr/attachments/download/2609/snf-image_0.9.1.orig.tar.gz</pre>
147 111 Nikos Skalkotos
# Untar, configure and compile the source:<pre>$ tar -xvf snf-image_0.9.orig.tar.gz
148 113 Nikos Skalkotos
$ cd snf-image-0.9.1
149 81 Nikos Skalkotos
$ ./autoget.sh
150 1 Constantinos Venetsanopoulos
$ ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
151 1 Constantinos Venetsanopoulos
$ make</pre>
152 81 Nikos Skalkotos
# Install snf-image:<pre>$ make install
153 81 Nikos Skalkotos
$ install -Dm600 defaults /etc/default/snf-image</pre>
154 102 Nikos Skalkotos
# Download the fixed [[wiki#Fixed-snf-image-helper-image|snf-image-helper appliance]]
155 85 Nikos Skalkotos
# Extract it's content under @/var/lib/snf-image/helper/@:<pre>$ mkdir -p /var/lib/snf-image/helper
156 110 Constantinos Venetsanopoulos
$ cd /var/lib/snf-image/helper
157 81 Nikos Skalkotos
$ tar -xvf /place/where/you/stored/snf_image_helper_image-0.9.1-1.tar.xz</pre>
158 130 Constantinos Venetsanopoulos
159 130 Constantinos Venetsanopoulos
You have now installed successfully snf-image and the helper VM. You should now configure snf-image before it is ready for use.
160 45 Nikos Skalkotos
161 45 Nikos Skalkotos
h2. Configuration
162 1 Constantinos Venetsanopoulos
163 107 Constantinos Venetsanopoulos
In this section, we assume that the image file you want to deploy resides under a local directory (e.g.: @/srv/myimages/@)
164 107 Constantinos Venetsanopoulos
Once you have installed @snf-image@ in the Ganeti node, you need to overwrite the value of the corresponding option in @/etc/default/snf-image@:
165 107 Constantinos Venetsanopoulos
166 108 Constantinos Venetsanopoulos
<pre>
167 108 Constantinos Venetsanopoulos
IMAGE_DIR = /srv/myimages/
168 108 Constantinos Venetsanopoulos
</pre>
169 108 Constantinos Venetsanopoulos
170 108 Constantinos Venetsanopoulos
@IMAGE_DIR@ should point to the location where the image files are stored [default: @/var/lib/snf-image@].
171 31 Vangelis Koukis
172 47 Nikos Skalkotos
h2. Sample Usage
173 1 Constantinos Venetsanopoulos
174 109 Constantinos Venetsanopoulos
h3. Download Image
175 65 Constantinos Venetsanopoulos
176 109 Constantinos Venetsanopoulos
Download a [[Wiki#Sample-Images|Sample Image]] and store it under @IMAGE_DIR@. Make sure you also have its corresponding metadata file.
177 1 Constantinos Venetsanopoulos
178 109 Constantinos Venetsanopoulos
h3. Spawn a @diskdump@ image
179 109 Constantinos Venetsanopoulos
180 109 Constantinos Venetsanopoulos
If you want to deploy an image of type @diskdump@, you need to provide the corresponding @img_properties@ as described in the [[Image_Format#OS-parameter-img_properties|Image Format]] section. If using a diskdump found in the [[Wiki#Sample-Images|Sample Images]] list, use the @img_properties@ described in the image's metadata file. For example:
181 109 Constantinos Venetsanopoulos
182 65 Constantinos Venetsanopoulos
<pre>gnt-instance add -o snf-image+default --os-parameters img_passwd=SamplePassw0rd,img_format=diskdump,img_id=debian_base-6.0-7-x86_64,img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"}' -t plain --disk=0:size=10G --no-name-check --no-ip-check --no-nics test1</pre>
183 65 Constantinos Venetsanopoulos
184 109 Constantinos Venetsanopoulos
h3. Spawn an @extdump@/@ntfsdump@ image
185 1 Constantinos Venetsanopoulos
186 56 Constantinos Venetsanopoulos
To create a ganeti instance with snf-image using @debian_base-6.0-5-x86_64.extdump@ hosted under @IMAGE_DIR@, use a command like the one below:
187 56 Constantinos Venetsanopoulos
<pre>gnt-instance add -o snf-image+default --os-parameters img_passwd=SamplePassw0rd,img_format=extdump,img_id=debian_base-6.0-5-x86_64 -t plain --disk=0:size=10G --no-name-check --no-ip-check --no-nics test1</pre>
188 1 Constantinos Venetsanopoulos
189 65 Constantinos Venetsanopoulos
h2. Advanced Usage
190 1 Constantinos Venetsanopoulos
191 65 Constantinos Venetsanopoulos
If you want to learn more and use the advanced features of snf-image please refer to following sections:
192 1 Constantinos Venetsanopoulos
193 76 Nikos Skalkotos
# [[Configuration Tasks|Image Configuration Tasks]]
194 76 Nikos Skalkotos
# [[Image Format|Supported Image formats]]
195 76 Nikos Skalkotos
# [[Image Store|Supported Image store locations]]
196 76 Nikos Skalkotos
# [[PMI|Progress Monitor Interface]]
197 22 Constantinos Venetsanopoulos
198 13 Constantinos Venetsanopoulos
h2. Developers
199 46 Nikos Skalkotos
200 45 Nikos Skalkotos
If you are a developer or want to use the latest source code you can download the latest development version by cloning the snf-image git repository:
201 18 Vangelis Koukis
<pre>$ git clone https://code.grnet.gr/git/snf-image</pre>
202 18 Vangelis Koukis
The _master_ branch contains the latest development version
203 18 Vangelis Koukis
The _debian_ branch contains the _master_ plus the debian packaging directories.
204 24 Constantinos Venetsanopoulos
205 27 Nikos Skalkotos
For more information on how to configure, install or package from source, see the [[Developer|Developer's page]].
206 19 Vangelis Koukis
207 1 Constantinos Venetsanopoulos
h2. Community & Support
208 13 Constantinos Venetsanopoulos
209 13 Constantinos Venetsanopoulos
Your help is very important. Any contributions and bug reports will be highly appreciated. You can contact the team at:
210 1 Constantinos Venetsanopoulos
211 75 Nikos Skalkotos
* Bug reports - feedback - support: synnefo@googlegroups.com