+2013-10-18, v0.11
+ * Update the documentation
+ * Change local-backend identifier from file:// to local//
+ * Make sure the user does not specify local images outside IMAGE_DIR by
+ using .. in the IMG_ID parameter
+ * HELPER: Fix a bug triggered when the image is corructed
+
2013-09-23, v0.10.1
* Merge the ChangeLog files of snf-image and snf-image-helper
* Add version check in snf-image-update-helper
* Fix a bug in the network back-end
2013-09-02, v0.9.3
- * Add code to download and install a fixed snf-image-helper image. This is the default
- behaviour for snf-image-update-helper.
+ * Add code to download and install a fixed snf-image-helper image. This is
+ the default behaviour for snf-image-update-helper.
2013-08-27, v0.9.2
* Use devflow building system
Advanced Topics
===============
+.. _image-format-advanced:
+
+Image Format
+^^^^^^^^^^^^
+
+snf-image supports 3 types of image formats:
+
+ * **extdump**: a raw dump of an ext{2,3,4} file system
+ * **ntfsdump**: a raw dump of an NTFS file system
+ * **diskdump** (recommended): a raw dump of a disk
+
+extdump and ntfsdump image formats
+++++++++++++++++++++++++++++++++++
+
+Those two formats are dumps (raw copies using dd) of partitions hosting Linux
+systems on ext{2,3,4} and Windows systems on ntfs file systems respectively.
+Partitions hosting a Windows or Linux system that are suitable for dumping
+should have the following properties:
+
+ * Be the first partition in the file system
+ * The OS they host should not depend on any other partitions
+ * Start at sector 2048
+ * Have a boot loader installed in the boot sector of the partition (not MBR)
+ * Have the root device in */etc/fstab* specified in a persistent way, using
+ UUID or LABEL (for extdump only)
+
+Known Issues
+------------
+
+ * For Linux systems, having grub installed in the partition is fragile and
+ things can go wrong when resizing the partitions, especially when shrinking.
+ * More complicated partition schemes are not supported.
+
+diskdump image format (recommended)
++++++++++++++++++++++++++++++++++++
+
+Diskdump is a newer format that overcomes most of the aforementioned issues.
+This format is a dump (raw copy using dd) of a whole disk.
+
+This design decision has the following benefits:
+
+ * Swap partitions are supported
+ * The system may use multiple partitions:
+ * dedicated partitions for /boot, /home etc in Linux
+ * system and boot partition in Windows
+ * There are no restrictions on starting sectors of partitions
+
+Although diskdump is a lot more flexible than the older formats, there are
+still some rules to follow:
+
+ * All devices in fstab should be specified by persistent names (UUID or LABEL)
+ * LVMs are not supported
+ * For Linux disks only ext{2,3,4} file systems are supported
+ * For FreeBSD disks only UFS file systems are supported
+ * For FreeBSD only GUID Partition Tables (GPT) are supported
+
Progress Monitoring Interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
image-info
++++++++++
-This message type is used to display random progress information. It has an
+This message type is used to display arbitrary progress information. It has an
extra *messages* field whose value is a list of strings. A valid ``image-info``
message looks like this:
Architecture
============
+Overview
+^^^^^^^^
+
+snf-image is a Ganeti OS definition. This means that Ganeti provisions a new
+disk (block device) and passes it to snf-image. Then, snf-image is responsible
+to deploy an Image on that disk. If snf-image returns successfully, Ganeti will
+then spawn a VM with that disk as its primary disk.
+
+Thus, snf-image is responsible for two (2) things, which are executed in two
+separate steps:
+
+| 1. Fill the newly provisioned disk with Image data
+| 2. Customize the Image accordingly
+
+For (1), snf-image can fetch the Image from a number of back-ends, as we
+describe later. For (2) snf-image spawns a helper VM and runs a number of
+configuration tasks inside the isolated environment. Once the last task returns
+successfully, the helper VM ceases and snf-image returns the newly configured
+disk to Ganeti.
+
+The whole procedure is configurable via OS interface parameters, that can be
+passed to snf-image from the Ganeti command line or RAPI.
+
snf-image is split in two components: The main program running on the Ganeti
-host with full root privilege (*snf-image* previously *snf-image-host*) and a
+host with full root privilege (*snf-image*, previously *snf-image-host*) and a
part running inside an unprivileged helper VM (*snf-image-helper*).
+We describe each part in the following sections:
+
snf-image
^^^^^^^^^
This part implements the Ganeti OS interface. It extracts the Image onto the
Ganeti-provided block device, using streaming block I/O (dd with oflag=direct),
-then passes control to snf-image-helper running inside a helper VM. The helper
-VM is created using KVM, runs as an unprivileged user, nobody by default.
+then spawns a helper VM, and passes control to snf-image-helper running inside
+that helper VM. The helper VM is created using either KVM or XEN depending on
+the supported hypervisor as dictated by Ganeti. It runs as an unprivileged
+user.
There is no restriction on the distribution running inside the helper VM, as
long as it executes the snf-image-helper component automatically upon bootup.
The snf-image-update-helper script is provided with snf-image to automate the
creation of a helper VM image based on Debian Stable, using multistrap.
-The snf-image-helper component is spawned inside a specific hardware
-environment:
+The snf-image-helper component runs inside a specific environment, which is
+created and ensured by snf-image:
- * The VM features a virtual floppy, containing an ext2 filesystem with all
+ * The VM features a virtual floppy, containing an ext2 file system with all
parameters needed for image customization.
- * The hard disk of the VM being deployed is accessible as the first virtio
- hard disk.
+ * The hard disk provided by Ganeti that we want to deploy and customize is
+ accessible as the first VirtIO hard disk.
* All kernel/console output is redirected to the first virtual serial console,
and eventually finds its way into the OS definition log files that Ganeti
maintains.
* The helper VM is expected to output "SUCCESS" to its second serial port if
image customization was successful inside the VM.
- * In any other case, execution of the helper VM or snf-image-helper has
- failed.
+ * If "SUCCESS" is not returned, snf-image assumes that, execution of the helper
+ VM or snf-image-helper has failed.
* The helper VM is expected to shutdown automatically once it is done. Its
execution is time-limited; if it has not terminated after a number of
- seconds, configurable via /etc/default/snf-image, it is sent a SIGTERM
- and/or a SIGKILL.
-
-KVM is currently a dependency for snf-image, meaning it is needed to spawn the
-helper VM. There is no restriction on the hypervisor used for the actual Ganeti
-instances. This is not a strict requirement; KVM could be replaced by qemu,
-doing full CPU virtualization without any kernel support for spawning the
-helper VM.
+ seconds, configurable via ``/etc/default/snf-image``, snf-image sends a
+ SIGTERM and/or a SIGKILL to it.
snf-image-helper
^^^^^^^^^^^^^^^^
-This part runs inside the helper VM and undertakes customization of the VM
-being deployed using a number of hooks, or tasks. The tasks run in an
-environment, specified by rules found in a virtual floppy, placed there by
-*snf-image*. *snf-image-helper* uses runparts to run tasks found under
-*/usr/lib/snf-image-helper/tasks* by default
+This part runs inside the helper VM during boot-up and undertakes customization
+of the target disk. It does so, by running a number of :ref:`configuration
+tasks <image-configuration-tasks>`. The exact tasks that should run, are
+specified by rules found in the virtual floppy, placed there by *snf-image*,
+before spawning the helper VM. *snf-image-helper* uses *runparts* to run the
+tasks which are found under ``/usr/lib/snf-image-helper/tasks``.
Graphical Representation
^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: /images/arch.png
+
+.. _storage-backends:
+
+Storage back-ends
+^^^^^^^^^^^^^^^^^
+
+As stated above, for step (1), *snf-image* is capable of fetching images that
+are stored in a variety of different back-ends and then extracting them onto
+the newly created block device. The following back-ends are supported:
+
+ * **Local back-end**:
+ The local back-end is used to retrieve images that are stored on the Ganeti
+ node that the image deployment takes place. All local images are expected to
+ be found under a predefined image directory. By default */var/lib/snf-image*
+ is used, but the user may change this by overwriting the value of the
+ *IMAGE_DIR* variable under ``/etc/default/snf-image``.
+
+ * **Network back-end**:
+ The network back-end is used to retrieve images that are accessible from the
+ network. snf-image can fetch images via *http:*, *https:*, *ftp:* or
+ *ftps:*, using `cURL <http://curl.haxx.se/>`_.
+
+ * **Pithos back-end**:
+ *snf-image* contains a special command-line tool (*pithcat*) for retrieving
+ images residing on a Pithos installation. To set up snf-image's Pithos
+ back-end the user needs to setup the ``PITHOS_DATA`` and ``PITHOS_DB``
+ variables inside ``/etc/default/snf-image`` accordingly.
+
+ * **Null back-end**:
+ If the null back-end is selected, no image copying is performed. This
+ actually is meant for bypassing step (1) altogether. This is useful, if the
+ disk provisioned by Ganeti already contains an OS installation before
+ *snf-image* is executed (for example if the disk was created as a clone of
+ an existing VM's hard disk).
+
.. _image-configuration-tasks:
Image Configuration Tasks
^^^^^^^^^^^^^^^^^^^^^^^^^
-Configuration tasks are scripts called by snf-image-helper to accomplish
-various configuration steps on the newly created instance. See below for a
-description of each one of them:
+Configuration tasks are scripts called by snf-image-helper inside the helper VM
+to accomplish various configuration steps on the newly created instance. See
+below for a description of each one of them:
**FixPartitionTable**: Enlarges the last partition in the partition table of
the instance, to consume all the available space and optionally adds a swap
by *SNF_IMAGE_TARGET*. The script will fail if any of those 3 variables has a
non-sane value.
-**AddSwap**: Formats the swap partion added by *FixPartitionTable* task and
+**AddSwap**: Formats the swap partition added by *FixPartitionTable* task and
adds an appropriate swap entry in the system's ``/etc/fstab``. The script will
only run if *SNF_IMAGE_PROPERTY_SWAP* is present and will fail if
*SNF_IMAGE_TARGET* in not defined.
-**DeleteSSHKeys**: For linux images, this script will clear out any ssh keys
-found in the image and for debian, it will recreate them too. In order to find
+**DeleteSSHKeys**: For Linux images, this script will clear out any ssh keys
+found in the image and for Debian, it will recreate them too. In order to find
the ssh keys, the script looks in default locations (/etc/ssh/ssh_*_key) and
also parses ``/etc/ssh/sshd_config`` file if present. The script will fail if
*SNF_IMAGE_TARGET* is not set.
needed by windows in order to perform an unattended setup. The
*SNF_IMAGE_TARGET* variables needs to be present for this task to run.
-**SELinuxAutorelabel**: Creates *.autorelabel* file in RedHat images. This is
+**SELinuxAutorelabel**: Creates *.autorelabel* file in Red Hat images. This is
needed if SELinux is enabled to enforce an automatic file system relabeling at
-the next boot. The only enviromental variable required by this task is
+the next boot. The only environmental variable required by this task is
*SNF_IMAGE_TARGET*.
**AssignHostname**: Assigns or changes the hostname in a Linux or Windows
image. The task will fail if the Linux distribution is not supported. For now,
-we support Debian, Redhat, Slackware, SUSE and Gentoo derived distros. The
-hostname is read from *SNF_IMAGE_HOSTNAME* variable. In addition to the latter,
-*SNF_IMAGE_TARGET* is also required.
+we support Debian, Red Hat, Slackware, SUSE and Gentoo derived distributions.
+The hostname is read from *SNF_IMAGE_HOSTNAME* variable. In addition to the
+latter, *SNF_IMAGE_TARGET* is also required.
**ChangePassword**: Changes the password for a list of users. For Linux systems
this is accomplished by directly altering the image's ``/etc/shadow`` file. For
Windows systems a script is injected into the VM's hard disk. This script will
-be executed during the specialize pass of the Windows setup. The list of users
-whose passwords will changed is determined by the *SNF_IMAGE_PROPERTY_USERS*
-variable (see :ref:`image-properties`). For this task to run *SNF_IMAGE_TARGET*
-and *SNF_IMAGE_PASSWORD* variables need to be present.
+be executed during the specialize pass of the Windows setup. For FreeBSD
+``/etc/master.passwd`` is altered, ``/etc/spwd.db`` is removed and a script is
+injected into the VM's hard disk that will recreate the aforementioned file.
+The list of users whose passwords will changed is determined by the
+*SNF_IMAGE_PROPERTY_USERS* variable (see :ref:`image-properties`). For this
+task to run *SNF_IMAGE_TARGET* and *SNF_IMAGE_PASSWORD* variables need to be
+present.
**FilesystemResizeMounted**: Injects a script into a Windows image file system
that will enlarge the last file system to cover up the whole partition. The
# built documents.
#
# The short X.Y version.
-version = '0.10.1'
+version = '0.11'
# The full version, including alpha/beta/rc tags.
-release = '0.10.1'
+release = '0.11'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
.. image:: /images/logo.png
-snf-image is a `Ganeti <http://code.google.com/p/ganeti/>`_ OS definition,
-primary used by `Synnefo <http://www.synnefo.org/>`_. It is rewritten from
-scratch and allows Ganeti to launch instances from predefined or untrusted
-custom Images. The whole process of deploying an Image onto the block device,
-as provided by Ganeti, is done in complete isolation from the physical host,
-enhancing robustness and security.
+snf-image is a `Ganeti <http://code.google.com/p/ganeti/>`_ OS definition. It
+allows Ganeti to launch instances from predefined or untrusted custom Images.
+The whole process of deploying an Image onto the block device, as provided by
+Ganeti, is done in complete isolation from the physical host, enhancing
+robustness and security.
snf-image supports `KVM <http://www.linux-kvm.org/page/Main_Page>`_ and
-`Xen <http://www.xenproject.org/>`_ based ganeti clusters.
+`Xen <http://www.xenproject.org/>`_ based Ganeti clusters.
-There are also additional hooks that can be enabled at image deployment. They
-allow for:
+snf-image also supports Image customization via hooks. Hooks allow for:
* changing the password of root or arbitrary users
* injecting files at arbitrary locations inside the filesystem, e.g., SSH keys
* setting a custom hostname
* re-creating SSH host keys to ensure the image uses unique keys
-snf-image has been used successfully to deploy many major Linux distributions
-(Debian, Ubuntu/Kubuntu, CentOS, Fedora, OpenSUSE), Windows 2008 R2 & Windows
-Server 2012, as well as FreeBSD 9.1
+snf-image is being used in large scale production environments with Ganeti to
+successfully deploy many major Linux distributions (Debian, Ubuntu/Kubuntu,
+CentOS, Fedora, OpenSUSE), Windows 2008 R2 & Windows Server 2012, as well as
+FreeBSD 9.1.
The snf-image Ganeti OS Definition is released under
`GPLv2 <http://www.gnu.org/licenses/gpl-2.0.html>`_.
-
Contents:
^^^^^^^^^
:maxdepth: 2
architecture
+ interface
installation
usage
advanced
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
-
--- /dev/null
+Interface
+=========
+
+Ganeti OS Interface
+^^^^^^^^^^^^^^^^^^^
+
+*snf-image* requires ganeti-os-interface v20 to operate and it introduces the
+following OS Parameters:
+
+ * **img_format** (required if *config_url* is missing): the image format type
+ (:ref:`details <image-format>`)
+ * **img_id** (required if *config_url* is missing): the URI used to identify
+ the image (:ref:`details <image-id>`)
+ * **img_passwd** (required if *config_url* is missing): the password to be
+ injected to the image
+ * **img_properties** (optional): additional image properties used to customize
+ the image (:ref:`details <image-properties>`)
+ * **img_personality** (optional): files to be injected into the image
+ filesystem (:ref:`details <image-personality>`)
+ * **config_url** (optional): the url to download configuration data from
+
+.. _image-format:
+
+Image Format (img_format)
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+snf-image supports 3 different types of image formats:
+
+ * **diskdump** (recommended): a raw dump of a disk
+ * **extdump**: a raw dump of an ext{2,3,4} file system
+ * **ntfsdump**: a raw dump of an NTFS file system
+
+These are also the only valid values for the **img_format** OS parameter.
+The **diskdump** type is the newest and recommended type. Thus, all sample
+images we provide are of this type. For more details about the internals of
+image formats please see the :ref:`corresponding advanced section
+<image-format-advanced>`.
+
+.. _image-id:
+
+Image ID (img_id)
+^^^^^^^^^^^^^^^^^
+
+The **img_id** OS parameter points to the actual Image that we want to deploy.
+It is a URI and its prefix denotes the type of :ref:`backend <storage-backends>`
+to be used. If no prefix is used, it defaults to the local backend:
+
+ * **Local backend**:
+ To select it, the prefix should be ``local://``, followed by the name of the
+ image. All local images are expected to be found under a predefined image
+ directory (``/var/lib/snf-image`` by default).
+
+ | For example, if we want to deploy the image file:
+ | ``/var/lib/snf-image/slackware.diskdump``
+ | We need to assign:
+ | ``img_format=diskdump`` and ``img_id=file://slackware.diskdump``
+
+ * **Network backend**:
+ If the **imd_id** starts with ``http:``, ``https:``, ``ftp:`` or ``ftps:``,
+ snf-image will treat the **img_id** as a remote URL and will try to fetch the
+ image using `cURL <http://curl.haxx.se/>`_.
+
+ | For example, if we want to deploy an image from an http location:
+ | ``img_id=http://www.synnefo.org/path/to/image/slackware-image``
+
+ * **Pithos backend**:
+ If the **img_id** is prefixed with ``pithos://`` or ``pithosmap://`` the
+ image is considered to reside on a Pithos deployment. For ``pithosmap://``
+ images, the user needs to have set a valid value for the
+ ``PITHOS_DATA`` variable in snf-image's configuration file
+ (``/etc/default/snf-image`` by default). For ``pithos://`` images, in
+ addition to ``PITHOS_DATA``, the user needs to have set a valid value for the
+ ``PITHOS_DB`` variable, too.
+
+ | For example, if we want to deploy using a full Pithos URI:
+ | ``img_id=pithos://<user-uuid>/<container>/<slackware-image>``
+ | or if we already know the map:
+ | ``img_id=pithosmap://<slackware-image-map-name>``
+
+ * **Null backend**:
+ To select the Null backend and skip the fetching and extraction step, we set
+ ``img_id=null``.
+
+.. _image-properties:
+
+Image Properties (img_properties)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+*snf-image* may use a number of properties to properly configure the image.
+Those image properties are passed to snf-image by Ganeti through the
+*img_poroperties* OS parameter (see Ganeti OS Interface). The name of all image
+properties is case-insensitive. For the diskdump format some properties are
+mandatory. For {ext,ntfs}dump formats all image properties are optional.
+
+We can group image properties in two categories:
+
+1. Generic properties (*OSFAMILY*, *ROOT_PARTITION*, *USERS*)
+2. Configuration tasks to run (*EXCLUDE_ALL_TASKS*, *EXCLUDE_TASK_<task_name>*)
+ (see here for :ref:`valid configuration tasks <image-configuration-tasks>`)
+
+A list of all properties follows:
+
+Mandatory properties (for diskdump only)
+++++++++++++++++++++++++++++++++++++++++
+
+ * **OSFAMILY={linux,windows}**
+ This specifies whether the image is a Linux or a Windows Image.
+ {ext,ntfs}dump formats are self descriptive regarding this property.
+ * **ROOT_PARTITION=n**
+ This specifies the partition number of the root partition. As mentioned
+ earlier, for now, only primary partitions are supported. This property is
+ trivial for {ext,ntfs}dump formats (they only host one partition).
+
+Optional properties
++++++++++++++++++++
+
+ * **USERS="username1 username2...."**
+ This is a space-separated list of users, whose password will be reset by
+ *snf-image*. The use of this property is optional, but highly recommended.
+ For now, if this property is missing, the users are chosen according to a
+ set of rules, but those rules may change or even be dropped in the future.
+ The rules we currently use are listed below:
+
+ * For Windows images, the *Administrator*'s password is reset.
+ * For Linux and FreeBSD images, the *root* password is reset.
+
+ * **EXCLUDE_ALL_TASKS=yes**
+ If this property is defined with a value other than null, then during the
+ deployment, the image will not be configured at all. This is really handy
+ because it gives the ability to deploy images hosting operating systems
+ whose configuration is not supported by snf-image.
+
+ * **EXCLUDE_TASK_<task_name>=yes**
+ This family of properties gives the ability to exclude individual
+ configuration tasks from running. Hence, if the property
+ *EXCLUDE_TASK_DeleteSSHKeys* with a value other than null is passed to
+ *snf-image*, the aforementioned configuration step will not be executed, and
+ the SSH Keys found in the image will not be removed during the deployment.
+ Task exclusion provides great flexibility, but it needs to be used with
+ great care. Tasks depend on each other and although those dependencies are
+ well documented, automatic task dependency resolution isn't yet supported in
+ *snf-image*. If you exclude task A but not task B which depends on A, you
+ will probably end up with an unsuccessful deployment because B will fail and
+ exit in an abnormal way. You can read more about configuration tasks here.
+
+img_properties OS parameter
++++++++++++++++++++++++++++
+
+Image properties are passed to snf_image through the img_properties OS
+parameter as a simple json string like the one below:
+
+| {
+| "PROPERTY1": "VALUE1",
+| "PROPERTY2": "VALUE2",
+| "PROPERTY3": "VALUE3",
+| ...
+| ...
+| ...
+| "PROPERTYn": "VALUEn"
+| }
+
+
+A real life example for creating a new Ganeti instance and passing image
+properties to snf-image looks like this:
+
+.. code-block:: console
+
+ ``gnt-instance add -O img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"2"\,"USERS":"root guest"}',img_format=diskdump,img_id=...``
+
+.. _image-personality:
+
+Image Personality (img_personality)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This parameter is an extension of the Server Personality notation proposed by
+the OpenStack Compute API v1.1 and defines a list of files to be injected into
+the image file system.
+
+Format
+++++++
+
+The format of this parameter is a JSON array of objects. Each object in the
+array supports the following keys:
+
+ * **path**: The absolute path of the file (string)
+ * **contents**: The content of the file encoded as a base64 string (string)
+ * **owner**: The user ownership of the file (string)
+ * **group**: The group ownership of the file (string)
+ * **mode**: The permission mode of the file (number)
+
+The first two (path, contents) are mandatory. The others (owner, group, mode)
+are optional and their default value is root, root and 0440 respectively.
+
+Example
++++++++
+
+The JSON string below defines two files (*/tmp/test1*, */tmp/test2*) whose
+content is ``test1\n`` and ``test2\n``, they are both owned by *root:root* and
+their permissions are ``-rw-r--r--`` [#]_
+
+| [
+| {
+| "path": "/tmp/test1",
+| "contents": "dGVzdDENCg==",
+| "owner": "root",
+| "group": "root",
+| "mode": 0644
+| },
+| {
+| "path": "/tmp/test2",
+| "contents": "dGVzdDINCg==",
+| "owner": "root",
+| "group": "root",
+| "mode": 420
+| }
+| ]
+
+.. [#] The first mode is in octal representation and the second in decimal.
Usage
=====
-Ganeti OS Interface
-^^^^^^^^^^^^^^^^^^^
-
-*snf-image* requires ganeti-os-interface v20 to operate and it introduces the
-following OS Parameters:
-
- * **img_format** (required if *config_url* is missing): the image format type
- (:ref:`details <image-format>`)
- * **img_id** (required if *config_url* is missing): the URI used to identify
- the image (:ref:`details <image-id>`)
- * **img_passwd** (required if *config_url* is missing): the password to be
- injected to the image
- * **img_properties** (optional): additional image properties used to customize
- the image (:ref:`details <image-properties>`)
- * **img_personality** (optional): files to be injected into the image
- filesystem (:ref:`details <image-personality>`)
- * **config_url** (optional): the url to download configuration data from
-
-.. _image-format:
-
-Image Format
-^^^^^^^^^^^^
-
-Right now 3 different types of image formats are supported:
-
- * **extdump**: a raw dump of an ext{2,3,4} file system
- * **ntfsdump**: a raw dump of an NTFS file system
- * **diskdump**: a raw dump of a disk
-
-extdump and ntfsdump image formats
-++++++++++++++++++++++++++++++++++
-
-Those two formats are dumps (raw copies using dd) of partitions hosting Linux
-systems on ext{2,3,4} and Windows systems on ntfs file systems respectively.
-Partitions hosting a Windows or Linux system that are suitable for dumping
-should have the following properties:
-
- * Be the first partition in the file system
- * The OS they host should not depend on any other partitions
- * Start at sector 2048
- * Have a boot loader installed in the boot sector of the partition (not MBR)
- * Have the root device in */etc/fstab* specified in a persistent way, using
- UUID or LABEL (for extdump only)
-
-Known Issues
-------------
-
- * For Linux systems, having grub installed in the partition is fragile and
- things can go wrong when resizing the partitions, especially when shrinking.
- * More complicated partition schemes are not supported.
-
-diskdump image format
-+++++++++++++++++++++
-
-Diskdump is a newer format that overcomes most of the aforementioned issues.
-This format is a dump (raw copy using dd) of a whole disk.
-
-This design decision has the following benefits:
-
- * Swap partitions are supported
- * The system may use multiple partitions:
- * dedicated partitions for /boot, /home etc in Linux
- * system and boot partition in Windows
- * There are no restrictions on starting sectors of partitions
-
-Although diskdump is a lot more flexible than the older formats, there are
-still some rules to follow:
-
- * All devices in fstab should be specified by persistent names (UUID or LABEL)
- * LVMs are not supported
- * For Linux disks only ext{2,3,4} file systems are supported
- * For FreeBSD disks only UFS file systems are supported
- * For FreeBSD only GUID Partition Tables (GPT) are supported
-
-.. _image-id:
-
-Image IDs & Storage back-ends
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-*snf-image* capable of deploying images that are stored in a variety of
-different back-ends. The back-end to be used is determined by the value of the
-*img_id* OS parameter. The following back-ends are supported:
-
- * **Local back-end**:
- The local back-end is used to retrieve images that are stored in the ganeti
- node that the image deployment takes place. The local back-end is used if
- the value of the *img_id* ganeti OS parameter is either prefixed with
- *file://* or is not prefixed at all. All local images are expected to be
- found under a predifined image directory. By default */var/lib/snf-image* is
- used, but the user may change this by overwriting the value of the
- *IMAGE_DIR* variable under ``/etc/default/snf-image``. The name of the image
- file is created by adding the image type extension in the end of the
- *img_id*. For example if the *img_id* is *file://slackware* and the image
- type is *diskdump*, snf-image will expect to find an image file under the
- following path: ``/usr/lib/snf-image/slackware.diskdump``
-
- * **Network back-end**:
- The network back-end is used to retrieve images that are accessible from the
- network. If the *imd_id* starts with *http:*, *https:*, *ftp:* or *ftps:*,
- snf-image will treat the *img_id* as a remote URL and will try to fetch the
- image using `cURL <http://curl.haxx.se/>`_.
-
- * **Pithos back-end**:
- If an *img_id* is prefixed with *pithos:* or *pithosmap:* the image is
- considered to be pithos back-ended. *snf-image* contains a special
- command-line tool (*pithcat*) for retrieving this kind of images. For
- *pithosmap:* images, the user needs to have set a valid value for the
- *PITHOS_DATA* variable. For *pithos:* images, in addition to PITHOS_DATA,
- the PITHOS_DB variable needs to contain a valid value too.
- ``/etc/default/snf-image`` may be used to set both values.
-
- * **Null back-end**:
- The null back-end is used if the *img_id* value is *null*. In this case no
- image copying is performed. This is usefull if the hard disk already
- contains an OS installation before *snf-image* is executed (for example if
- the hard disk is a snapshot of an existing VM's hard disk).
-
-.. _image-properties:
-
-Image Properties
-^^^^^^^^^^^^^^^^
-
-In order for *snf-image* to be able to properly configure an image, it may make
-use of a set of image properties. Those image properties are passed to
-*snf-image* by Ganeti through the *img_poroperties* OS parameter (see Ganeti OS
-Interface). The name of all image properties is case-insensitive. For the
-diskdump format some properties are mandatory. For {ext,ntfs}dump formats all
-image properties are optional.
-
-A list of mandatory and optional properties follows:
-
-Mandatory properties (diskdump only)
-++++++++++++++++++++++++++++++++++++
-
- * **OSFAMILY={linux,windows}**
- This specifies whether the image is a Linux or a Windows Image.
- {ext,ntfs}dump formats are self descriptive regarding this property.
- * **ROOT_PARTITION=n**
- This specifies the partition number of the root partition. As mentioned
- earlier, for now, only primary partitions are supported. This property is
- trivial for {ext,ntfs}dump formats (they only host one partition).
-
-Optional properties
-+++++++++++++++++++
-
- * **USERS="username1 username2...."**
- This is a space-seperated list of users, whose password will be reset by
- *snf-image*. The use of this property is optional, but highly recommended.
- For now, if this property is missing, the users are chosen according to a
- set of rules, but those rules may change or even be dropped in the future.
- The rules we currently use are listed below:
-
- * For Windows images, the *Administrator*'s password is reset.
- * For Linux and FreeBSD images, the *root* password is reset.
-
- * **EXCLUDE_ALL_TASKS=yes**
- If this property is defined with a value other than null, then during the
- deployment, the image will not be configured at all. This is really handy
- because it gives the ability to deploy images hosting operating systems
- whose configuration is not supported by snf-image.
-
- * **EXCLUDE_TASK_<task_name>=yes**
- This family of properties gives the ability to exclude individual
- configuration tasks from running. Hence, if the property
- *EXCLUDE_TASK_DeleteSSHKeys* with a value other than null is passed to
- *snf-image*, the aforementioned configuration step will not be executed, and
- the SSH Keys found in the image will not be removed during the deployment.
- Task exclusion provides great flexibility, but it needs to be used with
- great care. Tasks depend on each other and although those dependencies are
- well documented, automatic task dependency resolution isn't yet supported in
- *snf-image*. If you exclude task A but not task B which depends on A, you
- will probably end up with an unsuccessful deployment because B will fail and
- exit in an abnormal way. You can read more about configuration tasks here.
-
-
-img_properties OS parameter
-+++++++++++++++++++++++++++++++
-
-Image properties are passed to snf_image through the img_properties OS
-parameter as a simple json string like the one below:
-
-| {
-| "PROPERTY1": "VALUE1",
-| "PROPERTY2": "VALUE2",
-| "PROPERTY3": "VALUE3",
-| ...
-| ...
-| ...
-| "PROPERTYn": "VALUEn"
-| }
-
-
-A real life example for creating a new ganeti instance and passing image
-properties to snf-image would probably look more like this:
-
-``gnt-instance add -O img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"2"\,"USERS":"root guest"}',img_format=diskdump...``
-
-.. _image-personality:
-
-Personality OS Parameter
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-This parameter is an extension of the Server Personality notation proposed by
-the OpenStack Compute API v1.1 and defines a list of files to be injected into
-the image file system.
-
-Format
-++++++
-
-The format of this parameter is a JSON array of objects. Each object in the
-array supports the following keys:
-
- * **path**: The absolute path of the file (string)
- * **contents**: The content of the file encoded as a base64 string (string)
- * **owner**: The user ownership of the file (string)
- * **group**: The group ownership of the file (string)
- * **mode**: The permission mode of the file (number)
-
-The first two (path, contents) are mandatory. The others (owner, group, mode)
-are optional and their default value is root, root and 0440 respectively.
-
-Example
-+++++++
-
-The JSON string below defines two files (*/tmp/test1*, */tmp/test2*) whose
-content is ``test1\n`` and ``test2\n``, they are both owned by *root:root* and
-their permissions are ``-rw-r--r--`` [#]_
-
-| [
-| {
-| "path": "/tmp/test1",
-| "contents": "dGVzdDENCg==",
-| "owner": "root",
-| "group": "root",
-| "mode": 0644
-| },
-| {
-| "path": "/tmp/test2",
-| "contents": "dGVzdDINCg==",
-| "owner": "root",
-| "group": "root",
-| "mode": 420
-| }
-| ]
-
-.. [#] The first mode is in octal representation and the second in decimal.
-
-
.. _sample-images:
Sample Images
[`diskdump <http://cdn.synnefo.org/opensuse_desktop-12.3-x86_64.diskdump>`_]
[`md5sum <http://cdn.synnefo.org/opensuse_desktop-12.3-x86_64.diskdump.md5sum>`_]
[`metadata <http://cdn.synnefo.org/opensuse_desktop-12.3-x86_64.diskdump.meta>`_]
- * FreeBSD 9.1
- [`diskdump <http://cdn.synnefo.org/freebsd-9.1-x86_64.diskdump>`_]
- [`md5sum <http://cdn.synnefo.org/freebsd-9.1-x86_64.diskdump.md5sum>`_]
- [`metadata <http://cdn.synnefo.org/freebsd-9.1-x86_64.diskdump.meta>`_]
+ * FreeBSD 9.2
+ [`diskdump <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump>`_]
+ [`md5sum <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump.md5sum>`_]
+ [`metadata <http://cdn.synnefo.org/freebsd-9.2-x86_64.diskdump.meta>`_]
Sample Usage
^^^^^^^^^^^^
Spawn a diskdump image
++++++++++++++++++++++
-If you want to deploy an image of type diskdump, you need to provide the
-corresponding *img_properties* as described in the
+If you want to deploy an image of type diskdump, you
+need to provide the corresponding *img_properties* as described in the
:ref:`Image Format<image-format>` section. If using a diskdump found in the
:ref:`sample-images` list, use the *img_properties* described in the image's
-metadata file. For example:
+metadata file. For example, to successfully deploy the
+*debian_base-7.0-x86_64.diskdump* image file, you need to provide the following
+image properties:
+
+| OSFAMILY=linux
+| ROOT_PARTITION=1
+| USERS=root
+
+Hence, the ganeti command for creating a VM from this image file would look
+like this:
+
+.. code-block:: console
+
+ gnt-instance add -o snf-image+default \
+ -O img_passwd=1Ki77y,img_format=diskdump,img_id=debian_base-7.0-x86_64,img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"\,"USERS":"root"}' \
+ -t plain --disk=0:size=10G --no-name-check --no-ip-check --no-nics my_debian_server1
+
+If you don't want to configure the image at all and just copy it to the ganeti
+provided disk, use the ``EXCLUDE_ALL_TASKS`` image property, like this:
+
+.. code-block:: console
+
+ gnt-instance add -o snf-image+default \
+ -O img_passwd=1Ki77y,img_format=diskdump,img_id=debian_base-7.0-x86_64,img_properties='{"EXCLUDE_ALL_TASKS":"yes"}' \
+ -t plain --disk=0:size=10G --no-name-check --no-ip-check --no-nics my_debian_server2
+
+To configure a VM without first copying an image into the hard disk (e.g. if
+the hard disk is a snapshot from an existing VM's hard disk) you may use the
+*null* storage back-end like this:
+
+.. code-block:: console
-``gnt-instance add -o snf-image+default --os-parameters img_passwd=SamplePassw0rd,img_format=diskdump,img_id=debian_base-6.0-7-x86_64,img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"}' -t plain --disk=0:size=10G --no-name-check --no-ip-check --no-nics test1``
+ gnt-instance add -o snf-image+default \
+ -O img_passwd=1Ki77y,img_format=diskdump,img_id=null,img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"\,"USERS":"root"}' \
+ -t plain --disk=0:size=10G --no-name-check --no-ip-check --no-nics my_debian_server3
dev="$1"
table=$(get_partition_table "$dev")
+ if [ -z "$table" ]; then
+ return 0
+ fi
if [ $(get_partition_count "$table") -eq 0 ]; then
return 0
log_error "Required image property \`ROOT_PARTITION' is missing or empty."
fi
- export SNF_IMAGE_RESIZE_PART="$(get_partition_to_resize "$SNF_IMAGE_DEV")"
-
if [[ ! "$SNF_IMAGE_PROPERTY_OSFAMILY" =~ ^(linux|windows|freebsd)$ ]]; then
log_error "Supported values for OSFAMILY property are: linux|windows|freebsd"
fi
+ SNF_IMAGE_RESIZE_PART="$(get_partition_to_resize "$SNF_IMAGE_DEV")"
+ if [ -z "$SNF_IMAGE_RESIZE_PART" ]; then
+ exit 0
+ fi
+
+ export SNF_IMAGE_RESIZE_PART
+
# If something goes wrong with the tasks, try to umount the disk file
# systems that are still mounted.
trap '{ umount_all "$target"; }' ERR
echo "[ERROR] $*" >&2
}
+log_warning() {
+ echo "[WARNING] $*" >&2
+}
+
report_error() {
ERROR_MSGS+=("$@")
}
if [ -n "$CONFIG_URL" ]; then
local config config_params
+ echo "Downloading configuration parameters from: \`$CONFIG_URL'" >&2
config=$($CURL -f "$CONFIG_URL")
config_params=$(./decode-config.py $osparams <<< "$config")
eval "$config_params"
fi
}
+canonicalize() {
+ local name="$1"
+
+ if [ -d "$name" ]; then
+ name="$name/"
+ fi
+
+ local dname="${name%/*}"
+ local fname="${name##*/}"
+
+ if [ "x$dname" = "x" -a "${name:0:1}" = "/" ]; then
+ dname="/"
+ fi
+
+ if [ -d "$dname" ]; then
+ (cd -- "$dname" && echo "${PWD%/}/$fname")
+ else
+ echo
+ fi
+}
+
# this one is only to be called by create
ganeti_os_main() {
if [ -z "$OS_API_VERSION" -o "$OS_API_VERSION" = "5" ]; then
PS4='$(date "+%s.%N ($LINENO) + ")'
set -x
elif [ "$IMAGE_DEBUG" != "no" ]; then
- echo "Warning: Unsupported IMAGE_DEBUG value: \`$IMAGE_DEBUG'"
+ log_warning "Unsupported IMAGE_DEBUG value: \`$IMAGE_DEBUG'"
fi
monitor_pipe=$(mktemp -u)
trap report_and_cleanup EXIT
+echo "Processing image with ID: \`$IMG_ID' and type: \`$IMAGE_TYPE'" >&2
+
case $BACKEND_TYPE in
local)
- image_file="$IMAGE_DIR/$(echo "$IMAGE_NAME" | sed 's/^file://').$IMAGE_TYPE"
- if [ ! -e "$image_file" ]; then
- log_error "Image file \`$image_file' does not exit."
+ if [[ "$IMAGE_NAME" =~ ^local:// ]]; then
+ IMAGE_NAME="${IMAGE_NAME:8}"
+ elif [[ "$IMAGE_NAME" =~ ^file:// ]]; then
+ IMAGE_NAME="${IMAGE_NAME:7}"
+ log_warning "The file:// back-end identifier is deprecated and" \
+ "will be removed in the future. Use local:// instead."
+ fi
+
+ canonical_image_dir="$(canonicalize "$IMAGE_DIR")"
+ if [ ! -d "$canonical_image_dir" ]; then
+ log_error "The IMAGE_DIR directory: \`$IMAGE_DIR' does not exist."
report_error "Unable to retrieve image file."
+ fi
+
+ image_file="$IMAGE_DIR/$IMAGE_NAME"
+ if [ ! -e "$image_file" ]; then
+ if [ -e "$image_file.$IMAGE_TYPE" ] ; then
+ image_file="$image_file.$IMAGE_TYPE"
+ log_warning "The \`.$IMAGE_TYPE' extension is missing from" \
+ "the local back-end id. This id form is deprecated and" \
+ " will be remove in the future."
+ else
+ log_error "Image file \`$image_file' does not exist."
+ report_error "Unable to retrieve image file."
+ exit 1
+ fi
+ fi
+
+ canonical_image_file="$(canonicalize "$image_file")"
+
+ if [[ "$canonical_image_file" != "$canonical_image_dir"* ]]; then
+ log_error "Image ID points to a file outside the image directory: \`$IMAGE_DIR'"
+ report_error "Invalid image ID"
exit 1
fi
+
image_size="$(stat -L -c %s "$image_file")"
;;
null)
the URL as the user when connecting to the backend.
"""
-from optparse import OptionParser
+from optparse import OptionParser, OptionGroup
from sys import exit, stdout, stderr
from os import environ
from binascii import hexlify, unhexlify
parser = OptionParser(usage='%prog [options] <URL>')
-parser.add_option('--db', dest='db', metavar='URI',
- help='SQLAlchemy URI of the database [REQUIRED]')
parser.add_option('--data', dest='data', metavar='DIR',
- help='path to the directory where data are'
- ' stored [REQUIRED]')
+ help='path to the directory where data are stored')
parser.add_option('-s', action='store_true', dest='size', default=False,
help='print file size and exit')
+group = OptionGroup(
+ parser, "Dangerous Options",
+ "Caution: If the <URL> is a LocationURL (pithos://...), then you'll also "
+ "need to define a database URI. You can use the `--db' option to do so, "
+ "but this raises security concerns. For database URI's and pithos data "
+ "paths, the recommended way to define them is to use the PITHCAT_INPUT_DB "
+ "and PITHCAT_INPUT_DATA environmental variables respectfully.")
+group.add_option('--db', dest='db', metavar='URI',
+ help='SQLAlchemy URI of the database', default=None)
+parser.add_option_group(group)
LocationURL = namedtuple('LocationURL', ['account', 'container', 'object'])
HashmapURL = namedtuple('HashmapURL', ['hash', 'size'])
url = parse_url(args[0])
if not options.data and 'PITHCAT_INPUT_DATA' not in environ:
- stderr.write("Pithos data directory path is missing.\n")
+ stderr.write(
+ "Pithos data directory path is missing.\n"
+ "Use the PITHCAT_INPUT_DATA environmental variable (recommended) "
+ "or the --data command line option to define it.\n")
exit(1)
data_path = environ['PITHCAT_INPUT_DATA'] if not options.data else \
options.data
- if not options.db and 'PITHCAT_INPUT_DB' not in environ:
- stderr.write("Pithos database uri is missing.\n")
+ if options.db is None and 'PITHCAT_INPUT_DB' not in environ and \
+ type(url) is LocationURL:
+ stderr.write(
+ "Pithos database uri is missing.\n"
+ "Use the PITHCAT_INPUT_DB environmental variable (recommended) "
+ "or the --db command line option to define it.\n")
exit(1)
db_uri = environ['PITHCAT_INPUT_DB'] if not options.db else options.db
+ print "%s" % db_uri
+
backend = ModularBackend(None,
- db_uri if type(url) is LocationURL
- else None,
+ db_uri if type(url) is LocationURL else None,
None,
data_path)
if __name__ == '__main__':
main()
+
+# vim: set sta sts=4 shiftwidth=4 sw=4 et ai :
-m4_define([devflow_version], [0.10.1])
+m4_define([devflow_version], [0.11])
projects / snf-image / commitdiff