X-Git-Url: https://code.grnet.gr/git/snf-image-creator/blobdiff_plain/9de17202577f52d05e5512531b9edb8177ac0ac8..fab154f00c00474d8baf495cf432dc6054f1bc7e:/docs/usage.rst diff --git a/docs/usage.rst b/docs/usage.rst index 90cf460..c43fb29 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -2,9 +2,14 @@ Usage ^^^^^ snf-image-creator comes in 2 variants: + * snf-image-creator: A non-interactive command line program * snf-mkimage: A user-friendly dialog-based program +Both expect the input media as first argument. The input media may be a local +file, a block device or *"/"* if you want to create an image out of the running +system (see `host bundling operation`_). + Non-interactive version ======================= @@ -13,58 +18,58 @@ snf-image-creator receives the following options: .. code-block:: console $ snf-image-creator --help - Usage: snf-image-creator [options] Options: - --version show program's version number and exit - -h, --help show this help message and exit - -o FILE, --outfile=FILE - dump image to FILE - -f, --force overwrite output files if they exist - -s, --silent silent mode, only output errors - -u FILENAME, --upload=FILENAME - upload the image to pithos with name FILENAME - -r IMAGENAME, --register=IMAGENAME - register the image with ~okeanos as IMAGENAME - -a ACCOUNT, --account=ACCOUNT - Use this ACCOUNT when uploading/registering images - [Default: None] - -m KEY=VALUE, --metadata=KEY=VALUE - Add custom KEY=VALUE metadata to the image - -t TOKEN, --token=TOKEN - Use this token when uploading/registering images - [Default: None] - --print-sysprep print the enabled and disabled system preparation - operations for this input media - --enable-sysprep=SYSPREP - run SYSPREP operation on the input media - --disable-sysprep=SYSPREP - prevent SYSPREP operation from running on the input - media - --no-sysprep don't perform system preparation - --no-shrink don't shrink any partition - - -Most input options are self-describing. If you want to save a local copy for -the image you create, you specify *-o* option. To upload the image to -*pithos+*, you specify valid credentials with *-a* and *-t* options and a -filename using *-u*. If you want to register the image with *~okeanos*, -in addition to *-u* specify a registration name using *-r*. - -By default snf-image-creator will perform a number of system preparation -operations on the snapshot of the media and will shrink the last partition -found, before extracting the image. Both can be disabled by specifying + --version show program's version number and exit + -h, --help show this help message and exit + -o FILE, --outfile=FILE + dump image to FILE + -f, --force overwrite output files if they exist + -s, --silent output only errors + -u FILENAME, --upload=FILENAME + upload the image to pithos with name FILENAME + -r IMAGENAME, --register=IMAGENAME + register the image with ~okeanos as IMAGENAME + -m KEY=VALUE, --metadata=KEY=VALUE + add custom KEY=VALUE metadata to the image + -t TOKEN, --token=TOKEN + use this token when uploading/registering images + [Default: None] + --print-sysprep print the available enabled and disabled system + preparation operations for this input media + --enable-sysprep=SYSPREP + run SYSPREP operation on the input media + --disable-sysprep=SYSPREP + prevent SYSPREP operation from running on the input + media + --no-sysprep don't perform any system preparation operation + --no-shrink don't shrink the image + --public register image with cyclades as public + --tmpdir=DIR create large temporary image files under DIR + +Most input options are self-describing. If you want to save a local copy of +the image you create, provide a filename using the *-o* option. To upload the +image to *pithos+*, provide a valid authentication token using *-t* and a +filename using *-u*. If you also want to register the image with *~okeanos*, in +addition to *-u* provide a registration name using *-r*. All images are +registered as *private*. Only the user that registers the image can create +VM's out of it. If you want the image to be visible by other user too, use the +*--public* option. + +By default, before extracting the image, snf-image-creator will perform a +number of system preparation operations on the snapshot of the media and will +shrink the last partition found. Both actions can be disabled by specifying *--no-sysprep* and *--no-shrink* respectively. -If *--print-sysprep* is defined, the program will exit after outputing a -list of enabled and disabled system preparation operation appliable to this -media source. The user can enable or disable specific *syspreps* when creating -an image, using *-{enable,disable}-sysprep* options. You can specify those -options multiple times to enable or disable multiple *syspreps*. +If *--print-sysprep* is defined, the program will exit after printing a +list of enabled and disabled system preparation operation applicable to this +input media. The user can enable or disable specific *syspreps*, using +*-{enable,disable}-sysprep* options. The user may specify those options +multiple times. Running *snf-image-creator* with *--print-sysprep* on a raw file that hosts a -debian system, we get the following output: +debian system, we print the following output: .. _sysprep: @@ -126,8 +131,8 @@ debian system, we get the following output: cleaning up... -If we want the image to have all normal user accounts and all mail files -removed, we can create it specifying *--enable-sysprep* option like this: +If you want the image to have all normal user accounts and all mail files +removed, you should use *--enable-sysprep* option like this: .. code-block:: console @@ -140,18 +145,27 @@ Dialog-based version .. code-block:: console - $ Usage: snf-mkimage [options] [] + $ snf-mkimage --help + Usage: snf-mkimage [options] [] - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -l FILE, --logfile=FILE - log all messages to FILE + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -l FILE, --logfile=FILE + log all messages to FILE + --tmpdir=DIR create large temporary image files under DIR If the input media is not specified in the command line, in the first dialog -box the user will be asked to specify it. After the input media is examined and -the program is initialized, the user will be given the choice to run -*snf-mkimage* in *wizard* or *expert* mode. +box the user will be asked to specify it: + +.. image:: /snapshots/select_media.png + +The user can select a file (regular or block device) or use the *Bundle Host* +button to create an image out of the running system (see +`Host bundling operation`_). + +After the input media is examined and the program is initialized, the user will +be given the choice to run *snf-mkimage* in *wizard* or *expert* mode. Wizard mode ----------- @@ -160,20 +174,22 @@ When *snf-mkimage* runs in *wizard* mode, the user is just asked to provide the following basic information: * Name: A short name for the image (ex. "Slackware") - * Description: An one-line description for the image (ex. "Slackware Linux 14.0 with KDE") - * Account: An *~okeanos* account email - * Token: A token corresponding to the account defined previously + * Description: An one-line description for the image + (ex. "Slackware Linux 14.0 with KDE") + * Registration Type: Private or Public + * Account: The authentication token for an *~okeanos* account After confirming, the image will be extracted, uploaded to *pithos+* and -registered to *~okeanos*. The user will also be given the choice to keep a local -copy of it. For most users the functionality this mode provides should be -sufficient. +registered with *~okeanos*. The user will also be given the choice to keep a +local copy of it. + +For most users the functionality this mode provides should be sufficient. Expert mode ----------- Expert mode allows the user to have better control on the image creation -process. In the picture below the main menu can be seen: +process. The main menu can be seen in the picture below: .. image:: /snapshots/main_menu.png @@ -186,319 +202,166 @@ In the *Customize* sub-menu the user can control: In the *Register* sub-menu the user can provide: - * The credentials to login to *~okeanos* - * A pithos filename for the uploaded *diskdump* image - * A name for the image to be registered to *~okeanos* with + * The credentials (authentication token) to use when authenticating + to *~okeanos* + * A *pithos+* filename for the uploaded *diskdump* image + * A name for the image to use when registering it with *~okeanos*, as well as + the registration type (*private* or *public*) -By choosing the *Extract* menu entry the user can dump the image to the local -file system and finally, if the user selects *Reset*, the system will ignore +By choosing the *Extract* menu entry, the user can dump the image to the local +file system. Finally, if the user selects *Reset*, the system will ignore all changes made so far and will start the image creation process again. -Usage example -============= - -Supposing you have snf-image-creator installed on a machine (hereafter referred -to as your **host machine**), you can follow the next steps to upload and -register an image of an OS of your preference (hereafter referred to as your -**guest OS**) to your synnefo deployment. - - * `Step 1: Install the guest OS`_ - * `Step 2: Create and upload an image of the guest OS`_ - * `Step 3: Create your VM`_ - - -Step 1: Install the guest OS ------------------------------ - -The guest OS has to be installed on a media such as a block device or a regular -raw file, that can be **accessible** by your host machine. - -But why is accessible empasized? Well, because you don't need to do the -installation of the guest OS on your host machine. You can just as well install -it on a raw file, upload it on Pithos+, download it on your host machine and -use it for Step 2. - -*Note: If you have a guest OS already installed, you may want to skip the -next step. However, be sure to check out the* `Caveats`_ *section, where -some requirements about the guest OS are presented.* - -Installation example -"""""""""""""""""""" +Host bundling operation +======================= -To simplify things a bit, we will install the guest OS on the host machine -where snf-image-creator is installed. We will assume that the host machine is -an Ubuntu 12.04 ~okeanos VM, built with max specifications (4 CPUs, 2GB of -ram, 40GB of disk space at the time of writing this). +As a new feature in *v0.2*, snf-image-creator can create images out of the host +system that runs the program. This is done either by specifying / as input +media or by using the *Bundle Host* button in the media selection dialog of +snf-mkimage. During this operation, the files of the disk are copied into a +temporary image file, which means that the file system that will host the +temporary image needs to have a lot of free space (see `large temporary files`_ +for more information). -*Note: Since the installation of the guest OS will take place on your host -VM, you must be able to connect to it graphically. This is covered by our* -`connection guide `_. +Creating a new image +==================== -For our guest OS, we will choose, Linux Mint, which is the most hyped Linux -OS according to Distrowatch. A new version has just been released, so -this seems like a fine choice. :: +Suppose you want to create a new Ubuntu server image. Download the installation +disk from the Internet: - Warning: The installation might take a long time (~1 hour) and a bit of - lagginess due to nested virtualization is to be expected. +.. code-block:: console -Fire up your terminal, go to your home directory and type the following to -download the Linux Mint live cd:: + $ wget http://ubuntureleases.tsl.gr/12.04.2/ubuntu-12.04.2-server-amd64.iso - $ wget http://ftp.ntua.gr/pub/linux/linuxmint//stable/14/linuxmint-14-mate-dvd-64 bit.iso +Verify that it has been downloaded correctly: -Verify that it has been downloaded correctly. If the following command -prints "OK". then you are good to go:: +.. code-block:: console - $ echo 'b370ac59d1ac6362f1662cfcc22a489c linuxmint-14-mate-dvd-64bit.iso' > check.md5 + $ echo 'a8c667e871f48f3a662f3fbf1c3ddb17 ubuntu-12.04.2-server-amd64.iso' > check.md5 $ md5sum -c check.md5 -Allocate a few gigs of space to create a sparse file:: - - $ truncate -s 7G linuxmint.raw - -Use `kvm` to boot the Live CD:: - - $ sudo kvm -m 1200 -smp 4 -boot d -drive \ - file=linuxmint.raw,format=raw,cache=none,if=virtio \ - -cdrom linuxmint-14-mate-dvd-64bit.iso - - At a glance, let's see what the above options do: - -m 1200: Use 1200MB of RAM for the guest OS. You should - allocate as much as possible - -smp 4: Simulate an SMP system with 4 CPUs for the - guest OS to use. - -boot d: Place cdrom first in the boot order - file=opensuse.raw Use this raw file as the "hard disk" for the - installation - if=virtio: Inform the OS that it should preload the - VirtIO drivers (more on that on `Caveats`_ - section) - -cdrom linuxmint-14-mate-dvd-64bit.iso: - "Insert" Linux Mint's live cd in the cdrom - drive - -Wait a few seconds and then a new screen with the Linux Mint logo should -appear. You don't have to press any key since it will boot automatically to -Linux Mint's live desktop after a few minutes. - -|qemu-live| - -Choose "Install Linux Mint". The installation process should be pretty -straightforward. Just keep two things in mind: - -* The username you choose will also be used later on, when you create a VM - from this OS image. The password, however, will be removed and you will - be given a new one. -* The installed OS must have no more than one primary partition and - optionally a swap partition. You can read more in the `Caveats`_ - section below. You don't have to worry about it in this installation - however, since the default option takes care of that. - - |qemu-partition| - -After the installation is complete, you can close the QEMU window. You -will be able to boot your installed OS and make any changes you want to it -using the following command:: +Create a 2G sparse file to host the new system: - $ sudo kvm -m 1200 -smp 4 -boot d -drive \ - file=linuxmint.raw,format=raw,cache=none,if=virtio - -At the very least, you should install OpenSSH server to connect to your VM -properly. You can install OpenSSH server using the following command:: - - $ sudo apt-get install openssh-server - -Bear in mind that once the OS image has been uploaded to your synnefo -deployment, you will not be able to make changes to it. Since you can only -apply changes to your raw file, you are advised to do so now and then proceed -to Step 2. - -Caveats -""""""" -This is a list of restrictions you must have in mind while installing the -guest OS: - -**Partitions** - -The installation must consist of no more than one primary partition. It -can have a swap partition though, which should ideally - but not -necessarily - be located at the end of the media. In this case, the -uploaded image will be much smaller and the VM deployment process much -faster. - -**VirtIO drivers** - -Depending on your synnefo-deployment, you may need to use para-virtualized -drivers for your storage and network needs. - -~okeanos uses the VirtIO framework which is essential for the ~okeanos VMs -to work properly since their disk I/O controller and Ethernet cards are -para-virtualized. - -Fortunately, you will not need to deal with the installation of VirtIO drivers -directly, since they are included in Linux kernel since version 2.6.25 and -shipped with all the modern Linux distributions. However, if the VirtIO drivers -are built as a module (and most modern OSes do so), they need to be preloaded -using an initial ramdisk (initramfs), otherwise the VM you create from this OS -image will not be able to boot. - -Debian derivatives will create an initial ramdisk with VirtIO included if -they are connected during the installation on a para-virtualized interface -(the "if=virtio" option in the Linux Mint example). - -In many other distros though, this is not the case. In Arch Linux for -example, the user needs to manually add virtio_blk and virtio_pci drivers -in /etc/mkinitcpio.conf and rebuild the initial ramdisk to make the -virtio drivers get preloaded during boot. You can read more in the `Arch -Linux wiki `_ on how to do it. - -For now, snf-image-creator cannot resolve this kind of problems and it's -left to the user to do it. - -Step 2: Create and upload an image of the guest OS --------------------------------------------------- - -This is the step on which we use snf-image-creator. There are actually two -variants of snf-image-creator, `snf-image-creator`_ and `snf-mkimage`_, both -achieving the same results but suited for different needs. Their scope is -documented at the start of the `Usage`_ section of this document. - -*Note: Both tools take a snapshot of the installed OS on the media -provided to them. So, any changes they apply do not affect the OS -installed on the original media.* - -Let's see both tools in action. We will use them to create an image of the -Linux Mint 14 OS we installed in Step 2. - -snf-mkimage -""""""""""" - -In order to use snf-mkimage, simply type:: - - $ sudo snf-mkimage linuxmint.raw - -snf-mkimage will initially check if the media is indeed a single disk -partition or a raw file representing a hard disk. Then, it will use -heuristics to understand which OS has been installed on the media. After -that, you will be asked which mode you prefer. - -|mkimage-wizard| - -* Wizard mode is intuitive and consists of 4 simple steps. -* Expert mode has an abundance of options but requires a bit of knowledge - of the inner workings of Cyclades from your part. You can learn more on the - `Expert Mode`_ section of snf-mkimage. - -For our tutorial, we will use Wizard mode. So, choose "Wizard" and then provide -a name for the image. - -|mkimage1| - -This name will appear on Pithos+ and on the Public Images section of Cyclades. +.. code-block:: console -Then, provide a description for the image. + $ truncate -s 2G ubuntu_hd.raw -|mkimage2| +And install the Ubuntu system on this file: -This will appear under the chosen image name on the Public Images section of -cyclades. +.. code-block:: console -Next, add your account e-mail + $ sudo kvm -boot d -drive file=ubuntu_hd.raw,format=raw,cache=none,if=virtio \ + -m 1G -cdrom ubuntu-12.04.2-server-amd64.iso -|mkimage3| +.. warning:: -... your account token... + During the installation, you will be asked about the partition scheme. Don't + use LVM partitions. They are not supported by snf-image-creator. -|mkimage4| +You will be able to boot your installed OS and make any changes you want +(e.g. install openssh-server) using the following command:: -...and you're done! A list operations will appear on your console. + $ sudo kvm -m 1G -boot c -drive file=ubuntu_hd.raw,format=raw,cache=none,if=virtio -|mkimage-results| +After you're done, you may use *snf-mkimage* as root to create and upload the +image: -We will briefly comment on the above output. +.. code-block:: console -* **Sysprep:** Operations from 1/9 to 9/9 are part of the system - preparation operations and are best explained in the snf-image-creator's - `sysprep`_ section. -* **Shrinking:** When shrinking the image, we check if a swap partition - exists at the end of the media. If this is the case, it will be removed - and re-inserted upon the deployment process of the VM. Alternatively, if - the swap partition lies at the start of the media, it will be left - untouched. On both cases, the primary partition will be shrunken as much - as possible. On this example, we can see that the final size is 3.5GB, - whereas the orginal size was 7GB. This means that the image was reduced - by half, a pretty impressive feat. -* **MD5SUM:** The md5sum of the image is used later on to verify that the - image has been uploaded successfully. -* **Uploading:** Everytime you upload an OS image, every block is hashed, - checked against existing blocks in Pithos+ and finally uploaded, if no - other block has the same hash. + $ sudo -s + $ snf-mkimage ubuntu_hd.raw - *Consider this example: You have just uploaded a Gentoo Linux image but - had forgotten to install a necessary package. In this case, you would - probably edit the OS in the raw file and then use snf-mkimage to upload - the new image. However, since there is an almost identical image already - uploaded on Pithos+, you can just as well upload only the blocks that - differentiate those two images. This is both time and space efficient.* +In the first screen you will be asked to choose if you want to run the program +in *Wizard* or *Expert* mode. Choose *Wizard*. -Finally, after the image has been uploaded successfully, you will be asked -whether you want to save a local copy of the **shrunken** image. This is -just a copy of the diskdump that has been uploaded to Pithos+ and, in case -you are confused, the original OS installed on the media (linuxmint.raw in -our example) remains intact. +.. image:: /snapshots/wizard.png -snf-image-creator -""""""""""""""""" +Then you will be asked to provide a name, a description, a registration type +(*private* or *public*) and the authentication token corresponding to your +*~okeanos* account. Finally, you'll be asked to confirm the provided data. -snf-image-creator is the command-line equivalent of snf-mkimage. All the -info provided in the steps above are given now as options, which makes it -ideal for scripting purposes. The full set of options can be found in the -`Usage section <#non-interactive-version>`_ of snf-image-creator's -documentation. +.. image:: /snapshots/confirm.png -This tool is most commonly used with the following set of options:: +Choosing *YES* will create and upload the image to your *~okeanos* account. - $ sudo snf-image-creator linuxmint.raw -a user@email.com \ - -t hUudl4DEIlomlnvWnv7Rlw== -u linuxmint.diskdump -r "Linux Mint 14 Nadia" +Limitations +=========== -As you can see, these options are exactly what snf-mkimage's steps -translate to. You can also see that the output is nearly identical: +Supported operating systems +--------------------------- -|image-creator| +*snf-image-creator* can only fully function on input media hosting *Linux* +systems. The program will detect the needed metadata and you may use it to +upload and register other *Unix* or *Windows* images, but you cannot use it to +shrink them or perform system preparation operations. -Step 3: Create your VM ----------------------- +Logical Volumes +--------------- -Creating a VM out of an uploaded custom image is a fairly simple task. -Just select "New Machine", go to "My Images" section and select your -image. +The program cannot work on LVM partitions [#f1]_. The input media may only +contain primary or logical partitions. -|custom-vm| +Para-virtualized drivers +------------------------ -Alternatively, if you want to create a VM from another user's custom -image, you can go to the "Public Images" section. +*~Okeanos* uses the *VirtIO* framework. The disk I/O controller and the +Ethernet cards on the VM instances are para-virtualized and need special +*VirtIO* drivers. Those drivers are included in the Linux Kernel mainline since +version 2.6.25 and are shipped with all the popular Linux distributions. The +problem is that if the driver for the para-virtualized disk I/O controller is +built as module, it needs to be preloaded using an initial ramdisk, otherwise +the VM won't be able to boot. -.. |qemu-live| image:: /snapshots/qemu-live.png +Many popular Linux distributions, like Ubuntu and Debian, will automatically +create a generic initial ramdisk file that contains many different modules, +including the VirtIO drivers. Others that target more experienced users, like +Slackware, won't do that [#f2]_. *snf-image-creator* cannot resolve this kind +of problems and it's left to the user to do so. Please refer to your +distribution's documentation for more information on this. You can always check +if a system can boot with para-virtualized disk controller by launching it with +kvm using the *if=virtio* option (see the kvm command in the +`Creating a new image`_ section). -.. |qemu-partition| image:: /snapshots/qemu-partition.png +Some caveats on image creation +============================== -.. |mkimage-wizard| image:: /snapshots/mkimage-wizard.png +Image partition schemes and shrinking +------------------------------------- -.. |mkimage1| image:: /snapshots/mkimage1.png +When image shrinking is enabled, *snf-image-creator* will shrink the last +partition on the disk. If this is a swap partition, it will remove it, save +enough information to recreate it during image deployment and shrink the +partition that lays just before that. This will make the image smaller which +speeds up the deployment process. -.. |mkimage2| image:: /snapshots/mkimage2.png +During image deployment, the last partition is enlarged to occupy the available +space in the VM's hard disk and a swap partition is added at the end if a SWAP +image property is present. -.. |mkimage3| image:: /snapshots/mkimage3.png +Keep this in mind when creating images. It's always better to have your swap +partition placed as the last partition on the disk and have your largest +partition (*/* or */home*) just before that. -.. |mkimage4| image:: /snapshots/mkimage4.png +Large temporary files +--------------------- -.. |mkimage-fin| image:: /snapshots/mkimage-fin.png +*snf-image-creator* may create large temporary files when running: -.. |mkimage-results| image:: /snapshots/mkimage-results.png + * During image shrinking, the input media snapshot file may reach the size of + the original media. + * When bundling the host system, the temporary image file may became as large + as the rest of the disk files altogether. -.. |image-creator| image:: /snapshots/image-creator.png +*/tmp* directory is not a good place for hosting large files. In many systems +the contents of */tmp* are stored in volatile memory and the size they may occupy +is limited. By default, *snf-image-creator* will use a heuristic approach to +determine where to store large temporary files. It will examine the free space +under */var/tmp*, the user's home directory and */mnt* and will pick the one +with the most available space. The user may overwrite this behaviour and +indicate a different directory using the *tmpdir* option. This option is +supported by both *snf-image-creator* and *snf-mkimage*. -.. |custom-vm| image:: /snapshots/custom-vm.png +.. rubric:: Footnotes +.. [#f1] http://sourceware.org/lvm2/ +.. [#f2] http://mirrors.slackware.com/slackware/slackware-14.0/README.initrd