X-Git-Url: https://code.grnet.gr/git/snf-image-creator/blobdiff_plain/fa77d79a98c00b71221870f0ed7b6a6ca7b23f41..97d863ff7ce616ce9f595ba08a6460e0d08d2d3d:/docs/usage.rst diff --git a/docs/usage.rst b/docs/usage.rst index a7c371e..679b319 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -2,9 +2,14 @@ Usage ^^^^^ snf-image-creator comes in 2 variants: + * snf-image-creator: A non-interactive command line program * snf-mkimage: A user-friendly dialog-based program +Both expect the input media as first argument. The input media may be a local +file, a block device or *"/"* if you want to create an image out of the running +system (see `host bundling operation`_). + Non-interactive version ======================= @@ -12,124 +17,135 @@ snf-image-creator receives the following options: .. code-block:: console - $ snf-image-creator --help - - Usage: snf-image-creator [options] - - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -o FILE, --outfile=FILE - dump image to FILE - -f, --force overwrite output files if they exist - -s, --silent silent mode, only output errors - -u FILENAME, --upload=FILENAME - upload the image to pithos with name FILENAME - -r IMAGENAME, --register=IMAGENAME - register the image with ~okeanos as IMAGENAME - -a ACCOUNT, --account=ACCOUNT - Use this ACCOUNT when uploading/registering images - [Default: None] - -m KEY=VALUE, --metadata=KEY=VALUE - Add custom KEY=VALUE metadata to the image - -t TOKEN, --token=TOKEN - Use this token when uploading/registering images - [Default: None] - --print-sysprep print the enabled and disabled system preparation - operations for this input media - --enable-sysprep=SYSPREP - run SYSPREP operation on the input media - --disable-sysprep=SYSPREP - prevent SYSPREP operation from running on the input - media - --no-sysprep don't perform system preparation - --no-shrink don't shrink any partition - - -Most input options are self-describing. If you want to save a local copy for -the image, you need to specify *-o* option. In order to upload the image to -pithos, you need to specify valid credentials with *-a* and *-t* options and a -filename using *-u* option. To also register the image with ~okeanos, specify a -name using the *-r* option. - -By default snf-image-creator will run a number of system preparation -preparations on the snapshot of the media and will shrink the last partition -found, before extracting the image. Both can be disabled by specifying + $ snf-image-creator --help + Usage: snf-image-creator [options] + + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -o FILE, --outfile=FILE + dump image to FILE + -f, --force overwrite output files if they exist + -s, --silent output only errors + -u FILENAME, --upload=FILENAME + upload the image to the storage service with name FILENAME + -r IMAGENAME, --register=IMAGENAME + register the image with the compute service as IMAGENAME + -m KEY=VALUE, --metadata=KEY=VALUE + add custom KEY=VALUE metadata to the image + -t TOKEN, --token=TOKEN + use this authentication token when + uploading/registering images + -a URL, --authentication-url=URL + use this authentication URL when uploading/registering + images + -c CLOUD, --cloud=CLOUD + use this saved cloud account to authenticate against a + cloud when uploading/registering images + --print-sysprep print the enabled and disabled system preparation + operations for this input media + --enable-sysprep=SYSPREP + run SYSPREP operation on the input media + --disable-sysprep=SYSPREP + prevent SYSPREP operation from running on the input + media + --no-sysprep don't perform any system preparation operation + --no-shrink don't shrink any partition + --public register image with the compute service as public + --tmpdir=DIR create large temporary image files under DIR + +Most input options are self-describing. If you want to save a local copy of +the image you create, provide a filename using the *-o* option. To upload the +image to the storage service of a cloud, provide valid cloud API access info +(by either using a token and a URL with *-t* and *-a* respectively, or a cloud +name with *-c*) and a remote filename using *-u*. If you also want to register +the image with the compute service of the cloud, in addition to *-u* provide a +registration name using *-r*. All images are +registered as *private*. Only the user that registers the image can create +VM's out of it. If you want the image to be visible by other user too, use the +*--public* option. + +By default, before extracting the image, snf-image-creator will perform a +number of system preparation operations on the snapshot of the media and will +shrink the last partition found. Both actions can be disabled by specifying *--no-sysprep* and *--no-shrink* respectively. -If *--print-sysprep* is defined, then snf-image-creator will only run the OS -detection part and will output the system preparation operation that would and -would not run during image creation. This behavior is, convenient because it -allows you to see the available system preparation tasks that you can enable or -disable with *-{enable,disable}-sysprep* options when you create a new image. +If *--print-sysprep* is defined, the program will exit after printing a +list of enabled and disabled system preparation operation applicable to this +input media. The user can enable or disable specific *syspreps*, using +*-{enable,disable}-sysprep* options. The user may specify those options +multiple times. Running *snf-image-creator* with *--print-sysprep* on a raw file that hosts a -debian system, we get the following output: +debian system, we print the following output: -.. code-block:: console +.. _sysprep: - $ snf-image-creator --print-sysprep debian_desktop.img +.. code-block:: console - snf-image-creator 0.1 + $ snf-image-creator --print-sysprep ubuntu.raw + snf-image-creator 0.3 ===================== - Examining source media `debian_desktop.img'... looks like an image file - Snapshotting media source... done + Examining source media `ubuntu_hd.raw' ... looks like an image file + Snapshotting media source ... done Enabling recovery proc - Launching helper VM... done - Inspecting Operating System... found a(n) debian system - Mounting image... done + Launching helper VM (may take a while) ... done + Inspecting Operating System ... ubuntu + Mounting the media read-only ... done + Collecting image metadata ... done + Umounting the media ... done Enabled system preparation operations: cleanup-cache: - Remove all regular files under /var/cache + Remove all regular files under /var/cache cleanup-log: - Empty all files under /var/log + Empty all files under /var/log cleanup-passwords: - Remove all passwords and lock all user accounts + Remove all passwords and lock all user accounts cleanup-tmp: - Remove all files under /tmp and /var/tmp + Remove all files under /tmp and /var/tmp cleanup-userdata: - Delete sensitive userdata + Delete sensitive userdata fix-acpid: - Replace acpid powerdown action scripts to immediately shutdown the - system without checking if a GUI is running. + Replace acpid powerdown action scripts to immediately shutdown the + system without checking if a GUI is running. remove-persistent-net-rules: - Remove udev rules that will keep network interface names persistent - after hardware changes and reboots. Those rules will be created again - the next time the image runs. + Remove udev rules that will keep network interface names persistent + after hardware changes and reboots. Those rules will be created again + the next time the image runs. remove-swap-entry: - Remove swap entry from /etc/fstab. If swap is the last partition - then the partition will be removed when shrinking is performed. If the - swap partition is not the last partition in the disk or if you are not - going to shrink the image you should probably disable this. + Remove swap entry from /etc/fstab. If swap is the last partition + then the partition will be removed when shrinking is performed. If the + swap partition is not the last partition in the disk or if you are not + going to shrink the image you should probably disable this. use-persistent-block-device-names: - Scan fstab & grub configuration files and replace all non-persistent - device references with UUIDs. + Scan fstab & grub configuration files and replace all non-persistent + device references with UUIDs. Disabled system preparation operations: cleanup-mail: - Remove all files under /var/mail and /var/spool/mail + Remove all files under /var/mail and /var/spool/mail remove-user-accounts: - Remove all user accounts with id greater than 1000 + Remove all user accounts with id greater than 1000 - cleaning up... + cleaning up ... -If I want your images to also have all normal user accounts and all mail files -removed, you can create it specifying the *--enable-sysprep* option like this: +If you want the image to have all normal user accounts and all mail files +removed, you should use *--enable-sysprep* option like this: .. code-block:: console - $ snf-image-creator --enable-sysprep cleanup-mail,remove-user-accounts ... + $ snf-image-creator --enable-sysprep cleanup-mail --enable-sysprep remove-user-accounts ... Dialog-based version ==================== @@ -138,18 +154,27 @@ Dialog-based version .. code-block:: console - $ Usage: snf-mkimage [options] [] + $ snf-mkimage --help + Usage: snf-mkimage [options] [] - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -l FILE, --logfile=FILE - log all messages to FILE + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -l FILE, --logfile=FILE + log all messages to FILE + --tmpdir=DIR create large temporary image files under DIR -If the input media is not specified in the command line, then the user will be -asked to specify it in the first dialog box. After the input media is examined -and the program is initialized, the user is given the choice to run -*snf-mkimage* in *wizard* or *expert* mode. +If the input media is not specified in the command line, in the first dialog +box the user will be asked to specify it: + +.. image:: /snapshots/select_media.png + +The user can select a file (regular or block device) or use the *Bundle Host* +button to create an image out of the running system (see +`Host bundling operation`_). + +After the input media is examined and the program is initialized, the user will +be given the choice to run *snf-mkimage* in *wizard* or *expert* mode. Wizard mode ----------- @@ -157,10 +182,15 @@ Wizard mode When *snf-mkimage* runs in *wizard* mode, the user is just asked to provide the following basic information: - * Name: A short name for image (ex. "Slackware") - * Description: An one line description for the image (ex. "Slackware Linux 14.0 with KDE") - * Account: An ~okeanos account e-mail - * Token: A token corresponding to the account defined previously + * Cloud: The cloud account to use to upload and register the resulting image + * Name: A short name for the image (ex. "Slackware") + * Description: An one-line description for the image + (ex. "Slackware Linux 14.0 with KDE") + * Registration Type: Private or Public + +After confirming, the image will be extracted, uploaded to the storage service +and registered with the compute service of the selected cloud. The user will +also be given the choice to keep a local copy of it. For most users the functionality this mode provides should be sufficient. @@ -168,27 +198,39 @@ Expert mode ----------- Expert mode allows the user to have better control on the image creation -process. In the picture below the main menu can be seen: +process. The main menu can be seen in the picture below: .. image:: /snapshots/main_menu.png -In the *Customize* submenu the user can control: +In the *Customize* sub-menu the user can control: - * The system preparation operations that will run during the image creation process + * The system preparation operations that will be applied on the media * Whether the image will be shrunk or not * The properties associated with the image - * Which configuration tasks will run during image deployment + * The configuration tasks that will run during image deployment -In the *Register* submenu the user can provide: +In the *Register* sub-menu the user can provide: - * The credentials to login to ~okeanos - * A pithos filename for the uploaded diskdump image - * A name for the image to be registered to ~okeanos with + * Which cloud account to use + * A filename for the uploaded *diskdump* image + * A name for the image to use when registering it with the storage service of + the cloud, as well as the registration type (*private* or *public*) -By choosing the *Extract* menu entry the user can dump the image to the local -file system and finally, if the user selects *Reset*, the system will ignore +By choosing the *Extract* menu entry, the user can dump the image to the local +file system. Finally, if the user selects *Reset*, the system will ignore all changes made so far and will start the image creation process again. +Host bundling operation +======================= + +As a new feature in *v0.2*, snf-image-creator can create images out of the host +system that runs the program. This is done either by specifying / as input +media or by using the *Bundle Host* button in the media selection dialog of +snf-mkimage. During this operation, the files of the disk are copied into a +temporary image file, which means that the file system that will host the +temporary image needs to have a lot of free space (see `large temporary files`_ +for more information). + Creating a new image ==================== @@ -197,76 +239,137 @@ disk from the Internet: .. code-block:: console - $ wget http://ubuntureleases.tsl.gr/12.04.1/ubuntu-12.04.1-server-amd64.iso + $ wget http://ubuntureleases.tsl.gr/12.04.2/ubuntu-12.04.2-server-amd64.iso -Create a 2G sparce file to host the new system: +Verify that it has been downloaded correctly: .. code-block:: console - $ truncate -s 2G ubuntu_hd.raw + $ echo 'a8c667e871f48f3a662f3fbf1c3ddb17 ubuntu-12.04.2-server-amd64.iso' > check.md5 + $ md5sum -c check.md5 + +Create a 2G sparse file to host the new system: + +.. code-block:: console + + $ truncate -s 2G ubuntu.raw And install the Ubuntu system on this file: .. code-block:: console - $ sudo kvm -boot d -drive file=ubuntu_hd.raw,format=raw,cache=none,if=virtio \ - -cdrom ubuntu-12.04.1-server-amd64.iso + $ sudo kvm -boot d -drive file=ubuntu.raw,format=raw,cache=none,if=virtio \ + -m 1G -cdrom ubuntu-12.04.2-server-amd64.iso + +.. warning:: + + During the installation, you will be asked about the partition scheme. Don't + use LVM partitions. They are not supported by snf-image-creator. -After the installation finishes, become root, activate the virtual environment -you have installed snf-image-creator in, and use *snf-mkimage* to create and -upload the image: +You will be able to boot your installed OS and make any changes you want +(e.g. install openssh-server) using the following command:: + + $ sudo kvm -m 1G -boot c -drive file=ubuntu.raw,format=raw,cache=none,if=virtio + +After you're done, you may use *snf-mkimage* as root to create and upload the +image: .. code-block:: console $ sudo -s - $ source /path/to/snf-image-env/bin/activate - $ snf-mkimage ubuntu_hd.raw + $ snf-mkimage ubuntu.raw In the first screen you will be asked to choose if you want to run the program in *Wizard* or *Expert* mode. Choose *Wizard*. -.. image:: /snapshots/01_wizard.png +.. image:: /snapshots/wizard.png -Then you will be asked to provide a name, a description, an ~okeanos account -and the token corresponding to this account. After that you will be asked to +Then you will be asked to select a cloud and provide a name, a description and +a registration type (*private* or *public*). Finally, you'll be asked to confirm the provided data. -.. image:: /snapshots/06_confirm.png +.. image:: /snapshots/confirm.png -Choosing *YES* will create the image and upload it to your ~okeanos account. +Choosing *YES* will create and upload the image to your cloud account. -Things you need to pay attention on when creating images -======================================================== +Limitations +=========== -Para-virtualized drivers ------------------------- +Supported operating systems +--------------------------- + +*snf-image-creator* can only fully function on input media hosting *Linux* +systems. The program will detect the needed metadata and you may use it to +upload and register other *Unix* or *Windows* images, but you cannot use it to +shrink them or perform system preparation operations. -~Okeanos uses the VirtIO framework. The disk I/O controller and the Ethernet -cards on the VM instances are para-virtualized and need special VirtIO drivers. -Those drivers are included in the Linux Kernel mainline since version 2.6.25 -and are shipped with all the popular Linux distributions. The problem is that -if those drivers are built as modules, they need to be preloaded using an -initial ramdisk, otherwise the VM will not be able to boot. - -In the image creation demonstration above, we initially installed the Ubuntu -system on a a hard disk (ubuntu_hd.raw) that was para-virtualized (pay -attention on the *if=virtio* option of the kvm line). The Ubuntu installer -detected that the disk was paravirtualized and made sure the appropriate -drivers will be preloaded each time the system boots. In many distros this is -not the case. In Arch Linux for example, the user needs to manually add -*virtio_blk* and *virtio_pci* drivers in */etc/mkinitcpio.conf* and then -rebuild the initial ramdisk [#f1]_ to make the virtio drivers get preloaded -during boot. - -Swap partitions +Logical Volumes --------------- -If you want your image to have a swap partitions, make sure this is the last -partition on the disk. If snf-image-creator detects a swap partition in the end -of the input media, it will remove the partition during shrinking and will save -enough information to be able to recreate it during image deployment. This will -make your image smaller and will speed up the deployment process. +The program cannot work on LVM partitions [#f1]_. The input media may only +contain primary or logical partitions. + +Para-virtualized drivers +------------------------ + +Most synnefo deployments uses the *VirtIO* framework. The disk I/O controller +and the Ethernet cards on the VM instances are para-virtualized and need +special *VirtIO* drivers. Those drivers are included in the Linux Kernel +mainline since version 2.6.25 and are shipped with all the popular Linux +distributions. The problem is that if the driver for the para-virtualized disk +I/O controller is built as module, it needs to be preloaded using an initial +ramdisk, otherwise the VM won't be able to boot. + +Many popular Linux distributions, like Ubuntu and Debian, will automatically +create a generic initial ramdisk file that contains many different modules, +including the VirtIO drivers. Others that target more experienced users, like +Slackware, won't do that [#f2]_. *snf-image-creator* cannot resolve this kind +of problems and it's left to the user to do so. Please refer to your +distribution's documentation for more information on this. You can always check +if a system can boot with para-virtualized disk controller by launching it with +kvm using the *if=virtio* option (see the kvm command in the +`Creating a new image`_ section). + +Some caveats on image creation +============================== + +Image partition schemes and shrinking +------------------------------------- + +When image shrinking is enabled, *snf-image-creator* will shrink the last +partition on the disk. If this is a swap partition, it will remove it, save +enough information to recreate it during image deployment and shrink the +partition that lays just before that. This will make the image smaller which +speeds up the deployment process. + +During image deployment, the last partition is enlarged to occupy the available +space in the VM's hard disk and a swap partition is added at the end if a SWAP +image property is present. + +Keep this in mind when creating images. It's always better to have your swap +partition placed as the last partition on the disk and have your largest +partition (*/* or */home*) just before that. + +Large temporary files +--------------------- + +*snf-image-creator* may create large temporary files when running: + + * During image shrinking, the input media snapshot file may reach the size of + the original media. + * When bundling the host system, the temporary image file may became as large + as the rest of the disk files altogether. + +*/tmp* directory is not a good place for hosting large files. In many systems +the contents of */tmp* are stored in volatile memory and the size they may occupy +is limited. By default, *snf-image-creator* will use a heuristic approach to +determine where to store large temporary files. It will examine the free space +under */var/tmp*, the user's home directory and */mnt* and will pick the one +with the most available space. The user may overwrite this behaviour and +indicate a different directory using the *tmpdir* option. This option is +supported by both *snf-image-creator* and *snf-mkimage*. .. rubric:: Footnotes -.. [#f1] https://wiki.archlinux.org/index.php/KVM#Paravirtualized_guests_.28virtio.29 +.. [#f1] http://sourceware.org/lvm2/ +.. [#f2] http://mirrors.slackware.com/slackware/slackware-14.0/README.initrd