Statistics
| Branch: | Tag: | Revision:

root / doc / design-ovf-support.rst @ 54f834df

History | View | Annotate | Download (17.8 kB)

1
==============================================================
2
Ganeti Instance Import/Export using Open Virtualization Format
3
==============================================================
4

    
5
Background
6
==========
7

    
8
Open Virtualization Format is an open standard for packaging
9
information regarding virtual machines. It is used, among other, by
10
VMWare, VirtualBox and XenServer. OVF allows users to migrate between
11
virtualization software without the need of reconfiguring hardware,
12
network or operating system.
13

    
14
Currently, exporting instance in Ganeti results with a configuration
15
file that is readable only for Ganeti. It disallows the users to
16
change the platform they use without loosing all the machine's
17
configuration.  Import function in Ganeti is also currently limited to
18
the previously prepared instances.
19

    
20
Implementation of OVF support allows users to migrate to Ganeti from
21
other platforms, thus potentially increasing the usage. It also
22
enables virtual machine end-users to create their own machines
23
(e.g. in VirtualBox or SUSE Studio) and then add them to Ganeti
24
cluster, thus providing better personalization.
25

    
26
Overview
27
========
28

    
29
Open Virtualization Format description
30
--------------------------------------
31

    
32
According to the DMTF document introducing the standard: "The Open
33
Virtualization Format (OVF) Specification describes an open, secure,
34
portable, efficient and extensible format for the packaging and
35
distribution of software to be run in virtual machines."  OVF supports
36
both single and multiple- configurations of VMs in one package, is
37
host- and virtualization platform-independent and optimized for
38
distribution (e.g. by allowing usage of public key infrastructure and
39
providing tools for management of basic software licensing).
40

    
41
There are no limitations regarding disk images used, as long as the
42
description is provided. Any hardware described in a proper format
43
(i.e. CIM - Common Information Model) is accepted, although there is no
44
guarantee that every virtualization software will support all types of
45
hardware.
46

    
47
OVF package should contain exactly one file with ``.ovf`` extension,
48
which is an XML file specifying the following (per virtual machine):
49

    
50
- virtual disks
51
- network description
52
- list of virtual hardware
53
- operating system, if any
54

    
55
Each of the elements in ``.ovf`` file may, if desired, contain a
56
human-readable description to every piece of information given.
57

    
58
Additionally, the package may have some disk image files and other
59
additional resources (e.g. ISO images).
60

    
61
In order to provide secure means of distribution for OVF packages, the
62
manifest and certificate are provided. Manifest (``.mf`` file) contains
63
checksums for all the files in OVF package, whereas certificate
64
(``.cert`` file) contains X.509 certificate and a checksum of manifest
65
file. Both files are not compulsory, but certificate requires manifest
66
to be present.
67

    
68
Supported disk formats
69
----------------------
70

    
71
Although OVF is claimed to support 'any disk format', what we are
72
interested in is which formats are supported by VM managers that
73
currently use OVF.
74

    
75
- VMWare: ``.vmdk`` (which comes in at least 3 different flavours:
76
  ``sparse``, ``compressed`` and ``streamOptimized``)
77
- VirtualBox: ``.vdi`` (VirtualBox's format), ``.vmdk``, ``.vhd``
78
  (Microsoft and XenServer); export disk format is always ``.vmdk``
79
- XenServer: ``.vmdk``, ``.vhd``; export disk format is always
80
  ``.vhd``
81
- Red Hat Enterprise Virtualization: ``.raw`` (raw disk format),
82
  ``.cow`` (qemu's ``QCOW2``)
83
- other: AbiCloud, OpenNode Cloud, SUSE Studio, Morfeo Claudia,
84
  OpenStack: mostly ``.vmdk``
85

    
86
In our implementation of the OVF we allow a choice between raw, cow and
87
vmdk disk formats for both import and export. Other formats covertable
88
using ``qemu-img`` are allowed, but not tested.
89
The justification is the following:
90

    
91
- Raw format is supported as it is the main format of disk images used
92
  in Ganeti, thus it is effortless to provide support for this format
93
- Cow is used in Qemu
94
- Vmdk is most commonly supported in virtualization software, it also
95
  has the advantage of producing relatively small disk images, which
96
  is extremely important advantage when moving instances.
97

    
98
Import and export - the closer look
99
===================================
100

    
101
This section contains an overview of how different parts of
102
Ganeti's export info are included in ``.ovf`` configuration file.
103
It also explains how import is designed to work with incomplete
104
information.
105

    
106
Ganeti's backup format vs OVF
107
-----------------------------
108
.. highlight:: xml
109

    
110
The basic structure of Ganeti ``.ovf`` file is the following::
111

    
112
    <Envelope>
113
        <References></References>
114
        <DiskSection></DiskSection>
115
        <NetworkSection></NetworkSection>
116
        <VirtualSystem>
117
            <Name></Name>
118
            <OperatingSystemSection></OperatingSystemSection>
119
            <VirtualHardwareSection><VirtualHardwareSection>
120
        </VirtualSystem>
121
        <gnt:GanetiSection>
122
            <gnt:VersionId/>
123
            <gnt:AutoBalance/>
124
            <gnt:Tags></gnt:Tags>
125
            <gnt:DiskTemplate</gnt:DiskTemplate>
126
            <gnt:OperatingSystem>
127
                <gnt:Name/>
128
                <gnt:Parameters></gnt:Parameters>
129
            </gnt:OperatingSystem>
130
            <gnt:Hypervisor>
131
                <gnt:Name/>
132
                <gnt:Parameters></gnt:Parameters>
133
            </gnt:Hypervisor>
134
            <gnt:Network>
135
            <gnt:Mode/>
136
            <gnt:MACAddress/>
137
            <gnt:Link/>
138
            <gnt:IPAddress/>
139
            </gnt:Network>
140
        </gnt:GanetiSection>
141
    </Envelope>
142

    
143
.. note ::
144
    Tags with ``gnt:`` prefix are Ganeti-specific and are not a part of
145
    OVF standard.
146

    
147
.. highlight:: text
148

    
149
Whereas Ganeti's export info is of the following form, ``=>`` showing
150
where will the data be in OVF format::
151

    
152
  [instance]
153
      disk0_dump = filename     => File in References
154
      disk0_ivname = name       => generated automatically
155
      disk0_size = size_in_mb   => calculated after disk conversion
156
      disk_count = number       => generated automatically
157
      disk_template = disk_type => gnt:DiskTemplate
158
      hypervisor = hyp-name     => gnt:Name in gnt:Hypervisor
159
      name = inst-name          => Name in VirtualSystem
160
      nic0_ip = ip              => gnt:IPAddress in gnt:Network
161
      nic0_link = link          => gnt:Link in gnt:Network
162
      nic0_mac = mac            => gnt:MACAddress in gnt:Network or
163
                                   Item in VirtualHardwareSection
164
      nic0_mode = mode          => gnt:Mode in gnt:Network
165
      nic_count = number        => generated automatically
166
      tags                      => gnt:Tags
167

    
168
  [backend]
169
      auto_balanced             => gnt:AutoBalance
170
      memory = mem_in_mb        => Item in VirtualHardwareSection
171
      vcpus = number            => Item in VirtualHardwareSection
172

    
173
  [export]
174
      compression		=> ignored
175
      os                        => gnt:Name in gnt:OperatingSystem
176
      source                    => ignored
177
      timestamp                 => ignored
178
      version                   => gnt:VersionId or
179
                                   constants.EXPORT_VERSION
180

    
181
  [os]                          => gnt:Parameters in gnt:OperatingSystem
182

    
183
  [hypervisor]                  => gnt:Parameters in gnt:Hypervisor
184

    
185
In case of multiple networks/disks used by an instance, they will
186
all be saved in appropriate sections as specified above for the first
187
network/disk.
188

    
189
Import from other virtualization software
190
-----------------------------------------
191
In case of importing to Ganeti OVF package generated in other software,
192
e.g. VirtualBox, some fields required for Ganeti to properly handle
193
import may be missing. Most often it will happen that such OVF package
194
will lack the ``gnt:GanetiSection``.
195

    
196
If this happens you can specify all the missing parameters in
197
the command line. Please refer to `Command Line`_ section.
198

    
199
In the :doc:`ovfconverter` we provide examples of
200
options when converting from VirtualBox, VMWare and OpenSuseStudio.
201

    
202
Export to other virtualization software
203
---------------------------------------
204
When exporting to other virtualization software, you may notice that
205
there is a section ``gnt:GanetiSection``, containing Ganeti-specific
206
information. This may on **rare** cases cause trouble in importing your
207
instance. If that is the case please do one of the two:
208

    
209
1. Export from Ganeti to OVF with ``--external`` option - this will
210
cause to skip the non-standard information.
211

    
212
2. Manually remove the gnt:GanetiSection from the ``.ovf`` file. You
213
will also have to recompute sha1 sum (``sha1sum`` command) of the .ovf
214
file and update your ``.mf`` file with new value.
215

    
216
.. note::
217
    Manual change option is only recommended when you have exported your
218
    instance with ``-format`` option other that ``raw`` or selected
219
    ``--compress``. It saves you the time of converting or compressing
220
    the disk image.
221

    
222
Planned limitations
223
===================
224

    
225
The limitations regarding import of the OVF instances generated
226
outside Ganeti will be (in general) the same, as limitations for
227
Ganeti itself.  The desired behavior in case of encountering
228
unsupported element will be to ignore this element's tag without
229
interruption of the import process.
230

    
231
Package
232
-------
233

    
234
There are no limitations regarding support for multiple files in
235
package or packing the OVF package into one OVA (Open Virtual
236
Appliance) file. As for certificates and licenses in the package,
237
their support will be under discussion after completion of the basic
238
features implementation.
239

    
240
Multiple Virtual Systems
241
------------------------
242

    
243
At first only singular instances (i.e. VirtualSystem, not
244
VirtualSystemCollection) will be supported. In the future multi-tiered
245
appliances containing whole nodes (or even clusters) are considered an
246
option.
247

    
248
Disks
249
-----
250

    
251
As mentioned, Ganeti will allow export in  ``raw``, ``cow`` and ``vmdk``
252
formats.  This means i.e. that the appropriate ``ovf:format``
253
will be provided. It does not mean that other formats cannot be used,
254
rather that we did not test them.
255
As for import, we will support all formats that ``qemu-img`` can convert
256
to ``raw``. At this point this means ``raw``, ``cow``, ``qcow``,
257
``qcow2``, ``vmdk`` and ``cloop``.  We do not plan for now to support
258
``vdi`` or ``vhd`` unless they become part of qemu-img supported formats.
259

    
260
We plan to support compression both for import and export - in gzip
261
format. There is also a possibility to provide virtual disk in chunks
262
of equal size. The latter will not be implemented in the first version,
263
but we do plan to support it eventually.
264

    
265

    
266
The ``ovf:format`` tag is not used in our case when importing. Instead
267
we use ``qemu-img info``, which provides enough information for our
268
purposes and is better standardized.
269

    
270
Please note, that due to security reasons we require the disk image to
271
be in the same directory as the ``.ovf`` description file.
272

    
273
In order to completely ignore disk-related information in resulting
274
config file, please use ``--disk-template=diskless`` option.
275

    
276
Network
277
-------
278

    
279
Ganeti provides support for routed and bridged mode for the networks.
280
Since the standard OVF format does not contain any information regarding
281
used network type, we add our own source of such information in
282
``gnt:GanetiSection``. In case this additional information is not
283
present, we perform a simple check - if network name specified in
284
``NetworkSection`` contains words ``bridged`` or ``routed``, we consider
285
this to be the network type. Otherwise option ``auto`` is chosen, in
286
which case the cluster's default value for that field will be used when
287
importing.
288
This provides a safe fallback in case of NAT networks usage, which are
289
commonly used e.g. in VirtualBox.
290

    
291
Hardware
292
--------
293

    
294
The supported hardware is limited to virtual CPUs, RAM memory, disks and
295
networks. In particular, no USB support is currently provided, as Ganeti
296
does not support them.
297

    
298
Operating Systems
299
-----------------
300

    
301
Support for different operating systems depends solely on their
302
accessibility for Ganeti instances. List of installed OSes can be
303
checked using ``gnt-os list`` command.
304

    
305
Other
306
-----
307

    
308
The instance name (``gnt:VirtualSystem\gnt:Name`` or command line's
309
``--name`` option ) has to be resolvable in order for successful import
310
using ``gnt-backup import``.
311

    
312

    
313
_`Command Line`
314
===============
315

    
316
The basic usage of the ovf tool is one of the following::
317

    
318
    ovfconverter import filename
319
    ovfconverter export --format=<format> filename
320

    
321
This will result in a conversion based solely on the content of provided
322
file. In case some information required to make the conversion is
323
missing, an error will occur.
324

    
325
If output directory should be different than the standard Ganeti export
326
directory (usually ``/srv/ganeti/export``), option ``--output-dir``
327
can be used.
328

    
329
If name of resulting entity should be different than the one read from
330
the file, use ``--name`` option.
331

    
332
Import options
333
--------------
334

    
335
Import options that ``ovfconverter`` supports include options for
336
backend, disks, hypervisor, networks and operating system. If an option
337
is given, it overrides the values provided in the OVF file.
338

    
339
Backend
340
^^^^^^^
341
``--backend=option=value`` can be used to set auto balance, number of
342
vcpus and amount of RAM memory.
343

    
344
Please note that when you do not provide full set of options, the
345
omitted ones will be set to cluster defaults (``auto``).
346

    
347
Disks
348
^^^^^
349
``--disk-template=diskless`` causes the converter to ignore all other
350
disk option - both from .ovf file and the command line. Other disk
351
template options include ``plain``, ``drdb``, ``file``, ``sharedfile``
352
and ``blockdev``.
353

    
354
``--disk=number:size=value`` causes to create disks instead of
355
converting them from OVF package; numbers should start with ``0`` and be
356
consecutive.
357

    
358
Hypervisor
359
^^^^^^^^^^
360
``-H hypervisor_name`` and ``-H hypervisor_name:option=value``
361
provide options for hypervisor.
362

    
363
Network
364
^^^^^^^
365
``--no-nics`` option causes converter to ignore any network information
366
provided.
367

    
368
``--network=number:option=value`` sets network information according to
369
provided data, ignoring the OVF package configuration.
370

    
371
Operating System
372
^^^^^^^^^^^^^^^^
373
``--os-type=type`` sets os type accordingly, this option is **required**
374
when importing from OVF instance not created from Ganeti config file.
375

    
376
``--os-parameters`` provides options for chosen operating system.
377

    
378
Tags
379
^^^^
380
``--tags=tag1,tag2,tag3`` is a means of providing tags specific for the
381
instance.
382

    
383

    
384
After the conversion is completed, you may use ``gnt-backup import`` to
385
import the instance into Ganeti.
386

    
387
Example::
388

    
389
	ovfconverter import file.ovf --disk-template=diskless \
390
          --os-type=lenny-image \
391
          --backend=vcpus=1,memory=512,auto_balance \
392
          -H:xen-pvm \
393
          --net=0:mode=bridged,link=xen-br0 \
394
          --name=xen.i1
395
	[...]
396
	gnt-backup import xen.i1
397
	[...]
398
	gnt-instance list
399

    
400
Export options
401
--------------
402
Export options include choice of disk formats to convert the disk image
403
(``--format``) and compression of the disk into gzip format
404
(``--compress``). User has also the choice of allowing to skip the
405
Ganeti-specific part of the OVF document (``--external``).
406

    
407
By default, exported OVF package will not be contained in the OVA
408
package, but this may be changed by adding ``--ova`` option.
409

    
410
Please note that in order to create an OVF package, it is first
411
required that you export your VM using ``gnt-backup export``.
412

    
413
Example::
414

    
415
	gnt-backup export -n node1.xen xen.i1
416
	[...]
417
	ovfconverter export --format=vmdk --ova --external \
418
	  --output-dir=~/xen.i1 \
419
	  /srv/ganeti/export/xen.i1.node1.xen/config.ini
420

    
421
Implementation details
422
======================
423

    
424
Disk conversion
425
---------------
426

    
427
Disk conversion for both import and export is done using external tool
428
called qemu-tools. The same tool is used to determine the type of disk,
429
as well as its virtual size.
430

    
431

    
432
Import
433
------
434

    
435
Import functionality is implemented using two classes - OVFReader and
436
OVFImporter.
437

    
438
OVFReader class is used to read the contents of the ``.ovf`` file. Every
439
action that requires ``.ovf`` file access is done through that class.
440
It also performs validation of manifest, if one is present.
441

    
442
The result of reading some part of file is typically a dictionary or a
443
string, containing options which correspond to the ones in
444
``config.ini`` file. Only in case of disks, the resulting value is
445
different - it is then a list of disk names. The reason for that is the
446
need for conversion.
447

    
448
OVFImporter class performs all the command-line-like tasks, such as
449
unpacking OVA package, removing temporary directory, converting disk
450
file to raw format or saving the configuration file on disk.
451
It also contains a set of functions that read the options provided in
452
the command line.
453

    
454

    
455
Typical workflow for the import is very simple:
456

    
457
- read the ``.ovf`` file into memory
458
- verify manifest
459
- parse each element of the configuration file: name, disk template,
460
  hypervisor, operating system, backend parameters, network and disks
461

    
462
    - check if option for the element can be read from command line
463
      options
464

    
465
		- if yes: parse options from command line
466

    
467
		- otherwise: read the appropriate portion of ``.ovf`` file
468

    
469
- save gathered information in ``config.ini`` file
470

    
471
Export
472
------
473

    
474
Similar to import, export functionality also uses two classes -
475
OVFWriter and OVFExporter.
476

    
477
OVFWriter class produces XML output based on the information given. Its
478
sole role is to separate the creation of ``.ovf`` file content.
479

    
480
OVFExporter class gathers information from ``config.ini`` file or
481
command line and performs necessary operations like disk conversion, disk
482
compression, manifest creation and OVA package creation.
483

    
484
Typical workflow for the export is even simpler, than for the import:
485

    
486
- read the ``config.ini`` file into memory
487
- gather information about certain parts of the instance, convert and
488
  compress disks if desired
489
- save each of these elements as a fragment of XML tree
490
- save the XML tree as ``.ovf`` file
491
- create manifest file and fill it with appropriate checksums
492
- if ``--ova`` option was chosen, pack the results into ``.ova`` tarfile
493

    
494

    
495
.. vim: set textwidth=72 :
496
.. Local Variables:
497
.. mode: rst
498
.. fill-column: 72
499
.. End: