.. contents::
-.. highlight:: text
+.. highlight:: shell-example
Introduction
------------
Node can be added and removed (if they host no instances) at will from
the cluster. In a HA cluster and only with HA instances, the loss of any
single node will not cause disk data loss for any instance; of course,
-a node crash will cause the crash of the its primary instances.
+a node crash will cause the crash of its primary instances.
A node belonging to a cluster can be in one of the following roles at a
given time:
Depending on the role, each node will run a set of daemons:
-- the :command:`ganeti-noded` daemon, which control the manipulation of
+- the :command:`ganeti-noded` daemon, which controls the manipulation of
this node's hardware resources; it runs on all nodes which are in a
cluster
- the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
- nodes, but is only functional on master candidate nodes
+ nodes, but is only functional on master candidate nodes; this daemon
+ can be disabled at configuration time if you don't need its
+ functionality
- the :command:`ganeti-rapi` daemon which runs on the master node and
offers an HTTP-based API for the cluster
- the :command:`ganeti-masterd` daemon which runs on the master node and
the instance sees the same virtual drive in all cases, the node-level
configuration varies between them.
-There are four disk templates you can choose from:
+There are five disk templates you can choose from:
diskless
The instance has no disks. Only used for special purpose operating
file
The instance will use plain files as backend for its disks. No
redundancy is provided, and this is somewhat more difficult to
- configure for high performance.
+ configure for high performance. Note that for security reasons the
+ file storage directory must be listed under
+ ``/etc/ganeti/file-storage-paths``, and that file is not copied
+ automatically to all nodes by Ganeti. The format of that file is a
+ newline-separated list of directories.
+
+sharedfile
+ The instance will use plain files as backend, but Ganeti assumes that
+ those files will be available and in sync automatically on all nodes.
+ This allows live migration and failover of instances using this
+ method. As for ``file`` the file storage directory must be listed under
+ ``/etc/ganeti/file-storage-paths`` or ganeti will refuse to create
+ instances under it.
plain
The instance will use LVM devices as backend for its disks. No
to obtain a highly available instance that can be failed over to a
remote node should the primary one fail.
+ .. note:: Ganeti does not support DRBD stacked devices:
+ DRBD stacked setup is not fully symmetric and as such it is
+ not working with live migration.
+
+rbd
+ The instance will use Volumes inside a RADOS cluster as backend for its
+ disks. It will access them using the RADOS block device (RBD).
+
+ext
+ The instance will use an external storage provider. See
+ :manpage:`ganeti-extstorage-interface(7)` for how to implement one.
+
+
IAllocator
~~~~~~~~~~
information store for helping with cluster administration, for example
by attaching owner information to each instance after it's created::
- gnt-instance add … instance1
- gnt-instance add-tags instance1 owner:user2
+ $ gnt-instance add … %instance1%
+ $ gnt-instance add-tags %instance1% %owner:user2%
And then by listing each instance and its tags, this information could
be used for contacting the users of each instance.
While not directly visible by an end-user, it's useful to know that a
basic cluster operation (e.g. starting an instance) is represented
-internall by Ganeti as an *OpCode* (abbreviation from operation
+internally by Ganeti as an *OpCode* (abbreviation from operation
code). These OpCodes are executed as part of a *Job*. The OpCodes in a
single Job are processed serially by Ganeti, but different Jobs will be
processed (depending on resource availability) in parallel. They will
With the above parameters in mind, the command is::
- gnt-instance add \
- -n TARGET_NODE:SECONDARY_NODE \
- -o OS_TYPE \
- -t DISK_TEMPLATE -s DISK_SIZE \
- INSTANCE_NAME
+ $ gnt-instance add \
+ -n %TARGET_NODE%:%SECONDARY_NODE% \
+ -o %OS_TYPE% \
+ -t %DISK_TEMPLATE% -s %DISK_SIZE% \
+ %INSTANCE_NAME%
The instance name must be resolvable (e.g. exist in DNS) and usually
points to an address in the same subnet as the cluster itself.
The above command has the minimum required options; other options you
can give include, among others:
-- The memory size (``-B memory``)
+- The maximum/minimum memory size (``-B maxmem``, ``-B minmem``)
+ (``-B memory`` can be used to specify only one size)
- The number of virtual CPUs (``-B vcpus``)
- Arguments for the NICs of the instance; by default, a single-NIC
instance is created. The IP and/or bridge of the NIC can be changed
- via ``--nic 0:ip=IP,bridge=BRIDGE``
+ via ``--net 0:ip=IP,link=BRIDGE``
-See the manpage for gnt-instance for the detailed option list.
+See :manpage:`ganeti-instance(8)` for the detailed option list.
For example if you want to create an highly available instance, with a
single disk of 50GB and the default memory size, having primary node
``node1`` and secondary node ``node3``, use the following command::
- gnt-instance add -n node1:node3 -o debootstrap -t drbd \
+ $ gnt-instance add -n node1:node3 -o debootstrap -t drbd -s 50G \
instance1
There is a also a command for batch instance creation from a
irreversible and destroys all the contents of your instance. Use with
care::
- gnt-instance remove INSTANCE_NAME
+ $ gnt-instance remove %INSTANCE_NAME%
+
+.. _instance-startup-label:
Startup/shutdown
~~~~~~~~~~~~~~~~
Instances are automatically started at instance creation time. To
manually start one which is currently stopped you can run::
- gnt-instance startup INSTANCE_NAME
+ $ gnt-instance startup %INSTANCE_NAME%
+
+Ganeti will start an instance with up to its maximum instance memory. If
+not enough memory is available Ganeti will use all the available memory
+down to the instance minimum memory. If not even that amount of memory
+is free Ganeti will refuse to start the instance.
+
+Note, that this will not work when an instance is in a permanently
+stopped state ``offline``. In this case, you will first have to
+put it back to online mode by running::
+
+ $ gnt-instance modify --online %INSTANCE_NAME%
-While the command to stop one is::
+The command to stop the running instance is::
- gnt-instance shutdown INSTANCE_NAME
+ $ gnt-instance shutdown %INSTANCE_NAME%
+
+If you want to shut the instance down more permanently, so that it
+does not require dynamically allocated resources (memory and vcpus),
+after shutting down an instance, execute the following::
+
+ $ gnt-instance modify --offline %INSTANCE_NAME%
.. warning:: Do not use the Xen or KVM commands directly to stop
instances. If you run for example ``xm shutdown`` or ``xm destroy``
- on an instance Ganeti will automatically restart it (via the
- :command:`ganeti-watcher` command which is launched via cron).
+ on an instance Ganeti will automatically restart it (via
+ the :command:`ganeti-watcher(8)` command which is launched via cron).
Querying instances
~~~~~~~~~~~~~~~~~~
The command to see all the instances configured and their status is::
- gnt-instance list
+ $ gnt-instance list
The command can return a custom set of information when using the ``-o``
option (as always, check the manpage for a detailed specification). Each
To get more detailed information about an instance, you can run::
- gnt-instance info INSTANCE
+ $ gnt-instance info %INSTANCE%
which will give a multi-line block of information about the instance,
it's hardware resources (especially its disks and their redundancy
status), etc. This is harder to parse and is more expensive than the
list operation, but returns much more detailed information.
+Changing an instance's runtime memory
++++++++++++++++++++++++++++++++++++++
+
+Ganeti will always make sure an instance has a value between its maximum
+and its minimum memory available as runtime memory. As of version 2.6
+Ganeti will only choose a size different than the maximum size when
+starting up, failing over, or migrating an instance on a node with less
+than the maximum memory available. It won't resize other instances in
+order to free up space for an instance.
+
+If you find that you need more memory on a node any instance can be
+manually resized without downtime, with the command::
+
+ $ gnt-instance modify -m %SIZE% %INSTANCE_NAME%
+
+The same command can also be used to increase the memory available on an
+instance, provided that enough free memory is available on its node, and
+the specified size is not larger than the maximum memory size the
+instance had when it was first booted (an instance will be unable to see
+new memory above the maximum that was specified to the hypervisor at its
+boot time, if it needs to grow further a reboot becomes necessary).
Export/Import
+++++++++++++
configuration, which then you can backup, or import into another
cluster. The way to export an instance is::
- gnt-backup export -n TARGET_NODE INSTANCE_NAME
+ $ gnt-backup export -n %TARGET_NODE% %INSTANCE_NAME%
The target node can be any node in the cluster with enough space under
Importing an instance is similar to creating a new one, but additionally
one must specify the location of the snapshot. The command is::
- gnt-backup import -n TARGET_NODE \
- --src-node=NODE --src-dir=DIR INSTANCE_NAME
+ $ gnt-backup import -n %TARGET_NODE% \
+ --src-node=%NODE% --src-dir=%DIR% %INSTANCE_NAME%
By default, parameters will be read from the export information, but you
can of course pass them in via the command line - most of the options
then create a Ganeti instance in the usual way, except that instead of
passing the disk information you specify the current volumes::
- gnt-instance add -t plain -n HOME_NODE ... \
- --disk 0:adopt=lv_name[,vg=vg_name] INSTANCE_NAME
+ $ gnt-instance add -t plain -n %HOME_NODE% ... \
+ --disk 0:adopt=%lv_name%[,vg=%vg_name%] %INSTANCE_NAME%
This will take over the given logical volumes, rename them to the Ganeti
standard (UUID-based), and without installing the OS on them start
be seamless for the instance. For more than one disk, just pass another
disk parameter (e.g. ``--disk 1:adopt=...``).
+Instance kernel selection
++++++++++++++++++++++++++
+
+The kernel that instances uses to bootup can come either from the node,
+or from instances themselves, depending on the setup.
+
+Xen-PVM
+~~~~~~~
+
+With Xen PVM, there are three options.
+
+First, you can use a kernel from the node, by setting the hypervisor
+parameters as such:
+
+- ``kernel_path`` to a valid file on the node (and appropriately
+ ``initrd_path``)
+- ``kernel_args`` optionally set to a valid Linux setting (e.g. ``ro``)
+- ``root_path`` to a valid setting (e.g. ``/dev/xvda1``)
+- ``bootloader_path`` and ``bootloader_args`` to empty
+
+Alternatively, you can delegate the kernel management to instances, and
+use either ``pvgrub`` or the deprecated ``pygrub``. For this, you must
+install the kernels and initrds in the instance and create a valid GRUB
+v1 configuration file.
+
+For ``pvgrub`` (new in version 2.4.2), you need to set:
+
+- ``kernel_path`` to point to the ``pvgrub`` loader present on the node
+ (e.g. ``/usr/lib/xen/boot/pv-grub-x86_32.gz``)
+- ``kernel_args`` to the path to the GRUB config file, relative to the
+ instance (e.g. ``(hd0,0)/grub/menu.lst``)
+- ``root_path`` **must** be empty
+- ``bootloader_path`` and ``bootloader_args`` to empty
+
+While ``pygrub`` is deprecated, here is how you can configure it:
+
+- ``bootloader_path`` to the pygrub binary (e.g. ``/usr/bin/pygrub``)
+- the other settings are not important
+
+More information can be found in the Xen wiki pages for `pvgrub
+<http://wiki.xensource.com/xenwiki/PvGrub>`_ and `pygrub
+<http://wiki.xensource.com/xenwiki/PyGrub>`_.
+
+KVM
+~~~
+
+For KVM also the kernel can be loaded either way.
+
+For loading the kernels from the node, you need to set:
+
+- ``kernel_path`` to a valid value
+- ``initrd_path`` optionally set if you use an initrd
+- ``kernel_args`` optionally set to a valid value (e.g. ``ro``)
+
+If you want instead to have the instance boot from its disk (and execute
+its bootloader), simply set the ``kernel_path`` parameter to an empty
+string, and all the others will be ignored.
+
Instance HA features
--------------------
failed and it's not up anymore. Doing it is really easy, on the master
node you can just run::
- gnt-instance failover INSTANCE_NAME
+ $ gnt-instance failover %INSTANCE_NAME%
That's it. After the command completes the secondary node is now the
primary, and vice-versa.
+The instance will be started with an amount of memory between its
+``maxmem`` and its ``minmem`` value, depending on the free memory on its
+target node, or the operation will fail if that's not possible. See
+:ref:`instance-startup-label` for details.
+
+If the instance's disk template is of type rbd, then you can specify
+the target node (which can be any node) explicitly, or specify an
+iallocator plugin. If you omit both, the default iallocator will be
+used to determine the target node::
+
+ $ gnt-instance failover -n %TARGET_NODE% %INSTANCE_NAME%
+
Live migrating an instance
~~~~~~~~~~~~~~~~~~~~~~~~~~
If an instance is built in highly available mode, it currently runs and
-both its nodes are running fine, you can at migrate it over to its
+both its nodes are running fine, you can migrate it over to its
secondary node, without downtime. On the master node you need to run::
- gnt-instance migrate INSTANCE_NAME
+ $ gnt-instance migrate %INSTANCE_NAME%
The current load on the instance and its memory size will influence how
long the migration will take. In any case, for both KVM and Xen
hypervisors, the migration will be transparent to the instance.
+If the destination node has less memory than the instance's current
+runtime memory, but at least the instance's minimum memory available
+Ganeti will automatically reduce the instance runtime memory before
+migrating it, unless the ``--no-runtime-changes`` option is passed, in
+which case the target node should have at least the instance's current
+runtime memory free.
+
+If the instance's disk template is of type rbd, then you can specify
+the target node (which can be any node) explicitly, or specify an
+iallocator plugin. If you omit both, the default iallocator will be
+used to determine the target node::
+
+ $ gnt-instance migrate -n %TARGET_NODE% %INSTANCE_NAME%
+
Moving an instance (offline)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If an instance has not been create as mirrored, then the only way to
change its primary node is to execute the move command::
- gnt-instance move -n NEW_NODE INSTANCE
+ $ gnt-instance move -n %NEW_NODE% %INSTANCE%
This has a few prerequisites:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is important to note that for Ganeti to be able to do any disk
-operation, the Linux machines on top of which Ganeti must be consistent;
-for LVM, this means that the LVM commands must not return failures; it
-is common that after a complete disk failure, any LVM command aborts
-with an error similar to::
+operation, the Linux machines on top of which Ganeti runs must be
+consistent; for LVM, this means that the LVM commands must not return
+failures; it is common that after a complete disk failure, any LVM
+command aborts with an error similar to::
- # vgs
+ $ vgs
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
- /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output
- error
+ /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
- Couldn't find device with uuid
- 't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
+ Couldn't find device with uuid 't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'.
Couldn't find all physical volumes for volume group xenvg.
Before restoring an instance's disks to healthy status, it's needed to
#. first, if the disk is completely gone and LVM commands exit with
“Couldn't find device with uuid…” then you need to run the command::
- vgreduce --removemissing VOLUME_GROUP
+ $ vgreduce --removemissing %VOLUME_GROUP%
#. after the above command, the LVM commands should be executing
normally (warnings are normal, but the commands will not fail
#. if the failed disk is still visible in the output of the ``pvs``
command, you need to deactivate it from allocations by running::
- pvs -x n /dev/DISK
+ $ pvs -x n /dev/%DISK%
At this point, the volume group should be consistent and any bad
physical volumes should not longer be available for allocation.
For all three cases, the ``replace-disks`` operation can be used::
# re-create disks on the primary node
- gnt-instance replace-disks -p INSTANCE_NAME
+ $ gnt-instance replace-disks -p %INSTANCE_NAME%
# re-create disks on the current secondary
- gnt-instance replace-disks -s INSTANCE_NAME
+ $ gnt-instance replace-disks -s %INSTANCE_NAME%
# change the secondary node, via manual specification
- gnt-instance replace-disks -n NODE INSTANCE_NAME
+ $ gnt-instance replace-disks -n %NODE% %INSTANCE_NAME%
# change the secondary node, via an iallocator script
- gnt-instance replace-disks -I SCRIPT INSTANCE_NAME
+ $ gnt-instance replace-disks -I %SCRIPT% %INSTANCE_NAME%
# since Ganeti 2.1: automatically fix the primary or secondary node
- gnt-instance replace-disks -a INSTANCE_NAME
+ $ gnt-instance replace-disks -a %INSTANCE_NAME%
Since the process involves copying all data from the working node to the
target node, it will take a while, depending on the instance's disk
-size, node I/O system and network speed. But it is (baring any network
+size, node I/O system and network speed. But it is (barring any network
interruption) completely transparent for the instance.
Re-creating disks for non-redundant instances
disks, after which a reinstall can be run, via the ``recreate-disks``
command::
- gnt-instance recreate-disks INSTANCE
+ $ gnt-instance recreate-disks %INSTANCE%
-Note that this will fail if the disks already exists.
+Note that this will fail if the disks already exists. The instance can
+be assigned to new nodes automatically by specifying an iallocator
+through the ``--iallocator`` option.
Conversion of an instance's disk type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
modify`` command::
# start with a non-redundant instance
- gnt-instance add -t plain ... INSTANCE
+ $ gnt-instance add -t plain ... %INSTANCE%
# later convert it to redundant
- gnt-instance stop INSTANCE
- gnt-instance modify -t drbd -n NEW_SECONDARY INSTANCE
- gnt-instance start INSTANCE
+ $ gnt-instance stop %INSTANCE%
+ $ gnt-instance modify -t drbd -n %NEW_SECONDARY% %INSTANCE%
+ $ gnt-instance start %INSTANCE%
# and convert it back
- gnt-instance stop INSTANCE
- gnt-instance modify -t plain INSTANCE
- gnt-instance start INSTANCE
+ $ gnt-instance stop %INSTANCE%
+ $ gnt-instance modify -t plain %INSTANCE%
+ $ gnt-instance start %INSTANCE%
The conversion must be done while the instance is stopped, and
converting from plain to drbd template presents a small risk, especially
inconsistent. The correct way to access an instance's disks is to run
(on the master node, as usual) the command::
- gnt-instance activate-disks INSTANCE
+ $ gnt-instance activate-disks %INSTANCE%
And then, *on the primary node of the instance*, access the device that
gets created. For example, you could mount the given disks, then edit
Note that with partitioned disks (as opposed to whole-disk filesystems),
you will need to use a tool like :manpage:`kpartx(8)`::
- node1# gnt-instance activate-disks instance1
- …
- node1# ssh node3
- node3# kpartx -l /dev/…
- node3# kpartx -a /dev/…
- node3# mount /dev/mapper/… /mnt/
+ # on node1
+ $ gnt-instance activate-disks %instance1%
+ node3:disk/0:…
+ $ ssh node3
+ # on node 3
+ $ kpartx -l /dev/…
+ $ kpartx -a /dev/…
+ $ mount /dev/mapper/… /mnt/
# edit files under mnt as desired
- node3# umount /mnt/
- node3# kpartx -d /dev/…
- node3# exit
- node1#
+ $ umount /mnt/
+ $ kpartx -d /dev/…
+ $ exit
+ # back to node 1
After you've finished you can deactivate them with the deactivate-disks
command, which works in the same way::
- gnt-instance deactivate-disks INSTANCE
+ $ gnt-instance deactivate-disks %INSTANCE%
Note that if any process started by you is still using the disks, the
above command will error out, and you **must** cleanup and ensure that
The command to access a running instance's console is::
- gnt-instance console INSTANCE_NAME
+ $ gnt-instance console %INSTANCE_NAME%
Use the console normally and then type ``^]`` when done, to exit.
There is a wrapper command for rebooting instances::
- gnt-instance reboot instance2
+ $ gnt-instance reboot %instance2%
By default, this does the equivalent of shutting down and then starting
the instance, but it accepts parameters to perform a soft-reboot (via
Should you have any problems with instance operating systems the command
to see a complete status for all your nodes is::
- gnt-os diagnose
+ $ gnt-os diagnose
.. _instance-relocation-label:
steps::
# instance is located on A, B
- node1# gnt-instance replace -n nodeC instance1
+ $ gnt-instance replace-disks -n %nodeC% %instance1%
# instance has moved from (A, B) to (A, C)
# we now flip the primary/secondary nodes
- node1# gnt-instance migrate instance1
+ $ gnt-instance migrate %instance1%
# instance lives on (C, A)
# we can then change A to D via:
- node1# gnt-instance replace -n nodeD instance1
+ $ gnt-instance replace-disks -n %nodeD% %instance1%
Which brings it into the final configuration of ``(C, D)``. Note that we
needed to do two replace-disks operation (two copies of the instance
disks), because we needed to get rid of both the original nodes (A and
B).
+Network Management
+------------------
+
+Ganeti used to describe NICs of an Instance with an IP, a MAC, a connectivity
+link and mode. This had three major shortcomings:
+
+ * there was no easy way to assign a unique IP to an instance
+ * network info (subnet, gateway, domain, etc.) was not available on target
+ node (kvm-ifup, hooks, etc)
+ * one should explicitly pass L2 info (mode, and link) to every NIC
+
+Plus there was no easy way to get the current networking overview (which
+instances are on the same L2 or L3 network, which IPs are reserved, etc).
+
+All the above required an external management tool that has an overall view
+and provides the corresponding info to Ganeti.
+
+gnt-network aims to support a big part of this functionality inside Ganeti and
+abstract the network as a separate entity. Currently, a Ganeti network
+provides the following:
+
+ * A single IPv4 pool, subnet and gateway
+ * Connectivity info per nodegroup (mode, link)
+ * MAC prefix for each NIC inside the network
+ * IPv6 prefix/Gateway related to this network
+ * Tags
+
+IP pool management ensures IP uniqueness inside this network. The user can
+pass `ip=pool,network=test` and will:
+
+1. Get the first available IP in the pool
+2. Inherit the connectivity mode and link of the network's netparams
+3. NIC will obtain the MAC prefix of the network
+4. All network related info will be available as environment variables in
+ kvm-ifup scripts and hooks, so that they can dynamically manage all
+ networking-related setup on the host.
+
+Hands on with gnt-network
++++++++++++++++++++++++++
+
+To create a network do::
+
+ # gnt-network add --network=192.0.2.0/24 --gateway=192.0.2.1 test
+
+Please see all other available options (--add-reserved-ips, --mac-prefix,
+--network6, --subnet6, --tags).
+
+To make this network available on a nodegroup you should specify the
+connectivity mode and link during connection::
+
+ # gnt-network connect test bridged br100 default nodegroup1
+
+To add a NIC inside this network::
+
+ # gnt-instance modify --net -1:add,ip=pool,network=test inst1
+
+This will let a NIC obtain a unique IP inside this network, and inherit the
+nodegroup's netparams (bridged, br100). IP here is optional. If missing the
+NIC will just get the L2 info.
+
+To move an existing NIC from a network to another and remove its IP::
+
+ # gnt-instance modify --net -1:ip=none,network=test1 inst1
+
+This will release the old IP from the old IP pool and the NIC will inherit the
+new nicparams.
+
+On the above actions there is a extra option `--no-conflicts-ckeck`. This
+does not check for conflicting setups. Specifically:
+
+1. When a network is added, IPs of nodes and master are not being checked.
+2. When connecting a network on a nodegroup, IPs of instances inside this
+ nodegroup are not checked whether they reside inside the subnet or not.
+3. When specifying explicitly a IP without passing a network, Ganeti will not
+ check if this IP is included inside any available network on the nodegroup.
+
+External components
++++++++++++++++++++
+
+All the aforementioned steps assure NIC configuration from the Ganeti
+perspective. Of course this has nothing to do, how the instance eventually will
+get the desired connectivity (IPv4, IPv6, default routes, DNS info, etc) and
+where will the IP resolve. This functionality is managed by the external
+components.
+
+Let's assume that the VM will need to obtain a dynamic IP via DHCP, get a SLAAC
+address, and use DHCPv6 for other configuration information (in case RFC-6106
+is not supported by the client, e.g. Windows). This means that the following
+external services are needed:
+
+1. A DHCP server
+2. An IPv6 router sending Router Advertisements
+3. A DHCPv6 server exporting DNS info
+4. A dynamic DNS server
+
+These components must be configured dynamically and on a per NIC basis.
+The way to do this is by using custom kvm-ifup scripts and hooks.
+
+snf-network
+~~~~~~~~~~~
+
+The snf-network package [1,3] includes custom scripts that will provide the
+aforementioned functionality. `kvm-vif-bridge` and `vif-custom` is an
+alternative to `kvm-ifup` and `vif-ganeti` that take into account all network
+info being exported. Their actions depend on network tags. Specifically:
+
+`dns`: will update an external DDNS server (nsupdate on a bind server)
+
+`ip-less-routed`: will setup routes, rules and proxy ARP
+This setup assumes a pre-existing routing table along with some local
+configuration and provides connectivity to instances via an external
+gateway/router without requiring nodes to have an IP inside this network.
+
+`private-filtered`: will setup ebtables rules to ensure L2 isolation on a
+common bridge. Only packets with the same MAC prefix will be forwarded to the
+corresponding virtual interface.
+
+`nfdhcpd`: will update an external DHCP server
+
+nfdhcpd
+~~~~~~~
+
+snf-network works with nfdhcpd [2,3]: a custom user space DHCP
+server based on NFQUEUE. Currently, nfdhcpd replies on BOOTP/DHCP requests
+originating from a tap or a bridge. Additionally in case of a routed setup it
+provides a ra-stateless configuration by responding to router and neighbour
+solicitations along with DHCPv6 requests for DNS options. Its db is
+dynamically updated using text files inside a local dir with inotify
+(snf-network just adds a per NIC binding file with all relevant info if the
+corresponding network tag is found). Still we need to mangle all these
+packets and send them to the corresponding NFQUEUE.
+
+Known shortcomings
+++++++++++++++++++
+
+Currently the following things are some know weak points of the gnt-network
+design and implementation:
+
+ * Cannot define a network without an IP pool
+ * The pool defines the size of the network
+ * Reserved IPs must be defined explicitly (inconvenient for a big range)
+ * Cannot define an IPv6 only network
+
+Future work
++++++++++++
+
+Any upcoming patches should target:
+
+ * Separate L2, L3, IPv6, IP pool info
+ * Support a set of IP pools per network
+ * Make IP/network in NIC object take a list of entries
+ * Introduce external scripts for node configuration
+ (dynamically create/destroy bridges/routes upon network connect/disconnect)
+
+[1] https://code.grnet.gr/git/snf-network
+[2] https://code.grnet.gr/git/snf-nfdhcpd
+[3] deb http:/apt.dev.grnet.gr/ wheezy/
+
Node operations
---------------
It is at any time possible to extend the cluster with one more node, by
using the node add operation::
- gnt-node add NEW_NODE
+ $ gnt-node add %NEW_NODE%
If the cluster has a replication network defined, then you need to pass
the ``-s REPLICATION_IP`` parameter to this option.
Ganeti configuration is broken, for example if it has been reinstalled
by mistake::
- gnt-node add --readd EXISTING_NODE
+ $ gnt-node add --readd %EXISTING_NODE%
This will reinitialise the node as if it's been newly added, but while
keeping its existing configuration in the cluster (primary/secondary IP,
If you want to promote a different node to the master role (for whatever
reason), run on any other master-candidate node the command::
- gnt-cluster master-failover
+ $ gnt-cluster master-failover
and the node you ran it on is now the new master. In case you try to run
this on a non master-candidate node, you will get an error telling you
The ``gnt-node modify`` command can be used to select a new role::
# change to master candidate
- gnt-node modify -C yes NODE
+ $ gnt-node modify -C yes %NODE%
# change to drained status
- gnt-node modify -D yes NODE
+ $ gnt-node modify -D yes %NODE%
# change to offline status
- gnt-node modify -O yes NODE
+ $ gnt-node modify -O yes %NODE%
# change to regular mode (reset all flags)
- gnt-node modify -O no -D no -C no NODE
+ $ gnt-node modify -O no -D no -C no %NODE%
Note that the cluster requires that at any point in time, a certain
number of nodes are master candidates, so changing from master candidate
commands (as seen in :ref:`instance-change-primary-label`) or the bulk
per-node versions; these are::
- gnt-node migrate NODE
- gnt-node evacuate NODE
+ $ gnt-node migrate %NODE%
+ $ gnt-node evacuate -s %NODE%
Note that the instance “move” command doesn't currently have a node
equivalent.
For the evacuation of secondary instances, a command called
:command:`gnt-node evacuate` is provided and its syntax is::
- gnt-node evacuate -I IALLOCATOR_SCRIPT NODE
- gnt-node evacuate -n DESTINATION_NODE NODE
+ $ gnt-node evacuate -I %IALLOCATOR_SCRIPT% %NODE%
+ $ gnt-node evacuate -n %DESTINATION_NODE% %NODE%
The first version will compute the new secondary for each instance in
turn using the given iallocator script, whereas the second one will
Once a node no longer has any instances (neither primary nor secondary),
it's easy to remove it from the cluster::
- gnt-node remove NODE_NAME
+ $ gnt-node remove %NODE_NAME%
This will deconfigure the node, stop the ganeti daemons on it and leave
it hopefully like before it joined to the cluster.
+Replication network changes
++++++++++++++++++++++++++++
+
+The :command:`gnt-node modify -s` command can be used to change the
+secondary IP of a node. This operation can only be performed if:
+
+- No instance is active on the target node
+- The new target IP is reachable from the master's secondary IP
+
+Also this operation will not allow to change a node from single-homed
+(same primary and secondary ip) to multi-homed (separate replication
+network) or vice versa, unless:
+
+- The target node is the master node and `--force` is passed.
+- The target cluster is single-homed and the new primary ip is a change
+ to single homed for a particular node.
+- The target cluster is multi-homed and the new primary ip is a change
+ to multi homed for a particular node.
+
+For example to do a single-homed to multi-homed conversion::
+
+ $ gnt-node modify --force -s %SECONDARY_IP% %MASTER_NAME%
+ $ gnt-node modify -s %SECONDARY_IP% %NODE1_NAME%
+ $ gnt-node modify -s %SECONDARY_IP% %NODE2_NAME%
+ $ gnt-node modify -s %SECONDARY_IP% %NODE3_NAME%
+ ...
+
+The same commands can be used for multi-homed to single-homed except the
+secondary IPs should be the same as the primaries for each node, for
+that case.
+
Storage handling
++++++++++++++++
logical volumes on a given node or on all nodes and their association to
instances via the ``volumes`` command::
- node1# gnt-node volumes
+ $ gnt-node volumes
Node PhysDev VG Name Size Instance
node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17
node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19
First is listing the backend storage and their space situation::
- node1# gnt-node list-storage
+ $ gnt-node list-storage
Node Name Size Used Free
node1 /dev/sda7 673.8G 0M 673.8G
node1 /dev/sdb1 698.6G 1.5G 697.1G
The default is to list LVM physical volumes. It's also possible to list
the LVM volume groups::
- node1# gnt-node list-storage -t lvm-vg
+ $ gnt-node list-storage -t lvm-vg
Node Name Size
node1 xenvg 1.3T
node2 xenvg 1.3T
Next is repairing storage units, which is currently only implemented for
volume groups and does the equivalent of ``vgreduce --removemissing``::
- node1# gnt-node repair-storage node2 lvm-vg xenvg
+ $ gnt-node repair-storage %node2% lvm-vg xenvg
Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ...
Last is the modification of volume properties, which is (again) only
implemented for LVM physical volumes and allows toggling the
``allocatable`` value::
- node1# gnt-node modify-storage --allocatable=no node2 lvm-pv /dev/sdb1
+ $ gnt-node modify-storage --allocatable=no %node2% lvm-pv /dev/%sdb1%
Use of the storage commands
~~~~~~~~~~~~~~~~~~~~~~~~~~~
One of the few commands that can be run on any node (not only the
master) is the ``getmaster`` command::
- node2# gnt-cluster getmaster
+ # on node2
+ $ gnt-cluster getmaster
node1.example.com
- node2#
It is possible to query and change global cluster parameters via the
``info`` and ``modify`` commands::
- node1# gnt-cluster info
+ $ gnt-cluster info
Cluster name: cluster.example.com
Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c
Creation time: 2009-09-25 05:04:15
For detailed option list see the :manpage:`gnt-cluster(8)` man page.
The cluster version can be obtained via the ``version`` command::
- node1# gnt-cluster version
+ $ gnt-cluster version
Software version: 2.1.0
Internode protocol: 20
Configuration format: 2010000
There are two commands provided for replicating files to all nodes of a
cluster and for running commands on all the nodes::
- node1# gnt-cluster copyfile /path/to/file
- node1# gnt-cluster command ls -l /path/to/file
+ $ gnt-cluster copyfile %/path/to/file%
+ $ gnt-cluster command %ls -l /path/to/file%
These are simple wrappers over scp/ssh and more advanced usage can be
obtained using :manpage:`dsh(1)` and similar commands. But they are
highlighting any issues. In normal operation, this command should return
no ``ERROR`` messages::
- node1# gnt-cluster verify
+ $ gnt-cluster verify
Sun Oct 25 23:08:58 2009 * Verifying global settings
Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes)
Sun Oct 25 23:09:00 2009 * Verifying node status
disks have the correct status based on the desired instance state
(up/down)::
- node1# gnt-cluster verify-disks
+ $ gnt-cluster verify-disks
Note that this command will show no output when disks are healthy.
recorded disk size and the actual disk size (disk size information is
needed for proper activation and growth of DRBD-based disks)::
- node1# gnt-cluster repair-disk-sizes
+ $ gnt-cluster repair-disk-sizes
Sun Oct 25 23:13:16 2009 - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048
Sun Oct 25 23:13:17 2009 - WARNING: Invalid result from node node4, ignoring node results
configuration files, you can force an push of the master configuration
to all other nodes via the ``redist-conf`` command::
- node1# gnt-cluster redist-conf
- node1#
+ $ gnt-cluster redist-conf
This command will be silent unless there are problems sending updates to
the other nodes.
``rename`` command. If only the IP has changed, you need to pass the
current name and Ganeti will realise its IP has changed::
- node1# gnt-cluster rename cluster.example.com
+ $ gnt-cluster rename %cluster.example.com%
This will rename the cluster to 'cluster.example.com'. If
you are connected over the network to the cluster name, the operation
is very dangerous as the IP address will be removed from the node and
the change may not go through. Continue?
- y/[n]/?: y
+ y/[n]/?: %y%
Failure: prerequisites not met for this operation:
Neither the name nor the IP address of the cluster has changed
The job queue execution in Ganeti 2.0 and higher can be inspected,
suspended and resumed via the ``queue`` command::
- node1~# gnt-cluster queue info
+ $ gnt-cluster queue info
The drain flag is unset
- node1~# gnt-cluster queue drain
- node1~# gnt-instance stop instance1
+ $ gnt-cluster queue drain
+ $ gnt-instance stop %instance1%
Failed to submit job for instance1: Job queue is drained, refusing job
- node1~# gnt-cluster queue info
+ $ gnt-cluster queue info
The drain flag is set
- node1~# gnt-cluster queue undrain
+ $ gnt-cluster queue undrain
This is most useful if you have an active cluster and you need to
upgrade the Ganeti software, or simply restart the software on any node:
Watcher control
+++++++++++++++
-The :manpage:`ganeti-watcher` is a program, usually scheduled via
+The :manpage:`ganeti-watcher(8)` is a program, usually scheduled via
``cron``, that takes care of cluster maintenance operations (restarting
downed instances, activating down DRBD disks, etc.). However, during
maintenance and troubleshooting, this can get in your way; disabling it
forgotten. Thus there are some commands for automated control of the
watcher: ``pause``, ``info`` and ``continue``::
- node1~# gnt-cluster watcher info
+ $ gnt-cluster watcher info
The watcher is not paused.
- node1~# gnt-cluster watcher pause 1h
+ $ gnt-cluster watcher pause %1h%
The watcher is paused until Mon Oct 26 00:30:37 2009.
- node1~# gnt-cluster watcher info
+ $ gnt-cluster watcher info
The watcher is paused until Mon Oct 26 00:30:37 2009.
- node1~# ganeti-watcher -d
+ $ ganeti-watcher -d
2009-10-25 23:30:47,984: pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting
- node1~# gnt-cluster watcher continue
+ $ gnt-cluster watcher continue
The watcher is no longer paused.
- node1~# ganeti-watcher -d
+ $ ganeti-watcher -d
2009-10-25 23:31:04,789: pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0
2009-10-25 23:31:05,884: pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file
2009-10-25 23:31:06,061: pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file
- node1~# gnt-cluster watcher info
+ $ gnt-cluster watcher info
The watcher is not paused.
- node1~#
The exact details of the argument to the ``pause`` command are available
in the manpage.
6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``),
replacing the path with the correct path for your installation.
+7. If using RBD, run ``rbd unmap /dev/rbdN`` to unmap the RBD disks.
+ Then remove the RBD disk images used by Ganeti, identified by their
+ UUIDs (``rbd rm uuid.rbd.diskN``).
+
On the master node, remove the cluster from the master-netdev (usually
``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running
``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and
Otherwise, if you plan to re-create the cluster, you can just go ahead
and rerun ``gnt-cluster init``.
+Replacing the SSH and SSL keys
+++++++++++++++++++++++++++++++
+
+Ganeti uses both SSL and SSH keys, and actively modifies the SSH keys on
+the nodes. As result, in order to replace these keys, a few extra steps
+need to be followed: :doc:`cluster-keys-replacement`
+
+Monitoring the cluster
+----------------------
+
+Starting with Ganeti 2.8, a monitoring daemon is available, providing
+information about the status and the performance of the system.
+
+The monitoring daemon runs on every node, listening on TCP port 1815. Each
+instance of the daemon provides information related to the node it is running
+on.
+
+.. include:: monitoring-query-format.rst
+
Tags handling
-------------
Tags can be added via ``add-tags``::
- gnt-instance add-tags INSTANCE a b c
- gnt-node add-tags INSTANCE a b c
- gnt-cluster add-tags a b c
+ $ gnt-instance add-tags %INSTANCE% %a% %b% %c%
+ $ gnt-node add-tags %INSTANCE% %a% %b% %c%
+ $ gnt-cluster add-tags %a% %b% %c%
The above commands add three tags to an instance, to a node and to the
Tags can also be remove via a syntax very similar to the add one::
- gnt-instance remove-tags INSTANCE a b c
+ $ gnt-instance remove-tags %INSTANCE% %a% %b% %c%
And listed via::
- gnt-instance list-tags
- gnt-node list-tags
- gnt-cluster list-tags
+ $ gnt-instance list-tags
+ $ gnt-node list-tags
+ $ gnt-cluster list-tags
Global tag search
+++++++++++++++++
It is also possible to execute a global search on the all tags defined
in the cluster configuration, via a cluster command::
- gnt-cluster search-tags REGEXP
+ $ gnt-cluster search-tags %REGEXP%
The parameter expected is a regular expression (see
:manpage:`regex(7)`). This will return all tags that match the search,
together with the object they are defined in (the names being show in a
hierarchical kind of way)::
- node1# gnt-cluster search-tags o
+ $ gnt-cluster search-tags %o%
/cluster foo
/instances/instance1 owner:bar
+Autorepair
+----------
+
+The tool ``harep`` can be used to automatically fix some problems that are
+present in the cluster.
+
+It is mainly meant to be regularly and automatically executed
+as a cron job. This is quite evident by considering that, when executed, it does
+not immediately fix all the issues of the instances of the cluster, but it
+cycles the instances through a series of states, one at every ``harep``
+execution. Every state performs a step towards the resolution of the problem.
+This process goes on until the instance is brought back to the healthy state,
+or the tool realizes that it is not able to fix the instance, and
+therefore marks it as in failure state.
+
+Allowing harep to act on the cluster
+++++++++++++++++++++++++++++++++++++
+
+By default, ``harep`` checks the status of the cluster but it is not allowed to
+perform any modification. Modification must be explicitly allowed by an
+appropriate use of tags. Tagging can be applied at various levels, and can
+enable different kinds of autorepair, as hereafter described.
+
+All the tags that authorize ``harep`` to perform modifications follow this
+syntax::
+
+ ganeti:watcher:autorepair:<type>
+
+where ``<type>`` indicates the kind of intervention that can be performed. Every
+possible value of ``<type>`` includes at least all the authorization of the
+previous one, plus its own. The possible values, in increasing order of
+severity, are:
+
+- ``fix-storage`` allows a disk replacement or another operation that
+ fixes the instance backend storage without affecting the instance
+ itself. This can for example recover from a broken drbd secondary, but
+ risks data loss if something is wrong on the primary but the secondary
+ was somehow recoverable.
+- ``migrate`` allows an instance migration. This can recover from a
+ drained primary, but can cause an instance crash in some cases (bugs).
+- ``failover`` allows instance reboot on the secondary. This can recover
+ from an offline primary, but the instance will lose its running state.
+- ``reinstall`` allows disks to be recreated and an instance to be
+ reinstalled. This can recover from primary&secondary both being
+ offline, or from an offline primary in the case of non-redundant
+ instances. It causes data loss.
+
+These autorepair tags can be applied to a cluster, a nodegroup or an instance,
+and will act where they are applied and to everything in the entities sub-tree
+(e.g. a tag applied to a nodegroup will apply to all the instances contained in
+that nodegroup, but not to the rest of the cluster).
+
+If there are multiple ``ganeti:watcher:autorepair:<type>`` tags in an
+object (cluster, node group or instance), the least destructive tag
+takes precedence. When multiplicity happens across objects, the nearest
+tag wins. For example, if in a cluster with two instances, *I1* and
+*I2*, *I1* has ``failover``, and the cluster itself has both
+``fix-storage`` and ``reinstall``, *I1* will end up with ``failover``
+and *I2* with ``fix-storage``.
+
+Limiting harep
+++++++++++++++
+
+Sometimes it is useful to stop harep from performing its task temporarily,
+and it is useful to be able to do so without distrupting its configuration, that
+is, without removing the authorization tags. In order to do this, suspend tags
+are provided.
+
+Suspend tags can be added to cluster, nodegroup or instances, and act on the
+entire entities sub-tree. No operation will be performed by ``harep`` on the
+instances protected by a suspend tag. Their syntax is as follows::
+
+ ganeti:watcher:autorepair:suspend[:<timestamp>]
+
+If there are multiple suspend tags in an object, the form without timestamp
+takes precedence (permanent suspension); or, if all object tags have a
+timestamp, the one with the highest timestamp.
+
+Tags with a timestamp will be automatically removed when the time indicated by
+the timestamp is passed. Indefinite suspension tags have to be removed manually.
+
+Result reporting
+++++++++++++++++
+
+Harep will report about the result of its actions both through its CLI, and by
+adding tags to the instances it operated on. Such tags will follow the syntax
+hereby described::
+
+ ganeti:watcher:autorepair:result:<type>:<id>:<timestamp>:<result>:<jobs>
+
+If this tag is present a repair of type ``type`` has been performed on
+the instance and has been completed by ``timestamp``. The result is
+either ``success``, ``failure`` or ``enoperm``, and jobs is a
+*+*-separated list of jobs that were executed for this repair.
+
+An ``enoperm`` result is an error state due to permission problems. It
+is returned when the repair cannot proceed because it would require to perform
+an operation that is not allowed by the ``ganeti:watcher:autorepair:<type>`` tag
+that is defining the instance autorepair permissions.
+
+NB: if an instance repair ends up in a failure state, it will not be touched
+again by ``harep`` until it has been manually fixed by the system administrator
+and the ``ganeti:watcher:autorepair:result:failure:*`` tag has been manually
+removed.
Job operations
--------------
First is the job list command::
- node1# gnt-job list
+ $ gnt-job list
17771 success INSTANCE_QUERY_DATA
17773 success CLUSTER_VERIFY_DISKS
17775 success CLUSTER_REPAIR_DISK_SIZES
More detailed information about a job can be found via the ``info``
command::
- node1# gnt-job info 17776
+ $ gnt-job info %17776%
Job ID: 17776
Status: error
Received: 2009-10-25 23:18:02.180569
job, similar to the log that one get from the ``gnt-`` commands, via the
watch command::
- node1# gnt-instance add --submit … instance1
+ $ gnt-instance add --submit … %instance1%
JobID: 17818
- node1# gnt-job watch 17818
+ $ gnt-job watch %17818%
Output from job 17818 follows
-----------------------------
Mon Oct 26 00:22:48 2009 - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2
Mon Oct 26 00:23:03 2009 creating os for instance instance1 on node node1
Mon Oct 26 00:23:03 2009 * running the instance OS create scripts...
Mon Oct 26 00:23:13 2009 * starting instance...
- node1#
+ $
This is useful if you need to follow a job's progress from multiple
terminals.
A job that has not yet started to run can be canceled::
- node1# gnt-job cancel 17810
+ $ gnt-job cancel %17810%
But not one that has already started execution::
- node1# gnt-job cancel 17805
+ $ gnt-job cancel %17805%
Job 17805 is no longer waiting in the queue
There are two queues for jobs: the *current* and the *archive*
queue. Jobs are initially submitted to the current queue, and they stay
in that queue until they have finished execution (either successfully or
-not). At that point, they can be moved into the archive queue, and the
-ganeti-watcher script will do this automatically after 6 hours. The
-ganeti-cleaner script will remove the jobs from the archive directory
+not). At that point, they can be moved into the archive queue using e.g.
+``gnt-job autoarchive all``. The ``ganeti-watcher`` script will do this
+automatically 6 hours after a job is finished. The ``ganeti-cleaner``
+script will then remove archived the jobs from the archive directory
after three weeks.
-Note that only jobs in the current queue can be viewed via the list and
-info commands; Ganeti itself doesn't examine the archive directory. If
-you need to see an older job, either move the file manually in the
-top-level queue directory, or look at its contents (it's a
-JSON-formatted file).
+Note that ``gnt-job list`` only shows jobs in the current queue.
+Archived jobs can be viewed using ``gnt-job info <id>``.
Special Ganeti deployments
--------------------------
strategies, etc. In order to accomplish this, mark these nodes as
non-``vm_capable``::
- node1# gnt-node modify --vm-capable=no node3
+ $ gnt-node modify --vm-capable=no %node3%
The vm_capable status can be listed as usual via ``gnt-node list``::
- node1# gnt-node list -oname,vm_capable
+ $ gnt-node list -oname,vm_capable
Node VMCapable
node1 Y
node2 Y
As usual, the node modify operation can change this flag::
- node1# gnt-node modify --auto-promote --master-capable=no node3
+ $ gnt-node modify --auto-promote --master-capable=no %node3%
Fri Jan 7 06:23:07 2011 - INFO: Demoting from master candidate
Fri Jan 7 06:23:08 2011 - INFO: Promoted nodes to master candidate role: node4
Modified node node3
And the node list operation will list this flag::
- node1# gnt-node list -oname,master_capable node1 node2 node3
+ $ gnt-node list -oname,master_capable %node1% %node2% %node3%
Node MasterCapable
node1 Y
node2 Y
++++++++++
The ``cfgupgrade`` tools is used to upgrade between major (and minor)
-Ganeti versions. Point-releases are usually transparent for the admin.
+Ganeti versions, and to roll back. Point-releases are usually
+transparent for the admin.
More information about the upgrade procedure is listed on the wiki at
http://code.google.com/p/ganeti/wiki/UpgradeNotes.
See :doc:`separate documentation for move-instance <move-instance>`.
+users-setup
++++++++++++
+
+Ganeti can either be run entirely as root, or with every daemon running as
+its own specific user (if the parameters ``--with-user-prefix`` and/or
+``--with-group-prefix`` have been specified at ``./configure``-time).
+
+In case split users are activated, they are required to exist on the system,
+and they need to belong to the proper groups in order for the access
+permissions to files and programs to be correct.
+
+The ``users-setup`` tool, when run, takes care of setting up the proper
+users and groups.
+
+When invoked without parameters, the tool runs in interactive mode, showing the
+list of actions it will perform and asking for confirmation before proceeding.
+
+Providing the ``--yes-do-it`` parameter to the tool prevents the confirmation
+from being asked, and the users and groups will be created immediately.
+
.. TODO: document cluster-merge tool
projects / ganeti-local / blobdiff