Node can be added and removed (if they host no instances) at will from
the cluster. In a HA cluster and only with HA instances, the loss of any
single node will not cause disk data loss for any instance; of course,
-a node crash will cause the crash of the its primary instances.
+a node crash will cause the crash of its primary instances.
A node belonging to a cluster can be in one of the following roles at a
given time:
Depending on the role, each node will run a set of daemons:
-- the :command:`ganeti-noded` daemon, which control the manipulation of
+- the :command:`ganeti-noded` daemon, which controls the manipulation of
this node's hardware resources; it runs on all nodes which are in a
cluster
- the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all
file
The instance will use plain files as backend for its disks. No
redundancy is provided, and this is somewhat more difficult to
- configure for high performance.
+ configure for high performance. Note that for security reasons the
+ file storage directory must be listed under
+ ``/etc/ganeti/file-storage-paths``, and that file is not copied
+ automatically to all nodes by Ganeti. The format of that file is a
+ newline-separated list of directories.
+
+sharedfile
+ The instance will use plain files as backend, but Ganeti assumes that
+ those files will be available and in sync automatically on all nodes.
+ This allows live migration and failover of instances using this
+ method. As for ``file`` the file storage directory must be listed under
+ ``/etc/ganeti/file-storage-paths`` or ganeti will refuse to create
+ instances under it.
plain
The instance will use LVM devices as backend for its disks. No
to obtain a highly available instance that can be failed over to a
remote node should the primary one fail.
+ .. note:: Ganeti does not support DRBD stacked devices:
+ DRBD stacked setup is not fully symmetric and as such it is
+ not working with live migration.
+
rbd
The instance will use Volumes inside a RADOS cluster as backend for its
disks. It will access them using the RADOS block device (RBD).
+ext
+ The instance will use an external storage provider. See
+ :manpage:`ganeti-extstorage-interface(7)` for how to implement one.
+
+
IAllocator
~~~~~~~~~~
- Arguments for the NICs of the instance; by default, a single-NIC
instance is created. The IP and/or bridge of the NIC can be changed
- via ``--nic 0:ip=IP,bridge=BRIDGE``
+ via ``--net 0:ip=IP,link=BRIDGE``
-See the manpage for gnt-instance for the detailed option list.
+See :manpage:`ganeti-instance(8)` for the detailed option list.
For example if you want to create an highly available instance, with a
single disk of 50GB and the default memory size, having primary node
Ganeti will start an instance with up to its maximum instance memory. If
not enough memory is available Ganeti will use all the available memory
-down to the instance minumum memory. If not even that amount of memory
+down to the instance minimum memory. If not even that amount of memory
is free Ganeti will refuse to start the instance.
Note, that this will not work when an instance is in a permanently
.. warning:: Do not use the Xen or KVM commands directly to stop
instances. If you run for example ``xm shutdown`` or ``xm destroy``
on an instance Ganeti will automatically restart it (via
- the :command:`ganeti-watcher` command which is launched via cron).
+ the :command:`ganeti-watcher(8)` command which is launched via cron).
Querying instances
~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~
If an instance is built in highly available mode, it currently runs and
-both its nodes are running fine, you can at migrate it over to its
+both its nodes are running fine, you can migrate it over to its
secondary node, without downtime. On the master node you need to run::
$ gnt-instance migrate %INSTANCE_NAME%
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is important to note that for Ganeti to be able to do any disk
-operation, the Linux machines on top of which Ganeti must be consistent;
-for LVM, this means that the LVM commands must not return failures; it
-is common that after a complete disk failure, any LVM command aborts
-with an error similar to::
+operation, the Linux machines on top of which Ganeti runs must be
+consistent; for LVM, this means that the LVM commands must not return
+failures; it is common that after a complete disk failure, any LVM
+command aborts with an error similar to::
$ vgs
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
$ gnt-instance recreate-disks %INSTANCE%
-Note that this will fail if the disks already exists.
+Note that this will fail if the disks already exists. The instance can
+be assigned to new nodes automatically by specifying an iallocator
+through the ``--iallocator`` option.
Conversion of an instance's disk type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
steps::
# instance is located on A, B
- $ gnt-instance replace -n %nodeC% %instance1%
+ $ gnt-instance replace-disks -n %nodeC% %instance1%
# instance has moved from (A, B) to (A, C)
# we now flip the primary/secondary nodes
$ gnt-instance migrate %instance1%
# instance lives on (C, A)
# we can then change A to D via:
- $ gnt-instance replace -n %nodeD% %instance1%
+ $ gnt-instance replace-disks -n %nodeD% %instance1%
Which brings it into the final configuration of ``(C, D)``. Note that we
needed to do two replace-disks operation (two copies of the instance
disks), because we needed to get rid of both the original nodes (A and
B).
+Network Management
+------------------
+
+Ganeti used to describe NICs of an Instance with an IP, a MAC, a connectivity
+link and mode. This had three major shortcomings:
+
+ * there was no easy way to assign a unique IP to an instance
+ * network info (subnet, gateway, domain, etc.) was not available on target
+ node (kvm-ifup, hooks, etc)
+ * one should explicitly pass L2 info (mode, and link) to every NIC
+
+Plus there was no easy way to get the current networking overview (which
+instances are on the same L2 or L3 network, which IPs are reserved, etc).
+
+All the above required an external management tool that has an overall view
+and provides the corresponding info to Ganeti.
+
+gnt-network aims to support a big part of this functionality inside Ganeti and
+abstract the network as a separate entity. Currently, a Ganeti network
+provides the following:
+
+ * A single IPv4 pool, subnet and gateway
+ * Connectivity info per nodegroup (mode, link)
+ * MAC prefix for each NIC inside the network
+ * IPv6 prefix/Gateway related to this network
+ * Tags
+
+IP pool management ensures IP uniqueness inside this network. The user can
+pass `ip=pool,network=test` and will:
+
+1. Get the first available IP in the pool
+2. Inherit the connectivity mode and link of the network's netparams
+3. NIC will obtain the MAC prefix of the network
+4. All network related info will be available as environment variables in
+ kvm-ifup scripts and hooks, so that they can dynamically manage all
+ networking-related setup on the host.
+
+Hands on with gnt-network
++++++++++++++++++++++++++
+
+To create a network do::
+
+ # gnt-network add --network=192.0.2.0/24 --gateway=192.0.2.1 test
+
+Please see all other available options (--add-reserved-ips, --mac-prefix,
+--network6, --subnet6, --tags).
+
+To make this network available on a nodegroup you should specify the
+connectivity mode and link during connection::
+
+ # gnt-network connect test bridged br100 default nodegroup1
+
+To add a NIC inside this network::
+
+ # gnt-instance modify --net -1:add,ip=pool,network=test inst1
+
+This will let a NIC obtain a unique IP inside this network, and inherit the
+nodegroup's netparams (bridged, br100). IP here is optional. If missing the
+NIC will just get the L2 info.
+
+To move an existing NIC from a network to another and remove its IP::
+
+ # gnt-instance modify --net -1:ip=none,network=test1 inst1
+
+This will release the old IP from the old IP pool and the NIC will inherit the
+new nicparams.
+
+On the above actions there is a extra option `--no-conflicts-ckeck`. This
+does not check for conflicting setups. Specifically:
+
+1. When a network is added, IPs of nodes and master are not being checked.
+2. When connecting a network on a nodegroup, IPs of instances inside this
+ nodegroup are not checked whether they reside inside the subnet or not.
+3. When specifying explicitly a IP without passing a network, Ganeti will not
+ check if this IP is included inside any available network on the nodegroup.
+
+External components
++++++++++++++++++++
+
+All the aforementioned steps assure NIC configuration from the Ganeti
+perspective. Of course this has nothing to do, how the instance eventually will
+get the desired connectivity (IPv4, IPv6, default routes, DNS info, etc) and
+where will the IP resolve. This functionality is managed by the external
+components.
+
+Let's assume that the VM will need to obtain a dynamic IP via DHCP, get a SLAAC
+address, and use DHCPv6 for other configuration information (in case RFC-6106
+is not supported by the client, e.g. Windows). This means that the following
+external services are needed:
+
+1. A DHCP server
+2. An IPv6 router sending Router Advertisements
+3. A DHCPv6 server exporting DNS info
+4. A dynamic DNS server
+
+These components must be configured dynamically and on a per NIC basis.
+The way to do this is by using custom kvm-ifup scripts and hooks.
+
+snf-network
+~~~~~~~~~~~
+
+The snf-network package [1,3] includes custom scripts that will provide the
+aforementioned functionality. `kvm-vif-bridge` and `vif-custom` is an
+alternative to `kvm-ifup` and `vif-ganeti` that take into account all network
+info being exported. Their actions depend on network tags. Specifically:
+
+`dns`: will update an external DDNS server (nsupdate on a bind server)
+
+`ip-less-routed`: will setup routes, rules and proxy ARP
+This setup assumes a pre-existing routing table along with some local
+configuration and provides connectivity to instances via an external
+gateway/router without requiring nodes to have an IP inside this network.
+
+`private-filtered`: will setup ebtables rules to ensure L2 isolation on a
+common bridge. Only packets with the same MAC prefix will be forwarded to the
+corresponding virtual interface.
+
+`nfdhcpd`: will update an external DHCP server
+
+nfdhcpd
+~~~~~~~
+
+snf-network works with nfdhcpd [2,3]: a custom user space DHCP
+server based on NFQUEUE. Currently, nfdhcpd replies on BOOTP/DHCP requests
+originating from a tap or a bridge. Additionally in case of a routed setup it
+provides a ra-stateless configuration by responding to router and neighbour
+solicitations along with DHCPv6 requests for DNS options. Its db is
+dynamically updated using text files inside a local dir with inotify
+(snf-network just adds a per NIC binding file with all relevant info if the
+corresponding network tag is found). Still we need to mangle all these
+packets and send them to the corresponding NFQUEUE.
+
+Known shortcomings
+++++++++++++++++++
+
+Currently the following things are some know weak points of the gnt-network
+design and implementation:
+
+ * Cannot define a network without an IP pool
+ * The pool defines the size of the network
+ * Reserved IPs must be defined explicitly (inconvenient for a big range)
+ * Cannot define an IPv6 only network
+
+Future work
++++++++++++
+
+Any upcoming patches should target:
+
+ * Separate L2, L3, IPv6, IP pool info
+ * Support a set of IP pools per network
+ * Make IP/network in NIC object take a list of entries
+ * Introduce external scripts for node configuration
+ (dynamically create/destroy bridges/routes upon network connect/disconnect)
+
+[1] https://code.grnet.gr/git/snf-network
+[2] https://code.grnet.gr/git/snf-nfdhcpd
+[3] deb http:/apt.dev.grnet.gr/ wheezy/
+
Node operations
---------------
This will deconfigure the node, stop the ganeti daemons on it and leave
it hopefully like before it joined to the cluster.
+Replication network changes
++++++++++++++++++++++++++++
+
+The :command:`gnt-node modify -s` command can be used to change the
+secondary IP of a node. This operation can only be performed if:
+
+- No instance is active on the target node
+- The new target IP is reachable from the master's secondary IP
+
+Also this operation will not allow to change a node from single-homed
+(same primary and secondary ip) to multi-homed (separate replication
+network) or vice versa, unless:
+
+- The target node is the master node and `--force` is passed.
+- The target cluster is single-homed and the new primary ip is a change
+ to single homed for a particular node.
+- The target cluster is multi-homed and the new primary ip is a change
+ to multi homed for a particular node.
+
+For example to do a single-homed to multi-homed conversion::
+
+ $ gnt-node modify --force -s %SECONDARY_IP% %MASTER_NAME%
+ $ gnt-node modify -s %SECONDARY_IP% %NODE1_NAME%
+ $ gnt-node modify -s %SECONDARY_IP% %NODE2_NAME%
+ $ gnt-node modify -s %SECONDARY_IP% %NODE3_NAME%
+ ...
+
+The same commands can be used for multi-homed to single-homed except the
+secondary IPs should be the same as the primaries for each node, for
+that case.
+
Storage handling
++++++++++++++++
Watcher control
+++++++++++++++
-The :manpage:`ganeti-watcher` is a program, usually scheduled via
+The :manpage:`ganeti-watcher(8)` is a program, usually scheduled via
``cron``, that takes care of cluster maintenance operations (restarting
downed instances, activating down DRBD disks, etc.). However, during
maintenance and troubleshooting, this can get in your way; disabling it
Otherwise, if you plan to re-create the cluster, you can just go ahead
and rerun ``gnt-cluster init``.
+Replacing the SSH and SSL keys
+++++++++++++++++++++++++++++++
+
+Ganeti uses both SSL and SSH keys, and actively modifies the SSH keys on
+the nodes. As result, in order to replace these keys, a few extra steps
+need to be followed: :doc:`cluster-keys-replacement`
+
+Monitoring the cluster
+----------------------
+
+Starting with Ganeti 2.8, a monitoring daemon is available, providing
+information about the status and the performance of the system.
+
+The monitoring daemon runs on every node, listening on TCP port 1815. Each
+instance of the daemon provides information related to the node it is running
+on.
+
+.. include:: monitoring-query-format.rst
+
Tags handling
-------------
/cluster foo
/instances/instance1 owner:bar
+Autorepair
+----------
+
+The tool ``harep`` can be used to automatically fix some problems that are
+present in the cluster.
+
+It is mainly meant to be regularly and automatically executed
+as a cron job. This is quite evident by considering that, when executed, it does
+not immediately fix all the issues of the instances of the cluster, but it
+cycles the instances through a series of states, one at every ``harep``
+execution. Every state performs a step towards the resolution of the problem.
+This process goes on until the instance is brought back to the healthy state,
+or the tool realizes that it is not able to fix the instance, and
+therefore marks it as in failure state.
+
+Allowing harep to act on the cluster
+++++++++++++++++++++++++++++++++++++
+
+By default, ``harep`` checks the status of the cluster but it is not allowed to
+perform any modification. Modification must be explicitly allowed by an
+appropriate use of tags. Tagging can be applied at various levels, and can
+enable different kinds of autorepair, as hereafter described.
+
+All the tags that authorize ``harep`` to perform modifications follow this
+syntax::
+
+ ganeti:watcher:autorepair:<type>
+
+where ``<type>`` indicates the kind of intervention that can be performed. Every
+possible value of ``<type>`` includes at least all the authorization of the
+previous one, plus its own. The possible values, in increasing order of
+severity, are:
+
+- ``fix-storage`` allows a disk replacement or another operation that
+ fixes the instance backend storage without affecting the instance
+ itself. This can for example recover from a broken drbd secondary, but
+ risks data loss if something is wrong on the primary but the secondary
+ was somehow recoverable.
+- ``migrate`` allows an instance migration. This can recover from a
+ drained primary, but can cause an instance crash in some cases (bugs).
+- ``failover`` allows instance reboot on the secondary. This can recover
+ from an offline primary, but the instance will lose its running state.
+- ``reinstall`` allows disks to be recreated and an instance to be
+ reinstalled. This can recover from primary&secondary both being
+ offline, or from an offline primary in the case of non-redundant
+ instances. It causes data loss.
+
+These autorepair tags can be applied to a cluster, a nodegroup or an instance,
+and will act where they are applied and to everything in the entities sub-tree
+(e.g. a tag applied to a nodegroup will apply to all the instances contained in
+that nodegroup, but not to the rest of the cluster).
+
+If there are multiple ``ganeti:watcher:autorepair:<type>`` tags in an
+object (cluster, node group or instance), the least destructive tag
+takes precedence. When multiplicity happens across objects, the nearest
+tag wins. For example, if in a cluster with two instances, *I1* and
+*I2*, *I1* has ``failover``, and the cluster itself has both
+``fix-storage`` and ``reinstall``, *I1* will end up with ``failover``
+and *I2* with ``fix-storage``.
+
+Limiting harep
+++++++++++++++
+
+Sometimes it is useful to stop harep from performing its task temporarily,
+and it is useful to be able to do so without distrupting its configuration, that
+is, without removing the authorization tags. In order to do this, suspend tags
+are provided.
+
+Suspend tags can be added to cluster, nodegroup or instances, and act on the
+entire entities sub-tree. No operation will be performed by ``harep`` on the
+instances protected by a suspend tag. Their syntax is as follows::
+
+ ganeti:watcher:autorepair:suspend[:<timestamp>]
+
+If there are multiple suspend tags in an object, the form without timestamp
+takes precedence (permanent suspension); or, if all object tags have a
+timestamp, the one with the highest timestamp.
+
+Tags with a timestamp will be automatically removed when the time indicated by
+the timestamp is passed. Indefinite suspension tags have to be removed manually.
+
+Result reporting
+++++++++++++++++
+
+Harep will report about the result of its actions both through its CLI, and by
+adding tags to the instances it operated on. Such tags will follow the syntax
+hereby described::
+
+ ganeti:watcher:autorepair:result:<type>:<id>:<timestamp>:<result>:<jobs>
+
+If this tag is present a repair of type ``type`` has been performed on
+the instance and has been completed by ``timestamp``. The result is
+either ``success``, ``failure`` or ``enoperm``, and jobs is a
+*+*-separated list of jobs that were executed for this repair.
+
+An ``enoperm`` result is an error state due to permission problems. It
+is returned when the repair cannot proceed because it would require to perform
+an operation that is not allowed by the ``ganeti:watcher:autorepair:<type>`` tag
+that is defining the instance autorepair permissions.
+
+NB: if an instance repair ends up in a failure state, it will not be touched
+again by ``harep`` until it has been manually fixed by the system administrator
+and the ``ganeti:watcher:autorepair:result:failure:*`` tag has been manually
+removed.
Job operations
--------------
++++++++++
The ``cfgupgrade`` tools is used to upgrade between major (and minor)
-Ganeti versions. Point-releases are usually transparent for the admin.
+Ganeti versions, and to roll back. Point-releases are usually
+transparent for the admin.
More information about the upgrade procedure is listed on the wiki at
http://code.google.com/p/ganeti/wiki/UpgradeNotes.
See :doc:`separate documentation for move-instance <move-instance>`.
+users-setup
++++++++++++
+
+Ganeti can either be run entirely as root, or with every daemon running as
+its own specific user (if the parameters ``--with-user-prefix`` and/or
+``--with-group-prefix`` have been specified at ``./configure``-time).
+
+In case split users are activated, they are required to exist on the system,
+and they need to belong to the proper groups in order for the access
+permissions to files and programs to be correct.
+
+The ``users-setup`` tool, when run, takes care of setting up the proper
+users and groups.
+
+When invoked without parameters, the tool runs in interactive mode, showing the
+list of actions it will perform and asking for confirmation before proceeding.
+
+Providing the ``--yes-do-it`` parameter to the tool prevents the confirmation
+from being asked, and the users and groups will be created immediately.
+
.. TODO: document cluster-merge tool
projects / ganeti-local / blobdiff