1 Ganeti automatic instance allocation
2 ====================================
4 Documents Ganeti version 2.8
11 Currently in Ganeti the admin has to specify the exact locations for
12 an instance's node(s). This prevents a completely automatic node
13 evacuation, and is in general a nuisance.
15 The *iallocator* framework will enable automatic placement via
16 external scripts, which allows customization of the cluster layout per
17 the site's requirements.
22 There are two parts of the ganeti operation that are impacted by the
23 auto-allocation: how the cluster knows what the allocator algorithms
24 are and how the admin uses these in creating instances.
26 An allocation algorithm is just the filename of a program installed in
27 a defined list of directories.
32 At configure time, the list of the directories can be selected via the
33 ``--with-iallocator-search-path=LIST`` option, where *LIST* is a
34 comma-separated list of directories. If not given, this defaults to
35 ``$libdir/ganeti/iallocators``, i.e. for an installation under
36 ``/usr``, this will be ``/usr/lib/ganeti/iallocators``.
38 Ganeti will then search for allocator script in the configured list,
39 using the first one whose filename matches the one given by the user.
41 Command line interface changes
42 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44 The node selection options in instance add and instance replace disks
45 can be replace by the new ``--iallocator=NAME`` option (shortened to
46 ``-I``), which will cause the auto-assignement of nodes with the
47 passed iallocator. The selected node(s) will be show as part of the
53 The protocol for communication between Ganeti and an allocator script
54 will be the following:
56 #. ganeti launches the program with a single argument, a filename that
57 contains a JSON-encoded structure (the input message)
59 #. if the script finishes with exit code different from zero, it is
60 considered a general failure and the full output will be reported to
61 the users; this can be the case when the allocator can't parse the
64 #. if the allocator finishes with exit code zero, it is expected to
65 output (on its stdout) a JSON-encoded structure (the response)
70 The input message will be the JSON encoding of a dictionary containing
71 all the required information to perform the operation. We explain the
72 contents of this dictionary in two parts: common information that every
73 type of operation requires, and operation-specific information.
78 All input dictionaries to the IAllocator must carry the following keys:
81 the version of the protocol; this document
88 the list of cluster tags
91 the list of enabled hypervisors
94 the cluster-wide instance policy (for information; the per-node group
95 values take precedence and should be used instead)
98 a dictionary containing the details of the request; the keys vary
99 depending on the type of operation that's being requested, as
100 explained in `Operation-specific input`_ below.
103 a dictionary with the data for the cluster's node groups; it is keyed
104 on the group UUID, and the values are a dictionary with the following
110 the allocation policy of the node group (consult the semantics of
111 this attribute in the :manpage:`gnt-group(8)` manpage)
113 the list of network UUID's this node group is connected to
115 the instance policy of the node group
117 the list of node group tags
120 a dictionary with the data for the current existing instance on the
121 cluster, indexed by instance name; the contents are similar to the
122 instance definitions for the allocate mode, with the addition of:
125 if this instance is set to run (but not the actual status of the
129 list of nodes on which this instance is placed; the primary node
130 of the instance is always the first one
133 dictionary with the data for the nodes in the cluster, indexed by
134 the node name; the dict contains [*]_ :
137 the total disk size of this node (mebibytes)
140 the free disk space on the node
143 the total memory size
146 free memory on the node; note that currently this does not take
147 into account the instances which are down on the node
150 the physical number of CPUs present on the machine; depending on
151 the hypervisor, this might or might not be equal to how many CPUs
152 the node operating system sees;
155 the primary IP address of the node
158 the secondary IP address of the node (the one used for the DRBD
159 replication); note that this can be the same as the primary one
162 list with the tags of the node
165 a boolean flag denoting whether this node is a master candidate
168 a boolean flag denoting whether this node is being drained
171 a boolean flag denoting whether this node is offline
174 total memory required by primary instances
177 total memory required by running primary instances
180 the node group that this node belongs to
182 No allocations should be made on nodes having either the ``drained``
183 or ``offline`` flags set. More details about these of node status
184 flags is available in the manpage :manpage:`ganeti(7)`.
186 .. [*] Note that no run-time data is present for offline, drained or
187 non-vm_capable nodes; this means the tags total_memory,
188 reserved_memory, free_memory, total_disk, free_disk, total_cpus,
189 i_pri_memory and i_pri_up memory will be absent
191 Operation-specific input
192 ++++++++++++++++++++++++
194 All input dictionaries to the IAllocator carry, in the ``request``
195 dictionary, detailed information about the operation that's being
196 requested. The required keys vary depending on the type of operation, as
199 In all cases, it includes:
202 the request type; this can be either ``allocate``, ``relocate``,
203 ``change-group`` or ``node-evacuate``. The
204 ``allocate`` request is used when a new instance needs to be placed
205 on the cluster. The ``relocate`` request is used when an existing
206 instance needs to be moved within its node group.
208 The ``multi-evacuate`` protocol used to request that the script
209 computes the optimal relocate solution for all secondary instances
210 of the given nodes. It is now deprecated and needs only be
211 implemented if backwards compatibility with Ganeti 2.4 and lower is
214 The ``change-group`` request is used to relocate multiple instances
215 across multiple node groups. ``node-evacuate`` evacuates instances
216 off their node(s). These are described in a separate :ref:`design
217 document <multi-reloc-detailed-design>`.
219 The ``multi-allocate`` request is used to allocate multiple
220 instances on the cluster. The request is beside of that very
221 similiar to the ``allocate`` one. For more details look at
222 :doc:`Ganeti bulk create <design-bulk-create>`.
224 For both allocate and relocate mode, the following extra keys are needed
225 in the ``request`` dictionary:
228 the name of the instance; if the request is a realocation, then this
229 name will be found in the list of instances (see below), otherwise
230 is the FQDN of the new instance; type *string*
233 how many nodes should the algorithm return; while this information
234 can be deduced from the instace's disk template, it's better if
235 this computation is left to Ganeti as then allocator scripts are
236 less sensitive to changes to the disk templates; type *integer*
239 the total disk space that will be used by this instance on the
240 (new) nodes; again, this information can be computed from the list
241 of instance disks and its template type, but Ganeti is better
242 suited to compute it; type *integer*
246 constants.DISK_ACCESS_SET == set([constants.DISK_RDONLY,
247 constants.DISK_RDWR])
249 Allocation needs, in addition:
252 list of dictionaries holding the disk definitions for this
253 instance (in the order they are exported to the hypervisor):
256 either :pyeval:`constants.DISK_RDONLY` or
257 :pyeval:`constants.DISK_RDWR` denoting if the disk is read-only or
261 the size of this disk in mebibytes
264 a list of dictionaries holding the network interfaces for this
265 instance, containing:
268 the IP address that Ganeti know for this instance, or null
271 the MAC address for this interface
274 the bridge to which this interface will be connected
277 the number of VCPUs for the instance
280 the disk template for the instance
283 the memory size for the instance
286 the OS type for the instance
289 the list of the instance's tags
292 the hypervisor of this instance
297 a list of nodes to move the instance away from; for DRBD-based
298 instances, this will contain a single node, the current secondary
299 of the instance, whereas for shared-storage instance, this will
300 contain also a single node, the current primary of the instance;
301 type *list of strings*
303 As for ``node-evacuate``, it needs the following request arguments:
306 a list of instance names to evacuate; type *list of strings*
309 specify which instances to evacuate; one of ``primary-only``,
310 ``secondary-only``, ``all``, type *string*
312 ``change-group`` needs the following request arguments:
315 a list of instance names whose group to change; type
319 must either be the empty list, or contain a list of group UUIDs that
320 should be considered for relocating instances to; type
323 ``multi-allocate`` needs the following request arguments:
326 a list of request dicts
331 The response message is much more simple than the input one. It is
332 also a dict having three keys:
335 a boolean value denoting if the allocation was successful or not
338 a string with information from the scripts; if the allocation fails,
339 this will be shown to the user
342 the output of the algorithm; even if the algorithm failed
343 (i.e. success is false), this must be returned as an empty list
345 for allocate/relocate, this is the list of node(s) for the instance;
346 note that the length of this list must equal the ``requested_nodes``
347 entry in the input message, otherwise Ganeti will consider the result
350 for the ``node-evacuate`` and ``change-group`` modes, this is a
351 dictionary containing, among other information, a list of lists of
352 serialized opcodes; see the :ref:`design document
353 <multi-reloc-result>` for a detailed description
355 for the ``multi-allocate`` mode this is a tuple of 2 lists, the first
356 being element of the tuple is a list of succeeded allocation, with the
357 instance name as first element of each entry and the node placement in
358 the second. The second element of the tuple is the instance list of
361 .. note:: Current Ganeti version accepts either ``result`` or ``nodes``
362 as a backwards-compatibility measure (older versions only supported
368 Input messages to scripts
369 ~~~~~~~~~~~~~~~~~~~~~~~~~
371 Input message, new instance allocation (common elements are listed this
372 time, but not included in further examples below)::
376 "cluster_name": "cluster1.example.com",
378 "enabled_hypervisors": [
382 "f4e06e0d-528a-4963-a5ad-10f3e114232d": {
384 "alloc_policy": "preferred",
385 "networks": ["net-uuid-1", "net-uuid-2"],
387 "disk-templates": ["drbd", "plain"],
394 "memory-size": 12800,
408 "spindle-ratio": 32.0,
419 "tags": ["ng-tag-1", "ng-tag-2"]
423 "instance1.example.com": {
439 "mac": "aa:00:00:00:60:bf",
444 "disk_template": "plain",
449 "os": "debootstrap+default"
451 "instance2.example.com": {
467 "mac": "aa:00:00:55:f8:38",
472 "disk_template": "drbd",
478 "os": "debootstrap+default"
482 "node1.example.com": {
483 "total_disk": 858276,
484 "primary_ip": "198.51.100.1",
485 "secondary_ip": "192.0.2.1",
487 "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
492 "node2.example.com": {
493 "total_disk": 858240,
494 "primary_ip": "198.51.100.2",
495 "secondary_ip": "192.0.2.2",
497 "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
502 "node3.example.com.com": {
503 "total_disk": 572184,
504 "primary_ip": "198.51.100.3",
505 "secondary_ip": "192.0.2.3",
507 "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
515 "name": "instance3.example.com",
517 "disk_space_total": 3328,
531 "mac": "00:11:22:33:44:55",
536 "disk_template": "drbd",
538 "os": "debootstrap+default",
543 hypervisor: "xen-pvm"
547 Input message, reallocation::
554 "name": "instance2.example.com",
556 "disk_space_total": 832,
566 Successful response message::
570 "info": "Allocation successful",
577 Failed response message::
581 "info": "Can't find a suitable node for position 2 (already selected: node2.example.com)",
585 Successful node evacuation message::
589 "info": "Request successful",
603 Command line messages
604 ~~~~~~~~~~~~~~~~~~~~~
607 # gnt-instance add -t plain -m 2g --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance3
608 Selected nodes for the instance: node1.example.com
609 * creating instance disks...
612 # gnt-instance add -t plain -m 3400m --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance4
613 Failure: prerequisites not met for this operation:
614 Can't compute nodes using iallocator 'hail': Can't find a suitable node for position 1 (already selected: )
616 # gnt-instance add -t drbd -m 1400m --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance5
617 Failure: prerequisites not met for this operation:
618 Can't compute nodes using iallocator 'hail': Can't find a suitable node for position 2 (already selected: node1.example.com)
620 Reference implementation
621 ~~~~~~~~~~~~~~~~~~~~~~~~
623 Ganeti's default iallocator is "hail" which is available when "htools"
624 components have been enabled at build time (see :doc:`install-quick` for
627 .. vim: set textwidth=72 :