1 gnt-cluster(8) Ganeti | Version @GANETI_VERSION@
2 ================================================
7 gnt-cluster - Ganeti administration, cluster-wide
12 **gnt-cluster** {command} [arguments...]
17 The **gnt-cluster** is used for cluster-wide administration in the
26 **activate-master-ip**
28 Activates the master IP on the master node.
33 **add-tags** [--from *file*] {*tag*...}
35 Add tags to the cluster. If any of the tags contains invalid
36 characters, the entire operation will abort.
38 If the ``--from`` option is given, the list of tags will be
39 extended with the contents of that file (each line becomes a tag).
40 In this case, there is not need to pass tags on the command line
41 (if you do, both sources will be used). A file name of - will be
47 **command** [-n *node*] [-g *group*] {*command*}
49 Executes a command on all nodes. If the option ``-n`` is not given,
50 the command will be executed on all nodes, otherwise it will be
51 executed only on the node(s) specified. Use the option multiple
52 times for running it on multiple nodes, like::
54 # gnt-cluster command -n node1.example.com -n node2.example.com date
56 The ``-g`` option can be used to run a command only on a specific node
59 # gnt-cluster command -g default date
61 The command is executed serially on the selected nodes. If the
62 master node is present in the list, the command will be executed
63 last on the master. Regarding the other nodes, the execution order
64 is somewhat alphabetic, so that node2.example.com will be earlier
65 than node10.example.com but after node1.example.com.
67 So given the node names node1, node2, node3, node10, node11, with
68 node3 being the master, the order will be: node1, node2, node10,
71 The command is constructed by concatenating all other command line
72 arguments. For example, to list the contents of the /etc directory
75 # gnt-cluster command ls -l /etc
77 and the command which will be executed will be ``ls -l /etc``.
82 | **copyfile** [--use-replication-network] [-n *node*] [-g *group*]
85 Copies a file to all or to some nodes. The argument specifies the
86 source file (on the current system), the ``-n`` argument specifies
87 the target node, or nodes if the option is given multiple times. If
88 ``-n`` is not given at all, the file will be copied to all nodes. The
89 ``-g`` option can be used to only select nodes in a specific node group.
90 Passing the ``--use-replication-network`` option will cause the
91 copy to be done over the replication network (only matters if the
92 primary/secondary IPs are different). Example::
94 # gnt-cluster -n node1.example.com -n node2.example.com copyfile /tmp/test
96 This will copy the file /tmp/test from the current node to the two
102 **deactivate-master-ip** [--yes]
104 Deactivates the master IP on the master node.
106 This should be run only locally or on a connection to the node ip
107 directly, as a connection to the master ip will be broken by this
108 operation. Because of this risk it will require user confirmation
109 unless the ``--yes`` option is passed.
114 **destroy** {--yes-do-it}
116 Remove all configuration files related to the cluster, so that a
117 **gnt-cluster init** can be done again afterwards.
119 Since this is a dangerous command, you are required to pass the
120 argument *--yes-do-it.*
125 **epo** [--on] [--groups|--all] [--power-delay] *arguments*
127 Performs an emergency power-off on nodes given as arguments. If
128 ``--groups`` is given, arguments are node groups. If ``--all`` is
129 provided, the whole cluster will be shut down.
131 The ``--on`` flag recovers the cluster after an emergency power-off.
132 When powering on the cluster you can use ``--power-delay`` to define the
133 time in seconds (fractions allowed) waited between powering on
136 Please note that the master node will not be turned down or up
137 automatically. It will just be left in a state, where you can manully
138 perform the shutdown of that one node. If the master is in the list of
139 affected nodes and this is not a complete cluster emergency power-off
140 (e.g. using ``--all``), you're required to do a master failover to
141 another node not affected.
148 Displays the current master node.
155 Shows runtime cluster information: cluster name, architecture (32
156 or 64 bit), master node, node list and instance list.
158 Passing the ``--roman`` option gnt-cluster info will try to print
159 its integer fields in a latin friendly way. This allows further
160 diffusion of Ganeti among ancient cultures.
166 | [{-s|--secondary-ip} *secondary\_ip*]
167 | [--vg-name *vg-name*]
168 | [--master-netdev *interface-name*]
169 | [--master-netmask *netmask*]
170 | [--use-external-mip-script {yes \| no}]
171 | [{-m|--mac-prefix} *mac-prefix*]
175 | [--file-storage-dir *dir*]
176 | [--enabled-hypervisors *hypervisors*]
177 | [{-H|--hypervisor-parameters} *hypervisor*:*hv-param*=*value*[,*hv-param*=*value*...]]
178 | [{-B|--backend-parameters} *be-param*=*value* [,*be-param*=*value*...]]
179 | [{-N|--nic-parameters} *nic-param*=*value* [,*nic-param*=*value*...]]
180 | [{-D|--disk-parameters} *disk-template*:*disk-param*=*value* [,*disk-param*=*value*...]]
181 | [--maintain-node-health {yes \| no}]
182 | [--uid-pool *user-id pool definition*]
183 | [{-I|--default-iallocator} *default instance allocator*]
184 | [--primary-ip-version *version*]
185 | [--prealloc-wipe-disks {yes \| no}]
186 | [--node-parameters *ndparams*]
187 | [{-C|--candidate-pool-size} *candidate\_pool\_size*]
190 This commands is only run once initially on the first node of the
191 cluster. It will initialize the cluster configuration, setup the
192 ssh-keys, start the daemons on the master node, etc. in order to have
193 a working one-node cluster.
195 Note that the *clustername* is not any random name. It has to be
196 resolvable to an IP address using DNS, and it is best if you give the
197 fully-qualified domain name. This hostname must resolve to an IP
198 address reserved exclusively for this purpose, i.e. not already in
201 The cluster can run in two modes: single-home or dual-homed. In the
202 first case, all traffic (both public traffic, inter-node traffic and
203 data replication traffic) goes over the same interface. In the
204 dual-homed case, the data replication traffic goes over the second
205 network. The ``-s (--secondary-ip)`` option here marks the cluster as
206 dual-homed and its parameter represents this node's address on the
207 second network. If you initialise the cluster with ``-s``, all nodes
208 added must have a secondary IP as well.
210 Note that for Ganeti it doesn't matter if the secondary network is
211 actually a separate physical network, or is done using tunneling,
212 etc. For performance reasons, it's recommended to use a separate
215 The ``--vg-name`` option will let you specify a volume group
216 different than "xenvg" for Ganeti to use when creating instance
217 disks. This volume group must have the same name on all nodes. Once
218 the cluster is initialized this can be altered by using the
219 **modify** command. If you don't want to use lvm storage at all use
220 the ``--no-lvm-storage`` option. Once the cluster is initialized
221 you can change this setup with the **modify** command.
223 The ``--master-netdev`` option is useful for specifying a different
224 interface on which the master will activate its IP address. It's
225 important that all nodes have this interface because you'll need it
226 for a master failover.
228 The ``--master-netmask`` option allows to specify a netmask for the
229 master IP. The netmask must be specified as an integer, and will be
230 interpreted as a CIDR netmask. The default value is 32 for an IPv4
231 address and 128 for an IPv6 address.
233 The ``--use-external-mip-script`` options allows to specify
234 whether to use an user-supplied master IP address setup script, whose
235 location is ``/etc/ganeti/scripts/master-ip-setup``. If the option value
236 is set to False, the default script, whose location is
237 ``/usr/local/lib/ganeti/tools/master-ip-setup``, will be executed.
239 The ``-m (--mac-prefix)`` option will let you specify a three byte
240 prefix under which the virtual MAC addresses of your instances will be
241 generated. The prefix must be specified in the format ``XX:XX:XX`` and
242 the default is ``aa:00:00``.
244 The ``--no-lvm-storage`` option allows you to initialize the
245 cluster without lvm support. This means that only instances using
246 files as storage backend will be possible to create. Once the
247 cluster is initialized you can change this setup with the
250 The ``--no-etc-hosts`` option allows you to initialize the cluster
251 without modifying the /etc/hosts file.
253 The ``--no-ssh-init`` option allows you to initialize the cluster
254 without creating or distributing SSH key pairs.
256 The ``--file-storage-dir`` option allows you set the directory to
257 use for storing the instance disk files when using file storage as
258 backend for instance disks.
260 The ``--prealloc-wipe-disks`` sets a cluster wide configuration
261 value for wiping disks prior to allocation. This increases security
262 on instance level as the instance can't access untouched data from
263 it's underlying storage.
265 The ``--enabled-hypervisors`` option allows you to set the list of
266 hypervisors that will be enabled for this cluster. Instance
267 hypervisors can only be chosen from the list of enabled
268 hypervisors, and the first entry of this list will be used by
269 default. Currently, the following hypervisors are available:
281 a simple chroot manager that starts chroot based on a script at the
282 root of the filesystem holding the chroot
285 fake hypervisor for development/testing
287 Either a single hypervisor name or a comma-separated list of
288 hypervisor names can be specified. If this option is not specified,
289 only the xen-pvm hypervisor is enabled by default.
291 The ``-H (--hypervisor-parameters)`` option allows you to set default
292 hypervisor specific parameters for the cluster. The format of this
293 option is the name of the hypervisor, followed by a colon and a
294 comma-separated list of key=value pairs. The keys available for each
295 hypervisors are detailed in the gnt-instance(8) man page, in the
296 **add** command plus the following parameters which are only
297 configurable globally (at cluster level):
300 Valid for the Xen PVM and KVM hypervisors.
302 This options specifies the TCP port to use for live-migration. For
303 Xen, the same port should be configured on all nodes in the
304 ``/etc/xen/xend-config.sxp`` file, under the key
305 "xend-relocation-port".
308 Valid for the KVM hypervisor.
310 This option specifies the maximum bandwidth that KVM will use for
311 instance live migrations. The value is in MiB/s.
313 This option is only effective with kvm versions >= 78 and qemu-kvm
316 The ``-B (--backend-parameters)`` option allows you to set the default
317 backend parameters for the cluster. The parameter format is a
318 comma-separated list of key=value pairs with the following supported
322 Number of VCPUs to set for an instance by default, must be an
323 integer, will be set to 1 if no specified.
326 Maximum amount of memory to allocate for an instance by default, can
327 be either an integer or an integer followed by a unit (M for
328 mebibytes and G for gibibytes are supported), will be set to 128M if
332 Minimum amount of memory to allocate for an instance by default, can
333 be either an integer or an integer followed by a unit (M for
334 mebibytes and G for gibibytes are supported), will be set to 128M if
338 Value of the auto\_balance flag for instances to use by default,
339 will be set to true if not specified.
342 Default value for the ``always\_failover`` flag for instances; if
343 not set, ``False`` is used.
346 The ``-N (--nic-parameters)`` option allows you to set the default nic
347 parameters for the cluster. The parameter format is a comma-separated
348 list of key=value pairs with the following supported keys:
351 The default nic mode, 'routed' or 'bridged'.
354 In bridged mode the default NIC bridge. In routed mode it
355 represents an hypervisor-vif-script dependent value to allow
356 different instance groups. For example under the KVM default
357 network script it is interpreted as a routing table number or
360 The ``-D (--disk-parameters)`` option allows you to set the default disk
361 template parameters at cluster level. The format used for this option is
362 similar to the one use by the ``-H`` option: the disk template name
363 must be specified first, followed by a colon and by a comma-separated
364 list of key-value pairs. These parameters can only be specified at
365 cluster and node group level; the cluster-level parameter are inherited
366 by the node group at the moment of its creation, and can be further
367 modified at node group level using the **gnt-group**(8) command.
369 List of disk parameters available for the **drbd** template:
372 Re-synchronization rate, expressed in KiB/s
375 Number of stripes to use for data LVs
378 Number of stripes to use for meta LVs
381 What kind of barriers to **disable** for disks. It can either assume
382 the value "n", meaning no barrier disabled, or a non-empty string
383 containing a subset of the characters "bfd". "b" means disable disk
384 barriers, "f" means disable disk flushes, "d" disables disk drains
387 Boolean value indicating whether the meta barriers should be
388 disabled (True) or not (False).
391 String containing the name of the default LVM volume group for DRBD
392 metadata. By default, it is set to ``xenvg``. It can be overridden
393 during the instance creation process by using the ``metavg`` key of
394 the ``--disk`` parameter.
397 String containing additional parameters to be appended to the
398 arguments list of ``drbdsetup disk``.
401 String containing additional parameters to be appended to the
402 arguments list of ``drbdsetup net``.
404 List of parameters available for the **plain** template:
407 Number of stripes to use for new LVs
409 The option ``--maintain-node-health`` allows one to enable/disable
410 automatic maintenance actions on nodes. Currently these include
411 automatic shutdown of instances and deactivation of DRBD devices on
412 offline nodes; in the future it might be extended to automatic
413 removal of unknown LVM volumes, etc. Note that this option is only
414 useful if the use of ``ganeti-confd`` was enabled at compilation.
416 The ``--uid-pool`` option initializes the user-id pool. The
417 *user-id pool definition* can contain a list of user-ids and/or a
418 list of user-id ranges. The parameter format is a comma-separated
419 list of numeric user-ids or user-id ranges. The ranges are defined
420 by a lower and higher boundary, separated by a dash. The boundaries
421 are inclusive. If the ``--uid-pool`` option is not supplied, the
422 user-id pool is initialized to an empty list. An empty list means
423 that the user-id pool feature is disabled.
425 The ``-I (--default-iallocator)`` option specifies the default
426 instance allocator. The instance allocator will be used for operations
427 like instance creation, instance and node migration, etc. when no
428 manual override is specified. If this option is not specified and
429 htools was not enabled at build time, the default instance allocator
430 will be blank, which means that relevant operations will require the
431 administrator to manually specify either an instance allocator, or a
432 set of nodes. If the option is not specified but htools was enabled,
433 the default iallocator will be **hail**(1) (assuming it can be found
434 on disk). The default iallocator can be changed later using the
437 The ``--primary-ip-version`` option specifies the IP version used
438 for the primary address. Possible values are 4 and 6 for IPv4 and
439 IPv6, respectively. This option is used when resolving node names
440 and the cluster name.
442 The ``--node-parameters`` option allows you to set default node
443 parameters for the cluster. Please see **ganeti**(7) for more
444 information about supported key=value pairs.
446 The ``-C (--candidate-pool-size)`` option specifies the
447 ``candidate_pool_size`` cluster parameter. This is the number of nodes
448 that the master will try to keep as master\_candidates. For more
449 details about this role and other node roles, see the ganeti(7).
456 List the tags of the cluster.
461 **master-failover** [--no-voting]
463 Failover the master role to the current node.
465 The ``--no-voting`` option skips the remote node agreement checks.
466 This is dangerous, but necessary in some cases (for example failing
467 over the master role in a 2 node cluster with the original master
468 down). If the original master then comes up, it won't be able to
469 start its master daemon because it won't have enough votes, but so
470 won't the new master, if the master daemon ever needs a restart.
471 You can pass ``--no-voting`` to **ganeti-masterd** on the new
472 master to solve this problem, and run **gnt-cluster redist-conf**
473 to make sure the cluster is consistent again.
480 Checks if the master daemon is alive.
482 If the master daemon is alive and can respond to a basic query (the
483 equivalent of **gnt-cluster info**), then the exit code of the
484 command will be 0. If the master daemon is not alive (either due to
485 a crash or because this is not the master node), the exit code will
492 | [--vg-name *vg-name*]
494 | [--enabled-hypervisors *hypervisors*]
495 | [{-H|--hypervisor-parameters} *hypervisor*:*hv-param*=*value*[,*hv-param*=*value*...]]
496 | [{-B|--backend-parameters} *be-param*=*value* [,*be-param*=*value*...]]
497 | [{-N|--nic-parameters} *nic-param*=*value* [,*nic-param*=*value*...]]
498 | [{-D|--disk-parameters} *disk-template*:*disk-param*=*value* [,*disk-param*=*value*...]]
499 | [--uid-pool *user-id pool definition*]
500 | [--add-uids *user-id pool definition*]
501 | [--remove-uids *user-id pool definition*]
502 | [{-C|--candidate-pool-size} *candidate\_pool\_size*]
503 | [--maintain-node-health {yes \| no}]
504 | [--prealloc-wipe-disks {yes \| no}]
505 | [{-I|--default-iallocator} *default instance allocator*]
506 | [--reserved-lvs=*NAMES*]
507 | [--node-parameters *ndparams*]
508 | [--master-netdev *interface-name*]
509 | [--master-netmask *netmask*]
510 | [--use-external-mip-script {yes \| no}]
511 | [--hypervisor-state *hvstate*]
512 | [--disk-state *diskstate*]
514 Modify the options for the cluster.
516 The ``--vg-name``, ``--no-lvm-storarge``, ``--enabled-hypervisors``,
517 ``-H (--hypervisor-parameters)``, ``-B (--backend-parameters)``,
518 ``-D (--disk-parameters)``, ``--nic-parameters``, ``-C
519 (--candidate-pool-size)``, ``--maintain-node-health``,
520 ``--prealloc-wipe-disks``, ``--uid-pool``, ``--node-parameters``,
521 ``--master-netdev``, ``--master-netmask`` and
522 ``--use-external-mip-script`` options are described in the **init**
525 The ``--hypervisor-state`` and ``--disk-state`` options are described in
526 detail in **ganeti**(7).
528 The ``--add-uids`` and ``--remove-uids`` options can be used to
529 modify the user-id pool by adding/removing a list of user-ids or
532 The option ``--reserved-lvs`` specifies a list (comma-separated) of
533 logical volume group names (regular expressions) that will be
534 ignored by the cluster verify operation. This is useful if the
535 volume group used for Ganeti is shared with the system for other
536 uses. Note that it's not recommended to create and mark as ignored
537 logical volume names which match Ganeti's own name format (starting
538 with UUID and then .diskN), as this option only skips the
539 verification, but not the actual use of the names given.
541 To remove all reserved logical volumes, pass in an empty argument
542 to the option, as in ``--reserved-lvs=`` or ``--reserved-lvs ''``.
544 The ``-I (--default-iallocator)`` is described in the **init**
545 command. To clear the default iallocator, just pass an empty string
551 **queue** {drain | undrain | info}
553 Change job queue properties.
555 The ``drain`` option sets the drain flag on the job queue. No new
556 jobs will be accepted, but jobs already in the queue will be
559 The ``undrain`` will unset the drain flag on the job queue. New
560 jobs will be accepted.
562 The ``info`` option shows the properties of the job queue.
567 **watcher** {pause *duration* | continue | info}
569 Make the watcher pause or let it continue.
571 The ``pause`` option causes the watcher to pause for *duration*
574 The ``continue`` option will let the watcher continue.
576 The ``info`` option shows whether the watcher is currently paused.
581 **redist-conf** [--submit]
583 This command forces a full push of configuration files from the
584 master node to the other nodes in the cluster. This is normally not
585 needed, but can be run if the **verify** complains about
586 configuration mismatches.
588 The ``--submit`` option is used to send the job to the master
589 daemon but not wait for its completion. The job ID will be shown so
590 that it can be examined via **gnt-job info**.
595 **remove-tags** [--from *file*] {*tag*...}
597 Remove tags from the cluster. If any of the tags are not existing
598 on the cluster, the entire operation will abort.
600 If the ``--from`` option is given, the list of tags to be removed will
601 be extended with the contents of that file (each line becomes a tag).
602 In this case, there is not need to pass tags on the command line (if
603 you do, tags from both sources will be removed). A file name of - will
604 be interpreted as stdin.
609 **rename** [-f] {*name*}
611 Renames the cluster and in the process updates the master IP
612 address to the one the new name resolves to. At least one of either
613 the name or the IP address must be different, otherwise the
614 operation will be aborted.
616 Note that since this command can be dangerous (especially when run
617 over SSH), the command will require confirmation unless run with
623 | **renew-crypto** [-f]
624 | [--new-cluster-certificate] [--new-confd-hmac-key]
625 | [--new-rapi-certificate] [--rapi-certificate *rapi-cert*]
626 | [--new-spice-certificate | --spice-certificate *spice-cert*
627 | -- spice-ca-certificate *spice-ca-cert*]
628 | [--new-cluster-domain-secret] [--cluster-domain-secret *filename*]
630 This command will stop all Ganeti daemons in the cluster and start
631 them again once the new certificates and keys are replicated. The
632 options ``--new-cluster-certificate`` and ``--new-confd-hmac-key``
633 can be used to regenerate the cluster-internal SSL certificate
634 respective the HMAC key used by ganeti-confd(8).
636 To generate a new self-signed RAPI certificate (used by
637 ganeti-rapi(8)) specify ``--new-rapi-certificate``. If you want to
638 use your own certificate, e.g. one signed by a certificate
639 authority (CA), pass its filename to ``--rapi-certificate``.
641 To generate a new self-signed SPICE certificate, used by SPICE
642 connections to the KVM hypervisor, specify the
643 ``--new-spice-certificate`` option. If you want to provide a
644 certificate, pass its filename to ``--spice-certificate`` and pass the
645 signing CA certificate to ``--spice-ca-certificate``.
647 ``--new-cluster-domain-secret`` generates a new, random cluster
648 domain secret. ``--cluster-domain-secret`` reads the secret from a
649 file. The cluster domain secret is used to sign information
650 exchanged between separate clusters via a third party.
655 **repair-disk-sizes** [instance...]
657 This command checks that the recorded size of the given instance's
658 disks matches the actual size and updates any mismatches found.
659 This is needed if the Ganeti configuration is no longer consistent
660 with reality, as it will impact some disk operations. If no
661 arguments are given, all instances will be checked.
663 Note that only active disks can be checked by this command; in case
664 a disk cannot be activated it's advised to use
665 **gnt-instance activate-disks --ignore-size ...** to force
666 activation without regard to the current size.
668 When the all disk sizes are consistent, the command will return no
669 output. Otherwise it will log details about the inconsistencies in
675 **search-tags** {*pattern*}
677 Searches the tags on all objects in the cluster (the cluster
678 itself, the nodes and the instances) for a given pattern. The
679 pattern is interpreted as a regular expression and a search will be
680 done on it (i.e. the given pattern is not anchored to the beggining
681 of the string; if you want that, prefix the pattern with ^).
683 If no tags are matching the pattern, the exit code of the command
684 will be one. If there is at least one match, the exit code will be
685 zero. Each match is listed on one line, the object and the tag
686 separated by a space. The cluster will be listed as /cluster, a
687 node will be listed as /nodes/*name*, and an instance as
688 /instances/*name*. Example:
692 # gnt-cluster search-tags time
693 /cluster ctime:2007-09-01
694 /nodes/node1.example.com mtime:2007-10-04
699 | **verify** [--no-nplus1-mem] [--node-group *nodegroup*]
700 | [--error-codes] [{-I|--ignore-errors} *errorcode*]
701 | [{-I|--ignore-errors} *errorcode*...]
703 Verify correctness of cluster configuration. This is safe with
704 respect to running instances, and incurs no downtime of the
707 If the ``--no-nplus1-mem`` option is given, Ganeti won't check
708 whether if it loses a node it can restart all the instances on
709 their secondaries (and report an error otherwise).
711 With ``--node-group``, restrict the verification to those nodes and
712 instances that live in the named group. This will not verify global
713 settings, but will allow to perform verification of a group while other
714 operations are ongoing in other groups.
716 The ``--error-codes`` option outputs each error in the following
717 parseable format: *ftype*:*ecode*:*edomain*:*name*:*msg*.
718 These fields have the following meaning:
721 Failure type. Can be *WARNING* or *ERROR*.
724 Error code of the failure. See below for a list of error codes.
727 Can be *cluster*, *node* or *instance*.
730 Contains the name of the item that is affected from the failure.
733 Contains a descriptive error message about the error
735 ``gnt-cluster verify`` will have a non-zero exit code if at least one of
736 the failures that are found are of type *ERROR*.
738 The ``--ignore-errors`` option can be used to change this behaviour,
739 because it demotes the error represented by the error code received as a
740 parameter to a warning. The option must be repeated for each error that
741 should be ignored (e.g.: ``-I ENODEVERSION -I ENODEORPHANLV``). The
742 ``--error-codes`` option can be used to determine the error code of a
754 The command checks which instances have degraded DRBD disks and
755 activates the disks of those instances.
757 This command is run from the **ganeti-watcher** tool, which also
758 has a different, complementary algorithm for doing this check.
759 Together, these two should ensure that DRBD disks are kept
767 Show the cluster version.
769 .. vim: set textwidth=72 :