1 ganeti(7) Ganeti | Version @GANETI_VERSION@
2 ===========================================
7 ganeti - cluster-based virtualization management
14 # gnt-cluster init cluster1.example.com
15 # gnt-node add node2.example.com
16 # gnt-instance add -n node2.example.com \
17 > -o debootstrap --disk 0:size=30g \
18 > -t plain instance1.example.com
24 The Ganeti software manages physical nodes and virtual instances of a
25 cluster based on a virtualization software. The current version (2.3)
26 supports Xen 3.x and KVM (72 or above) as hypervisors, and LXC as an
27 experimental hypervisor.
32 First you must install the software on all the cluster nodes, either
33 from sources or (if available) from a package. The next step is to
34 create the initial cluster configuration, using **gnt-cluster init**.
36 Then you can add other nodes, or start creating instances.
41 In Ganeti 2.0, the architecture of the cluster is a little more
42 complicated than in 1.2. The cluster is coordinated by a master daemon
43 (**ganeti-masterd**\(8)), running on the master node. Each node runs
44 (as before) a node daemon, and the master has the RAPI daemon running
50 Each node can be in one of the following states:
53 Only one node per cluster can be in this role, and this node is the
54 one holding the authoritative copy of the cluster configuration and
55 the one that can actually execute commands on the cluster and
56 modify the cluster state. See more details under
57 *Cluster configuration*.
60 The node receives the full cluster configuration (configuration
61 file and jobs) and can become a master via the
62 **gnt-cluster master-failover** command. Nodes that are not in this
63 state cannot transition into the master role due to missing state.
66 This the normal state of a node.
69 Nodes in this state are functioning normally but cannot receive
70 new instances, because the intention is to set them to *offline*
71 or remove them from the cluster.
74 These nodes are still recorded in the Ganeti configuration, but
75 except for the master daemon startup voting procedure, they are not
76 actually contacted by the master. This state was added in order to
77 allow broken machines (that are being repaired) to remain in the
78 cluster but without creating problems.
84 Nodes have two flags which govern which roles they can take:
87 The node can become a master candidate, and furthermore the master
88 node. When this flag is disabled, the node cannot become a
89 candidate; this can be useful for special networking cases, or less
93 The node can host instances. When enabled (the default state), the
94 node will participate in instance allocation, capacity calculation,
95 etc. When disabled, the node will be skipped in many cluster checks
102 The ``ndparams`` refer to node parameters. These can be set as defaults
103 on cluster and node group levels, but they take effect for nodes only.
105 Currently we support the following node parameters:
108 Path to an executable used as the out-of-band helper as described in
109 the `Ganeti Node OOB Management Framework <design-oob.rst>`_ design
113 This should reflect the I/O performance of local attached storage
114 (e.g. for "file", "plain" and "drbd" disk templates). It doesn't
115 have to match the actual spindle count of (any eventual) mechanical
116 hard-drives, its meaning is site-local and just the relative values
120 When this Boolean flag is enabled, physical disks on the node are
121 assigned to instance disks in an exclusive manner, so as to lower I/O
122 interference between instances. See the `Partitioned Ganeti
123 <design-partitioned.rst>`_ design document for more details. This
124 parameter cannot be set on individual nodes, as its value must be
125 the same within each node group.
128 Hypervisor State Parameters
129 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
131 Using ``--hypervisor-state`` you can set hypervisor specific states as
132 pointed out in ``Ganeti Resource Model <design-resource-model.rst>``.
134 The format is: ``hypervisor:option=value``.
136 Currently we support the following hypervisor state values:
139 Total node memory, as discovered by this hypervisor
141 Memory used by, or reserved for, the node itself; note that some
142 hypervisors can report this in an authoritative way, other not
144 Memory used either by the hypervisor itself or lost due to instance
145 allocation rounding; usually this cannot be precisely computed, but
146 only roughly estimated
148 Total node cpu (core) count; usually this can be discovered
151 Number of cores reserved for the node itself; this can either be
152 discovered or set manually. Only used for estimating how many VCPUs
153 are left for instances
155 Note that currently this option is unused by Ganeti; values will be
156 recorded but will not influence the Ganeti operation.
159 Disk State Parameters
160 ~~~~~~~~~~~~~~~~~~~~~
162 Using ``--disk-state`` you can set disk specific states as pointed out
163 in ``Ganeti Resource Model <design-resource-model.rst>``.
165 The format is: ``storage_type/identifier:option=value``. Where we
166 currently just support ``lvm`` as storage type. The identifier in this
167 case is the LVM volume group. By default this is ``xenvg``.
169 Currently we support the following hypervisor state values:
172 Total disk size (usually discovered automatically)
174 Reserved disk size; this is a lower limit on the free space, if such a
177 Disk that is expected to be used by other volumes (set via
178 ``reserved_lvs``); usually should be zero
180 Note that currently this option is unused by Ganeti; values will be
181 recorded but will not influence the Ganeti operation.
184 Cluster configuration
185 ~~~~~~~~~~~~~~~~~~~~~
187 The master node keeps and is responsible for the cluster
188 configuration. On the filesystem, this is stored under the
189 ``@LOCALSTATEDIR@/ganeti/lib`` directory, and if the master daemon is
190 stopped it can be backed up normally.
192 The master daemon will replicate the configuration database called
193 ``config.data`` and the job files to all the nodes in the master
194 candidate role. It will also distribute a copy of some configuration
195 values via the *ssconf* files, which are stored in the same directory
196 and start with a ``ssconf_`` prefix, to all nodes.
201 All cluster modification are done via jobs. A job consists of one
202 or more opcodes, and the list of opcodes is processed serially. If
203 an opcode fails, the entire job is failed and later opcodes are no
204 longer processed. A job can be in one of the following states:
207 The job has been submitted but not yet processed by the master
211 The job is waiting for for locks before the first of its opcodes.
214 The job is waiting for locks, but is has been marked for
215 cancellation. It will not transition to *running*, but to
219 The job is currently being executed.
222 The job has been canceled before starting execution.
225 The job has finished successfully.
228 The job has failed during runtime, or the master daemon has been
229 stopped during the job execution.
232 Common command line features
233 ----------------------------
238 Many Ganeti commands provide the following options. The
239 availability for a certain command can be checked by calling the
240 command using the ``--help`` option.
242 | **gnt-...** *command* [\--dry-run] [\--priority {low | normal | high}]
243 | [\--submit] [\--print-job-id]
245 The ``--dry-run`` option can be used to check whether an operation
248 The option ``--priority`` sets the priority for opcodes submitted
251 The ``--submit`` option is used to send the job to the master daemon but
252 not wait for its completion. The job ID will be shown so that it can be
253 examined using **gnt-job info**.
255 The ``--print-job-id`` option makes the command print the job id as first
256 line on stdout, so that it is easy to parse by other programs.
261 For certain commands you can use environment variables to provide
262 default command line arguments. Just assign the arguments as a string to
263 the corresponding environment variable. The format of that variable
264 name is **binary**_*command*. **binary** is the name of the ``gnt-*``
265 script all upper case and dashes replaced by underscores, and *command*
266 is the command invoked on that script.
268 Currently supported commands are ``gnt-node list``, ``gnt-group list``
269 and ``gnt-instance list``. So you can configure default command line
270 flags by setting ``GNT_NODE_LIST``, ``GNT_GROUP_LIST`` and
271 ``GNT_INSTANCE_LIST``.
276 If the variable ``FORCE_LUXI_SOCKET`` is set, it will override the
277 socket used for LUXI connections by command-line tools
278 (``gnt-*``). This is useful mostly for debugging, and some operations
279 won't work at all if, for example, you point this variable to the
280 confd-supplied query socket and try to submit a job.
282 If the variable is set to the value ``master``, it will connect to the
283 correct path for the master daemon (even if, for example, split
284 queries are enabled and this is a query operation). If set to
285 ``query``, it will always (try to) connect to the query socket, even
286 if split queries are disabled. Otherwise, the value is taken to
287 represent a filesystem path to the socket to use.
292 Multiple ganeti commands use the same framework for tabular listing of
293 resources (e.g. **gnt-instance list**, **gnt-node list**, **gnt-group
294 list**, **gnt-debug locks**, etc.). For these commands, special states
295 are denoted via a special symbol (in terse mode) or a string (in
299 The node in question is marked offline, and thus it cannot be
300 queried for data. This result is persistent until the node is
304 Ganeti expected to receive an answer from this entity, but the
305 cluster RPC call failed and/or we didn't receive a valid answer;
306 usually more information is available in the node daemon log (if
307 the node is alive) or the master daemon log. This result is
308 transient, and re-running command might return a different result.
311 The respective field doesn't make sense for this entity;
312 e.g. querying a down instance for its current memory 'live' usage,
313 or querying a non-vm_capable node for disk/memory data. This
314 result is persistent, and until the entity state is changed via
315 ganeti commands, the result won't change.
318 This field is not known (note that this is different from entity
319 being unknown). Either you have mis-typed the field name, or you
320 are using a field that the running Ganeti master daemon doesn't
321 know. This result is persistent, re-running the command won't
327 Multiple options take parameters that are of the form
328 ``key=value,key=value,...`` or ``category:key=value,...``. Examples
329 are the hypervisor parameters, backend parameters, etc. For these,
330 it's possible to use values that contain commas by escaping with via a
331 backslash (which needs two if not single-quoted, due to shell
334 # gnt-instance modify -H kernel_path=an\\,example instance1
335 # gnt-instance modify -H kernel_path='an\,example' instance1
340 Most commands listing resources (e.g. instances or nodes) support filtering.
341 The filter language is similar to Python expressions with some elements from
342 Perl. The language is not generic. Each condition must consist of a field name
343 and a value (except for boolean checks), a field can not be compared to another
344 field. Keywords are case-sensitive.
346 Examples (see below for syntax details):
350 gnt-instance list --filter 'name =* "web*.example.com"'
352 - List instances with three or six virtual CPUs and whose primary
353 nodes reside in groups starting with the string "rack"::
355 gnt-instance list --filter
356 '(be/vcpus == 3 or be/vcpus == 6) and pnode.group =~ m/^rack/'
358 - Nodes hosting primary instances::
360 gnt-node list --filter 'pinst_cnt != 0'
362 - Nodes which aren't master candidates::
364 gnt-node list --filter 'not master_candidate'
366 - Short version for globbing patterns::
368 gnt-instance list '*.site1' '*.site2'
370 Syntax in pseudo-BNF::
372 <quoted-string> ::= /* String quoted with single or double quotes,
373 backslash for escaping */
375 <integer> ::= /* Number in base-10 positional notation */
377 <re> ::= /* Regular expression */
380 Modifier "i": Case-insensitive matching, see
381 http://docs.python.org/library/re#re.IGNORECASE
383 Modifier "s": Make the "." special character match any character,
384 including newline, see http://docs.python.org/library/re#re.DOTALL
386 <re-modifiers> ::= /* empty */ | i | s
388 <value> ::= <quoted-string> | <integer>
391 { /* Value comparison */
392 <field> { == | != | < | <= | >= | > } <value>
394 /* Collection membership */
395 | <value> [ not ] in <field>
397 /* Regular expressions (recognized delimiters
398 are "/", "#", "^", and "|"; backslash for escaping)
400 | <field> { =~ | !~ } m/<re>/<re-modifiers>
403 | <field> { =* | !* } <quoted-string>
410 { [ not ] <condition> | ( <filter> ) }
411 [ { and | or } <filter> ]
426 Greater than or equal
428 Pattern match using regular expression
430 Logically negated from *=~*
432 Globbing, see **glob**\(7), though only * and ? are supported
434 Logically negated from *=\**
436 Collection membership and negation
439 Common daemon functionality
440 ---------------------------
442 All Ganeti daemons re-open the log file(s) when sent a SIGHUP signal.
443 **logrotate**\(8) can be used to rotate Ganeti's log files.
445 .. vim: set textwidth=72 :