root / doc / admin.rst @ ab6536ba
History | View | Annotate | Download (66.2 kB)
1 | ffa6869f | Iustin Pop | Ganeti administrator's guide |
---|---|---|---|
2 | ffa6869f | Iustin Pop | ============================ |
3 | ffa6869f | Iustin Pop | |
4 | fd07c6b3 | Iustin Pop | Documents Ganeti version |version| |
5 | ffa6869f | Iustin Pop | |
6 | ffa6869f | Iustin Pop | .. contents:: |
7 | ffa6869f | Iustin Pop | |
8 | 73225861 | Iustin Pop | .. highlight:: shell-example |
9 | c71a1a3d | Iustin Pop | |
10 | ffa6869f | Iustin Pop | Introduction |
11 | ffa6869f | Iustin Pop | ------------ |
12 | ffa6869f | Iustin Pop | |
13 | c71a1a3d | Iustin Pop | Ganeti is a virtualization cluster management software. You are expected |
14 | c71a1a3d | Iustin Pop | to be a system administrator familiar with your Linux distribution and |
15 | c71a1a3d | Iustin Pop | the Xen or KVM virtualization environments before using it. |
16 | ffa6869f | Iustin Pop | |
17 | ffa6869f | Iustin Pop | The various components of Ganeti all have man pages and interactive |
18 | c71a1a3d | Iustin Pop | help. This manual though will help you getting familiar with the system |
19 | c71a1a3d | Iustin Pop | by explaining the most common operations, grouped by related use. |
20 | ffa6869f | Iustin Pop | |
21 | ffa6869f | Iustin Pop | After a terminology glossary and a section on the prerequisites needed |
22 | c71a1a3d | Iustin Pop | to use this manual, the rest of this document is divided in sections |
23 | c71a1a3d | Iustin Pop | for the different targets that a command affects: instance, nodes, etc. |
24 | ffa6869f | Iustin Pop | |
25 | c71a1a3d | Iustin Pop | .. _terminology-label: |
26 | ffa6869f | Iustin Pop | |
27 | ffa6869f | Iustin Pop | Ganeti terminology |
28 | c71a1a3d | Iustin Pop | ++++++++++++++++++ |
29 | ffa6869f | Iustin Pop | |
30 | c71a1a3d | Iustin Pop | This section provides a small introduction to Ganeti terminology, which |
31 | c71a1a3d | Iustin Pop | might be useful when reading the rest of the document. |
32 | ffa6869f | Iustin Pop | |
33 | ffa6869f | Iustin Pop | Cluster |
34 | c71a1a3d | Iustin Pop | ~~~~~~~ |
35 | ffa6869f | Iustin Pop | |
36 | c71a1a3d | Iustin Pop | A set of machines (nodes) that cooperate to offer a coherent, highly |
37 | c71a1a3d | Iustin Pop | available virtualization service under a single administration domain. |
38 | ffa6869f | Iustin Pop | |
39 | c71a1a3d | Iustin Pop | Node |
40 | c71a1a3d | Iustin Pop | ~~~~ |
41 | c71a1a3d | Iustin Pop | |
42 | c71a1a3d | Iustin Pop | A physical machine which is member of a cluster. Nodes are the basic |
43 | c71a1a3d | Iustin Pop | cluster infrastructure, and they don't need to be fault tolerant in |
44 | c71a1a3d | Iustin Pop | order to achieve high availability for instances. |
45 | c71a1a3d | Iustin Pop | |
46 | c71a1a3d | Iustin Pop | Node can be added and removed (if they host no instances) at will from |
47 | c71a1a3d | Iustin Pop | the cluster. In a HA cluster and only with HA instances, the loss of any |
48 | c71a1a3d | Iustin Pop | single node will not cause disk data loss for any instance; of course, |
49 | 7e2f08ba | Helga Velroyen | a node crash will cause the crash of its primary instances. |
50 | c71a1a3d | Iustin Pop | |
51 | c71a1a3d | Iustin Pop | A node belonging to a cluster can be in one of the following roles at a |
52 | c71a1a3d | Iustin Pop | given time: |
53 | c71a1a3d | Iustin Pop | |
54 | c71a1a3d | Iustin Pop | - *master* node, which is the node from which the cluster is controlled |
55 | c71a1a3d | Iustin Pop | - *master candidate* node, only nodes in this role have the full cluster |
56 | c71a1a3d | Iustin Pop | configuration and knowledge, and only master candidates can become the |
57 | c71a1a3d | Iustin Pop | master node |
58 | c71a1a3d | Iustin Pop | - *regular* node, which is the state in which most nodes will be on |
59 | c71a1a3d | Iustin Pop | bigger clusters (>20 nodes) |
60 | c71a1a3d | Iustin Pop | - *drained* node, nodes in this state are functioning normally but the |
61 | c71a1a3d | Iustin Pop | cannot receive new instances; the intention is that nodes in this role |
62 | c71a1a3d | Iustin Pop | have some issue and they are being evacuated for hardware repairs |
63 | c71a1a3d | Iustin Pop | - *offline* node, in which there is a record in the cluster |
64 | c71a1a3d | Iustin Pop | configuration about the node, but the daemons on the master node will |
65 | c71a1a3d | Iustin Pop | not talk to this node; any instances declared as having an offline |
66 | c71a1a3d | Iustin Pop | node as either primary or secondary will be flagged as an error in the |
67 | c71a1a3d | Iustin Pop | cluster verify operation |
68 | c71a1a3d | Iustin Pop | |
69 | c71a1a3d | Iustin Pop | Depending on the role, each node will run a set of daemons: |
70 | c71a1a3d | Iustin Pop | |
71 | 5e360222 | Michele Tartara | - the :command:`ganeti-noded` daemon, which controls the manipulation of |
72 | c71a1a3d | Iustin Pop | this node's hardware resources; it runs on all nodes which are in a |
73 | c71a1a3d | Iustin Pop | cluster |
74 | c71a1a3d | Iustin Pop | - the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all |
75 | 10d3f678 | Iustin Pop | nodes, but is only functional on master candidate nodes; this daemon |
76 | 10d3f678 | Iustin Pop | can be disabled at configuration time if you don't need its |
77 | 10d3f678 | Iustin Pop | functionality |
78 | c71a1a3d | Iustin Pop | - the :command:`ganeti-rapi` daemon which runs on the master node and |
79 | c71a1a3d | Iustin Pop | offers an HTTP-based API for the cluster |
80 | c71a1a3d | Iustin Pop | - the :command:`ganeti-masterd` daemon which runs on the master node and |
81 | c71a1a3d | Iustin Pop | allows control of the cluster |
82 | ffa6869f | Iustin Pop | |
83 | feec31d1 | Iustin Pop | Beside the node role, there are other node flags that influence its |
84 | feec31d1 | Iustin Pop | behaviour: |
85 | feec31d1 | Iustin Pop | |
86 | feec31d1 | Iustin Pop | - the *master_capable* flag denotes whether the node can ever become a |
87 | feec31d1 | Iustin Pop | master candidate; setting this to 'no' means that auto-promotion will |
88 | feec31d1 | Iustin Pop | never make this node a master candidate; this flag can be useful for a |
89 | feec31d1 | Iustin Pop | remote node that only runs local instances, and having it become a |
90 | feec31d1 | Iustin Pop | master is impractical due to networking or other constraints |
91 | feec31d1 | Iustin Pop | - the *vm_capable* flag denotes whether the node can host instances or |
92 | feec31d1 | Iustin Pop | not; for example, one might use a non-vm_capable node just as a master |
93 | feec31d1 | Iustin Pop | candidate, for configuration backups; setting this flag to no |
94 | feec31d1 | Iustin Pop | disallows placement of instances of this node, deactivates hypervisor |
95 | feec31d1 | Iustin Pop | and related checks on it (e.g. bridge checks, LVM check, etc.), and |
96 | feec31d1 | Iustin Pop | removes it from cluster capacity computations |
97 | feec31d1 | Iustin Pop | |
98 | feec31d1 | Iustin Pop | |
99 | ffa6869f | Iustin Pop | Instance |
100 | c71a1a3d | Iustin Pop | ~~~~~~~~ |
101 | ffa6869f | Iustin Pop | |
102 | c71a1a3d | Iustin Pop | A virtual machine which runs on a cluster. It can be a fault tolerant, |
103 | c71a1a3d | Iustin Pop | highly available entity. |
104 | ffa6869f | Iustin Pop | |
105 | c71a1a3d | Iustin Pop | An instance has various parameters, which are classified in three |
106 | c71a1a3d | Iustin Pop | categories: hypervisor related-parameters (called ``hvparams``), general |
107 | c71a1a3d | Iustin Pop | parameters (called ``beparams``) and per network-card parameters (called |
108 | c71a1a3d | Iustin Pop | ``nicparams``). All these parameters can be modified either at instance |
109 | c71a1a3d | Iustin Pop | level or via defaults at cluster level. |
110 | ffa6869f | Iustin Pop | |
111 | c71a1a3d | Iustin Pop | Disk template |
112 | ffa6869f | Iustin Pop | ~~~~~~~~~~~~~ |
113 | ffa6869f | Iustin Pop | |
114 | c71a1a3d | Iustin Pop | The are multiple options for the storage provided to an instance; while |
115 | c71a1a3d | Iustin Pop | the instance sees the same virtual drive in all cases, the node-level |
116 | c71a1a3d | Iustin Pop | configuration varies between them. |
117 | ffa6869f | Iustin Pop | |
118 | 7ed400f0 | Stratos Psomadakis | There are five disk templates you can choose from: |
119 | ffa6869f | Iustin Pop | |
120 | c71a1a3d | Iustin Pop | diskless |
121 | c71a1a3d | Iustin Pop | The instance has no disks. Only used for special purpose operating |
122 | c71a1a3d | Iustin Pop | systems or for testing. |
123 | c71a1a3d | Iustin Pop | |
124 | c71a1a3d | Iustin Pop | file |
125 | c71a1a3d | Iustin Pop | The instance will use plain files as backend for its disks. No |
126 | c71a1a3d | Iustin Pop | redundancy is provided, and this is somewhat more difficult to |
127 | da535d0d | Guido Trotter | configure for high performance. Note that for security reasons the |
128 | da535d0d | Guido Trotter | file storage directory must be listed under |
129 | da535d0d | Guido Trotter | ``/etc/ganeti/file-storage-paths``, and that file is not copied |
130 | da535d0d | Guido Trotter | automatically to all nodes by Ganeti. |
131 | da535d0d | Guido Trotter | |
132 | da535d0d | Guido Trotter | sharedfile |
133 | da535d0d | Guido Trotter | The instance will use plain files as backend, but Ganeti assumes that |
134 | da535d0d | Guido Trotter | those files will be available and in sync automatically on all nodes. |
135 | da535d0d | Guido Trotter | This allows live migration and failover of instances using this |
136 | da535d0d | Guido Trotter | method. As for ``file`` the file storage directory must be listed under |
137 | da535d0d | Guido Trotter | ``/etc/ganeti/file-storage-paths`` or ganeti will refuse to create |
138 | da535d0d | Guido Trotter | instances under it. |
139 | c71a1a3d | Iustin Pop | |
140 | c71a1a3d | Iustin Pop | plain |
141 | c71a1a3d | Iustin Pop | The instance will use LVM devices as backend for its disks. No |
142 | c71a1a3d | Iustin Pop | redundancy is provided. |
143 | c71a1a3d | Iustin Pop | |
144 | c71a1a3d | Iustin Pop | drbd |
145 | c71a1a3d | Iustin Pop | .. note:: This is only valid for multi-node clusters using DRBD 8.0+ |
146 | c71a1a3d | Iustin Pop | |
147 | c71a1a3d | Iustin Pop | A mirror is set between the local node and a remote one, which must be |
148 | c71a1a3d | Iustin Pop | specified with the second value of the --node option. Use this option |
149 | c71a1a3d | Iustin Pop | to obtain a highly available instance that can be failed over to a |
150 | c71a1a3d | Iustin Pop | remote node should the primary one fail. |
151 | c71a1a3d | Iustin Pop | |
152 | 24ccfd74 | Johan Euphrosine | .. note:: Ganeti does not support DRBD stacked devices: |
153 | 24ccfd74 | Johan Euphrosine | DRBD stacked setup is not fully symmetric and as such it is |
154 | 24ccfd74 | Johan Euphrosine | not working with live migration. |
155 | 24ccfd74 | Johan Euphrosine | |
156 | 7ed400f0 | Stratos Psomadakis | rbd |
157 | 7ed400f0 | Stratos Psomadakis | The instance will use Volumes inside a RADOS cluster as backend for its |
158 | 7ed400f0 | Stratos Psomadakis | disks. It will access them using the RADOS block device (RBD). |
159 | 7ed400f0 | Stratos Psomadakis | |
160 | da535d0d | Guido Trotter | ext |
161 | da535d0d | Guido Trotter | The instance will use an external storage provider. See |
162 | da535d0d | Guido Trotter | :manpage:`ganeti-extstorage-interface(7)` for how to implement one. |
163 | da535d0d | Guido Trotter | |
164 | da535d0d | Guido Trotter | |
165 | c71a1a3d | Iustin Pop | IAllocator |
166 | c71a1a3d | Iustin Pop | ~~~~~~~~~~ |
167 | c71a1a3d | Iustin Pop | |
168 | c71a1a3d | Iustin Pop | A framework for using external (user-provided) scripts to compute the |
169 | c71a1a3d | Iustin Pop | placement of instances on the cluster nodes. This eliminates the need to |
170 | c71a1a3d | Iustin Pop | manually specify nodes in instance add, instance moves, node evacuate, |
171 | c71a1a3d | Iustin Pop | etc. |
172 | c71a1a3d | Iustin Pop | |
173 | c71a1a3d | Iustin Pop | In order for Ganeti to be able to use these scripts, they must be place |
174 | c71a1a3d | Iustin Pop | in the iallocator directory (usually ``lib/ganeti/iallocators`` under |
175 | c71a1a3d | Iustin Pop | the installation prefix, e.g. ``/usr/local``). |
176 | c71a1a3d | Iustin Pop | |
177 | c71a1a3d | Iustin Pop | โPrimaryโ and โsecondaryโ concepts |
178 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
179 | c71a1a3d | Iustin Pop | |
180 | c71a1a3d | Iustin Pop | An instance has a primary and depending on the disk configuration, might |
181 | c71a1a3d | Iustin Pop | also have a secondary node. The instance always runs on the primary node |
182 | c71a1a3d | Iustin Pop | and only uses its secondary node for disk replication. |
183 | c71a1a3d | Iustin Pop | |
184 | c71a1a3d | Iustin Pop | Similarly, the term of primary and secondary instances when talking |
185 | c71a1a3d | Iustin Pop | about a node refers to the set of instances having the given node as |
186 | c71a1a3d | Iustin Pop | primary, respectively secondary. |
187 | c71a1a3d | Iustin Pop | |
188 | c71a1a3d | Iustin Pop | Tags |
189 | c71a1a3d | Iustin Pop | ~~~~ |
190 | c71a1a3d | Iustin Pop | |
191 | c71a1a3d | Iustin Pop | Tags are short strings that can be attached to either to cluster itself, |
192 | c71a1a3d | Iustin Pop | or to nodes or instances. They are useful as a very simplistic |
193 | c71a1a3d | Iustin Pop | information store for helping with cluster administration, for example |
194 | c71a1a3d | Iustin Pop | by attaching owner information to each instance after it's created:: |
195 | ffa6869f | Iustin Pop | |
196 | 73225861 | Iustin Pop | $ gnt-instance add โฆ %instance1% |
197 | 73225861 | Iustin Pop | $ gnt-instance add-tags %instance1% %owner:user2% |
198 | c71a1a3d | Iustin Pop | |
199 | c71a1a3d | Iustin Pop | And then by listing each instance and its tags, this information could |
200 | c71a1a3d | Iustin Pop | be used for contacting the users of each instance. |
201 | c71a1a3d | Iustin Pop | |
202 | c71a1a3d | Iustin Pop | Jobs and OpCodes |
203 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~ |
204 | c71a1a3d | Iustin Pop | |
205 | c71a1a3d | Iustin Pop | While not directly visible by an end-user, it's useful to know that a |
206 | c71a1a3d | Iustin Pop | basic cluster operation (e.g. starting an instance) is represented |
207 | 73225861 | Iustin Pop | internally by Ganeti as an *OpCode* (abbreviation from operation |
208 | c71a1a3d | Iustin Pop | code). These OpCodes are executed as part of a *Job*. The OpCodes in a |
209 | c71a1a3d | Iustin Pop | single Job are processed serially by Ganeti, but different Jobs will be |
210 | f313e7be | Michael Hanselmann | processed (depending on resource availability) in parallel. They will |
211 | f313e7be | Michael Hanselmann | not be executed in the submission order, but depending on resource |
212 | f313e7be | Michael Hanselmann | availability, locks and (starting with Ganeti 2.3) priority. An earlier |
213 | f313e7be | Michael Hanselmann | job may have to wait for a lock while a newer job doesn't need any locks |
214 | f313e7be | Michael Hanselmann | and can be executed right away. Operations requiring a certain order |
215 | f313e7be | Michael Hanselmann | need to be submitted as a single job, or the client must submit one job |
216 | f313e7be | Michael Hanselmann | at a time and wait for it to finish before continuing. |
217 | c71a1a3d | Iustin Pop | |
218 | c71a1a3d | Iustin Pop | For example, shutting down the entire cluster can be done by running the |
219 | c71a1a3d | Iustin Pop | command ``gnt-instance shutdown --all``, which will submit for each |
220 | c71a1a3d | Iustin Pop | instance a separate job containing the โshutdown instanceโ OpCode. |
221 | c71a1a3d | Iustin Pop | |
222 | c71a1a3d | Iustin Pop | |
223 | c71a1a3d | Iustin Pop | Prerequisites |
224 | c71a1a3d | Iustin Pop | +++++++++++++ |
225 | c71a1a3d | Iustin Pop | |
226 | c71a1a3d | Iustin Pop | You need to have your Ganeti cluster installed and configured before you |
227 | c71a1a3d | Iustin Pop | try any of the commands in this document. Please follow the |
228 | c71a1a3d | Iustin Pop | :doc:`install` for instructions on how to do that. |
229 | c71a1a3d | Iustin Pop | |
230 | c71a1a3d | Iustin Pop | Instance management |
231 | c71a1a3d | Iustin Pop | ------------------- |
232 | c71a1a3d | Iustin Pop | |
233 | c71a1a3d | Iustin Pop | Adding an instance |
234 | c71a1a3d | Iustin Pop | ++++++++++++++++++ |
235 | c71a1a3d | Iustin Pop | |
236 | c71a1a3d | Iustin Pop | The add operation might seem complex due to the many parameters it |
237 | c71a1a3d | Iustin Pop | accepts, but once you have understood the (few) required parameters and |
238 | c71a1a3d | Iustin Pop | the customisation capabilities you will see it is an easy operation. |
239 | c71a1a3d | Iustin Pop | |
240 | c71a1a3d | Iustin Pop | The add operation requires at minimum five parameters: |
241 | c71a1a3d | Iustin Pop | |
242 | c71a1a3d | Iustin Pop | - the OS for the instance |
243 | c71a1a3d | Iustin Pop | - the disk template |
244 | c71a1a3d | Iustin Pop | - the disk count and size |
245 | c71a1a3d | Iustin Pop | - the node specification or alternatively the iallocator to use |
246 | c71a1a3d | Iustin Pop | - and finally the instance name |
247 | c71a1a3d | Iustin Pop | |
248 | c71a1a3d | Iustin Pop | The OS for the instance must be visible in the output of the command |
249 | c71a1a3d | Iustin Pop | ``gnt-os list`` and specifies which guest OS to install on the instance. |
250 | c71a1a3d | Iustin Pop | |
251 | c71a1a3d | Iustin Pop | The disk template specifies what kind of storage to use as backend for |
252 | c71a1a3d | Iustin Pop | the (virtual) disks presented to the instance; note that for instances |
253 | c71a1a3d | Iustin Pop | with multiple virtual disks, they all must be of the same type. |
254 | c71a1a3d | Iustin Pop | |
255 | c71a1a3d | Iustin Pop | The node(s) on which the instance will run can be given either manually, |
256 | c71a1a3d | Iustin Pop | via the ``-n`` option, or computed automatically by Ganeti, if you have |
257 | c71a1a3d | Iustin Pop | installed any iallocator script. |
258 | c71a1a3d | Iustin Pop | |
259 | c71a1a3d | Iustin Pop | With the above parameters in mind, the command is:: |
260 | ffa6869f | Iustin Pop | |
261 | 73225861 | Iustin Pop | $ gnt-instance add \ |
262 | 73225861 | Iustin Pop | -n %TARGET_NODE%:%SECONDARY_NODE% \ |
263 | 73225861 | Iustin Pop | -o %OS_TYPE% \ |
264 | 73225861 | Iustin Pop | -t %DISK_TEMPLATE% -s %DISK_SIZE% \ |
265 | 73225861 | Iustin Pop | %INSTANCE_NAME% |
266 | ffa6869f | Iustin Pop | |
267 | ffa6869f | Iustin Pop | The instance name must be resolvable (e.g. exist in DNS) and usually |
268 | c71a1a3d | Iustin Pop | points to an address in the same subnet as the cluster itself. |
269 | ffa6869f | Iustin Pop | |
270 | c71a1a3d | Iustin Pop | The above command has the minimum required options; other options you |
271 | c71a1a3d | Iustin Pop | can give include, among others: |
272 | ffa6869f | Iustin Pop | |
273 | d4fcd298 | Guido Trotter | - The maximum/minimum memory size (``-B maxmem``, ``-B minmem``) |
274 | d4fcd298 | Guido Trotter | (``-B memory`` can be used to specify only one size) |
275 | ffa6869f | Iustin Pop | |
276 | ffa6869f | Iustin Pop | - The number of virtual CPUs (``-B vcpus``) |
277 | ffa6869f | Iustin Pop | |
278 | ffa6869f | Iustin Pop | - Arguments for the NICs of the instance; by default, a single-NIC |
279 | ffa6869f | Iustin Pop | instance is created. The IP and/or bridge of the NIC can be changed |
280 | 0da22bc3 | Sebastian Gebhard | via ``--net 0:ip=IP,link=BRIDGE`` |
281 | ffa6869f | Iustin Pop | |
282 | da535d0d | Guido Trotter | See :manpage:`ganeti-instance(8)` for the detailed option list. |
283 | ffa6869f | Iustin Pop | |
284 | c71a1a3d | Iustin Pop | For example if you want to create an highly available instance, with a |
285 | c71a1a3d | Iustin Pop | single disk of 50GB and the default memory size, having primary node |
286 | c71a1a3d | Iustin Pop | ``node1`` and secondary node ``node3``, use the following command:: |
287 | ffa6869f | Iustin Pop | |
288 | 73225861 | Iustin Pop | $ gnt-instance add -n node1:node3 -o debootstrap -t drbd -s 50G \ |
289 | c71a1a3d | Iustin Pop | instance1 |
290 | ffa6869f | Iustin Pop | |
291 | c71a1a3d | Iustin Pop | There is a also a command for batch instance creation from a |
292 | c71a1a3d | Iustin Pop | specification file, see the ``batch-create`` operation in the |
293 | c71a1a3d | Iustin Pop | gnt-instance manual page. |
294 | ffa6869f | Iustin Pop | |
295 | c71a1a3d | Iustin Pop | Regular instance operations |
296 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++++ |
297 | ffa6869f | Iustin Pop | |
298 | c71a1a3d | Iustin Pop | Removal |
299 | c71a1a3d | Iustin Pop | ~~~~~~~ |
300 | ffa6869f | Iustin Pop | |
301 | c71a1a3d | Iustin Pop | Removing an instance is even easier than creating one. This operation is |
302 | c71a1a3d | Iustin Pop | irreversible and destroys all the contents of your instance. Use with |
303 | c71a1a3d | Iustin Pop | care:: |
304 | ffa6869f | Iustin Pop | |
305 | 73225861 | Iustin Pop | $ gnt-instance remove %INSTANCE_NAME% |
306 | ffa6869f | Iustin Pop | |
307 | d4fcd298 | Guido Trotter | .. _instance-startup-label: |
308 | d4fcd298 | Guido Trotter | |
309 | c71a1a3d | Iustin Pop | Startup/shutdown |
310 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~ |
311 | ffa6869f | Iustin Pop | |
312 | ffa6869f | Iustin Pop | Instances are automatically started at instance creation time. To |
313 | ffa6869f | Iustin Pop | manually start one which is currently stopped you can run:: |
314 | ffa6869f | Iustin Pop | |
315 | 73225861 | Iustin Pop | $ gnt-instance startup %INSTANCE_NAME% |
316 | ffa6869f | Iustin Pop | |
317 | d4fcd298 | Guido Trotter | Ganeti will start an instance with up to its maximum instance memory. If |
318 | d4fcd298 | Guido Trotter | not enough memory is available Ganeti will use all the available memory |
319 | 7e2f08ba | Helga Velroyen | down to the instance minimum memory. If not even that amount of memory |
320 | d4fcd298 | Guido Trotter | is free Ganeti will refuse to start the instance. |
321 | d4fcd298 | Guido Trotter | |
322 | edc282ad | Agata Murawska | Note, that this will not work when an instance is in a permanently |
323 | edc282ad | Agata Murawska | stopped state ``offline``. In this case, you will first have to |
324 | edc282ad | Agata Murawska | put it back to online mode by running:: |
325 | edc282ad | Agata Murawska | |
326 | 73225861 | Iustin Pop | $ gnt-instance modify --online %INSTANCE_NAME% |
327 | edc282ad | Agata Murawska | |
328 | edc282ad | Agata Murawska | The command to stop the running instance is:: |
329 | ffa6869f | Iustin Pop | |
330 | 73225861 | Iustin Pop | $ gnt-instance shutdown %INSTANCE_NAME% |
331 | ffa6869f | Iustin Pop | |
332 | edc282ad | Agata Murawska | If you want to shut the instance down more permanently, so that it |
333 | edc282ad | Agata Murawska | does not require dynamically allocated resources (memory and vcpus), |
334 | edc282ad | Agata Murawska | after shutting down an instance, execute the following:: |
335 | edc282ad | Agata Murawska | |
336 | 73225861 | Iustin Pop | $ gnt-instance modify --offline %INSTANCE_NAME% |
337 | edc282ad | Agata Murawska | |
338 | c71a1a3d | Iustin Pop | .. warning:: Do not use the Xen or KVM commands directly to stop |
339 | c71a1a3d | Iustin Pop | instances. If you run for example ``xm shutdown`` or ``xm destroy`` |
340 | 44275c2d | Andrea Spadaccini | on an instance Ganeti will automatically restart it (via |
341 | 22ac4136 | Michael Hanselmann | the :command:`ganeti-watcher(8)` command which is launched via cron). |
342 | c71a1a3d | Iustin Pop | |
343 | c71a1a3d | Iustin Pop | Querying instances |
344 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~ |
345 | c71a1a3d | Iustin Pop | |
346 | c71a1a3d | Iustin Pop | There are two ways to get information about instances: listing |
347 | c71a1a3d | Iustin Pop | instances, which does a tabular output containing a given set of fields |
348 | c71a1a3d | Iustin Pop | about each instance, and querying detailed information about a set of |
349 | c71a1a3d | Iustin Pop | instances. |
350 | c71a1a3d | Iustin Pop | |
351 | ffa6869f | Iustin Pop | The command to see all the instances configured and their status is:: |
352 | ffa6869f | Iustin Pop | |
353 | 73225861 | Iustin Pop | $ gnt-instance list |
354 | ffa6869f | Iustin Pop | |
355 | c71a1a3d | Iustin Pop | The command can return a custom set of information when using the ``-o`` |
356 | c71a1a3d | Iustin Pop | option (as always, check the manpage for a detailed specification). Each |
357 | c71a1a3d | Iustin Pop | instance will be represented on a line, thus making it easy to parse |
358 | c71a1a3d | Iustin Pop | this output via the usual shell utilities (grep, sed, etc.). |
359 | c71a1a3d | Iustin Pop | |
360 | c71a1a3d | Iustin Pop | To get more detailed information about an instance, you can run:: |
361 | c71a1a3d | Iustin Pop | |
362 | 73225861 | Iustin Pop | $ gnt-instance info %INSTANCE% |
363 | c71a1a3d | Iustin Pop | |
364 | c71a1a3d | Iustin Pop | which will give a multi-line block of information about the instance, |
365 | c71a1a3d | Iustin Pop | it's hardware resources (especially its disks and their redundancy |
366 | c71a1a3d | Iustin Pop | status), etc. This is harder to parse and is more expensive than the |
367 | c71a1a3d | Iustin Pop | list operation, but returns much more detailed information. |
368 | ffa6869f | Iustin Pop | |
369 | d4fcd298 | Guido Trotter | Changing an instance's runtime memory |
370 | d4fcd298 | Guido Trotter | +++++++++++++++++++++++++++++++++++++ |
371 | d4fcd298 | Guido Trotter | |
372 | d4fcd298 | Guido Trotter | Ganeti will always make sure an instance has a value between its maximum |
373 | d4fcd298 | Guido Trotter | and its minimum memory available as runtime memory. As of version 2.6 |
374 | d4fcd298 | Guido Trotter | Ganeti will only choose a size different than the maximum size when |
375 | d4fcd298 | Guido Trotter | starting up, failing over, or migrating an instance on a node with less |
376 | d4fcd298 | Guido Trotter | than the maximum memory available. It won't resize other instances in |
377 | d4fcd298 | Guido Trotter | order to free up space for an instance. |
378 | d4fcd298 | Guido Trotter | |
379 | d4fcd298 | Guido Trotter | If you find that you need more memory on a node any instance can be |
380 | d4fcd298 | Guido Trotter | manually resized without downtime, with the command:: |
381 | d4fcd298 | Guido Trotter | |
382 | 73225861 | Iustin Pop | $ gnt-instance modify -m %SIZE% %INSTANCE_NAME% |
383 | d4fcd298 | Guido Trotter | |
384 | d4fcd298 | Guido Trotter | The same command can also be used to increase the memory available on an |
385 | d4fcd298 | Guido Trotter | instance, provided that enough free memory is available on its node, and |
386 | d4fcd298 | Guido Trotter | the specified size is not larger than the maximum memory size the |
387 | d4fcd298 | Guido Trotter | instance had when it was first booted (an instance will be unable to see |
388 | d4fcd298 | Guido Trotter | new memory above the maximum that was specified to the hypervisor at its |
389 | d4fcd298 | Guido Trotter | boot time, if it needs to grow further a reboot becomes necessary). |
390 | ffa6869f | Iustin Pop | |
391 | c71a1a3d | Iustin Pop | Export/Import |
392 | c71a1a3d | Iustin Pop | +++++++++++++ |
393 | c71a1a3d | Iustin Pop | |
394 | c71a1a3d | Iustin Pop | You can create a snapshot of an instance disk and its Ganeti |
395 | ffa6869f | Iustin Pop | configuration, which then you can backup, or import into another |
396 | ffa6869f | Iustin Pop | cluster. The way to export an instance is:: |
397 | ffa6869f | Iustin Pop | |
398 | 73225861 | Iustin Pop | $ gnt-backup export -n %TARGET_NODE% %INSTANCE_NAME% |
399 | ffa6869f | Iustin Pop | |
400 | c71a1a3d | Iustin Pop | |
401 | ffa6869f | Iustin Pop | The target node can be any node in the cluster with enough space under |
402 | c71a1a3d | Iustin Pop | ``/srv/ganeti`` to hold the instance image. Use the ``--noshutdown`` |
403 | c71a1a3d | Iustin Pop | option to snapshot an instance without rebooting it. Note that Ganeti |
404 | c71a1a3d | Iustin Pop | only keeps one snapshot for an instance - any previous snapshot of the |
405 | c71a1a3d | Iustin Pop | same instance existing cluster-wide under ``/srv/ganeti`` will be |
406 | c71a1a3d | Iustin Pop | removed by this operation: if you want to keep them, you need to move |
407 | c71a1a3d | Iustin Pop | them out of the Ganeti exports directory. |
408 | ffa6869f | Iustin Pop | |
409 | c71a1a3d | Iustin Pop | Importing an instance is similar to creating a new one, but additionally |
410 | c71a1a3d | Iustin Pop | one must specify the location of the snapshot. The command is:: |
411 | ffa6869f | Iustin Pop | |
412 | 73225861 | Iustin Pop | $ gnt-backup import -n %TARGET_NODE% \ |
413 | 73225861 | Iustin Pop | --src-node=%NODE% --src-dir=%DIR% %INSTANCE_NAME% |
414 | ffa6869f | Iustin Pop | |
415 | 33ea43b6 | Iustin Pop | By default, parameters will be read from the export information, but you |
416 | 33ea43b6 | Iustin Pop | can of course pass them in via the command line - most of the options |
417 | 33ea43b6 | Iustin Pop | available for the command :command:`gnt-instance add` are supported here |
418 | 33ea43b6 | Iustin Pop | too. |
419 | ffa6869f | Iustin Pop | |
420 | 17227cd1 | Iustin Pop | Import of foreign instances |
421 | 17227cd1 | Iustin Pop | +++++++++++++++++++++++++++ |
422 | 17227cd1 | Iustin Pop | |
423 | 17227cd1 | Iustin Pop | There is a possibility to import a foreign instance whose disk data is |
424 | 17227cd1 | Iustin Pop | already stored as LVM volumes without going through copying it: the disk |
425 | 17227cd1 | Iustin Pop | adoption mode. |
426 | 17227cd1 | Iustin Pop | |
427 | 17227cd1 | Iustin Pop | For this, ensure that the original, non-managed instance is stopped, |
428 | 17227cd1 | Iustin Pop | then create a Ganeti instance in the usual way, except that instead of |
429 | 17227cd1 | Iustin Pop | passing the disk information you specify the current volumes:: |
430 | 17227cd1 | Iustin Pop | |
431 | 73225861 | Iustin Pop | $ gnt-instance add -t plain -n %HOME_NODE% ... \ |
432 | 73225861 | Iustin Pop | --disk 0:adopt=%lv_name%[,vg=%vg_name%] %INSTANCE_NAME% |
433 | 17227cd1 | Iustin Pop | |
434 | 17227cd1 | Iustin Pop | This will take over the given logical volumes, rename them to the Ganeti |
435 | 17227cd1 | Iustin Pop | standard (UUID-based), and without installing the OS on them start |
436 | 17227cd1 | Iustin Pop | directly the instance. If you configure the hypervisor similar to the |
437 | 17227cd1 | Iustin Pop | non-managed configuration that the instance had, the transition should |
438 | 17227cd1 | Iustin Pop | be seamless for the instance. For more than one disk, just pass another |
439 | 17227cd1 | Iustin Pop | disk parameter (e.g. ``--disk 1:adopt=...``). |
440 | 17227cd1 | Iustin Pop | |
441 | fc3914e6 | Iustin Pop | Instance kernel selection |
442 | fc3914e6 | Iustin Pop | +++++++++++++++++++++++++ |
443 | fc3914e6 | Iustin Pop | |
444 | fc3914e6 | Iustin Pop | The kernel that instances uses to bootup can come either from the node, |
445 | fc3914e6 | Iustin Pop | or from instances themselves, depending on the setup. |
446 | fc3914e6 | Iustin Pop | |
447 | fc3914e6 | Iustin Pop | Xen-PVM |
448 | fc3914e6 | Iustin Pop | ~~~~~~~ |
449 | fc3914e6 | Iustin Pop | |
450 | fc3914e6 | Iustin Pop | With Xen PVM, there are three options. |
451 | fc3914e6 | Iustin Pop | |
452 | fc3914e6 | Iustin Pop | First, you can use a kernel from the node, by setting the hypervisor |
453 | fc3914e6 | Iustin Pop | parameters as such: |
454 | fc3914e6 | Iustin Pop | |
455 | fc3914e6 | Iustin Pop | - ``kernel_path`` to a valid file on the node (and appropriately |
456 | fc3914e6 | Iustin Pop | ``initrd_path``) |
457 | fc3914e6 | Iustin Pop | - ``kernel_args`` optionally set to a valid Linux setting (e.g. ``ro``) |
458 | fc3914e6 | Iustin Pop | - ``root_path`` to a valid setting (e.g. ``/dev/xvda1``) |
459 | fc3914e6 | Iustin Pop | - ``bootloader_path`` and ``bootloader_args`` to empty |
460 | fc3914e6 | Iustin Pop | |
461 | 96514751 | Michael Hanselmann | Alternatively, you can delegate the kernel management to instances, and |
462 | fc3914e6 | Iustin Pop | use either ``pvgrub`` or the deprecated ``pygrub``. For this, you must |
463 | 96514751 | Michael Hanselmann | install the kernels and initrds in the instance and create a valid GRUB |
464 | fc3914e6 | Iustin Pop | v1 configuration file. |
465 | fc3914e6 | Iustin Pop | |
466 | fc3914e6 | Iustin Pop | For ``pvgrub`` (new in version 2.4.2), you need to set: |
467 | fc3914e6 | Iustin Pop | |
468 | fc3914e6 | Iustin Pop | - ``kernel_path`` to point to the ``pvgrub`` loader present on the node |
469 | fc3914e6 | Iustin Pop | (e.g. ``/usr/lib/xen/boot/pv-grub-x86_32.gz``) |
470 | 96514751 | Michael Hanselmann | - ``kernel_args`` to the path to the GRUB config file, relative to the |
471 | fc3914e6 | Iustin Pop | instance (e.g. ``(hd0,0)/grub/menu.lst``) |
472 | fc3914e6 | Iustin Pop | - ``root_path`` **must** be empty |
473 | fc3914e6 | Iustin Pop | - ``bootloader_path`` and ``bootloader_args`` to empty |
474 | fc3914e6 | Iustin Pop | |
475 | fc3914e6 | Iustin Pop | While ``pygrub`` is deprecated, here is how you can configure it: |
476 | fc3914e6 | Iustin Pop | |
477 | fc3914e6 | Iustin Pop | - ``bootloader_path`` to the pygrub binary (e.g. ``/usr/bin/pygrub``) |
478 | fc3914e6 | Iustin Pop | - the other settings are not important |
479 | fc3914e6 | Iustin Pop | |
480 | fc3914e6 | Iustin Pop | More information can be found in the Xen wiki pages for `pvgrub |
481 | fc3914e6 | Iustin Pop | <http://wiki.xensource.com/xenwiki/PvGrub>`_ and `pygrub |
482 | fc3914e6 | Iustin Pop | <http://wiki.xensource.com/xenwiki/PyGrub>`_. |
483 | fc3914e6 | Iustin Pop | |
484 | fc3914e6 | Iustin Pop | KVM |
485 | fc3914e6 | Iustin Pop | ~~~ |
486 | fc3914e6 | Iustin Pop | |
487 | fc3914e6 | Iustin Pop | For KVM also the kernel can be loaded either way. |
488 | fc3914e6 | Iustin Pop | |
489 | fc3914e6 | Iustin Pop | For loading the kernels from the node, you need to set: |
490 | fc3914e6 | Iustin Pop | |
491 | fc3914e6 | Iustin Pop | - ``kernel_path`` to a valid value |
492 | fc3914e6 | Iustin Pop | - ``initrd_path`` optionally set if you use an initrd |
493 | fc3914e6 | Iustin Pop | - ``kernel_args`` optionally set to a valid value (e.g. ``ro``) |
494 | fc3914e6 | Iustin Pop | |
495 | fc3914e6 | Iustin Pop | If you want instead to have the instance boot from its disk (and execute |
496 | fc3914e6 | Iustin Pop | its bootloader), simply set the ``kernel_path`` parameter to an empty |
497 | fc3914e6 | Iustin Pop | string, and all the others will be ignored. |
498 | fc3914e6 | Iustin Pop | |
499 | c71a1a3d | Iustin Pop | Instance HA features |
500 | c71a1a3d | Iustin Pop | -------------------- |
501 | ffa6869f | Iustin Pop | |
502 | ffa6869f | Iustin Pop | .. note:: This section only applies to multi-node clusters |
503 | ffa6869f | Iustin Pop | |
504 | c71a1a3d | Iustin Pop | .. _instance-change-primary-label: |
505 | c71a1a3d | Iustin Pop | |
506 | c71a1a3d | Iustin Pop | Changing the primary node |
507 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++ |
508 | c71a1a3d | Iustin Pop | |
509 | c71a1a3d | Iustin Pop | There are three ways to exchange an instance's primary and secondary |
510 | c71a1a3d | Iustin Pop | nodes; the right one to choose depends on how the instance has been |
511 | c71a1a3d | Iustin Pop | created and the status of its current primary node. See |
512 | c71a1a3d | Iustin Pop | :ref:`rest-redundancy-label` for information on changing the secondary |
513 | c71a1a3d | Iustin Pop | node. Note that it's only possible to change the primary node to the |
514 | c71a1a3d | Iustin Pop | secondary and vice-versa; a direct change of the primary node with a |
515 | c71a1a3d | Iustin Pop | third node, while keeping the current secondary is not possible in a |
516 | c71a1a3d | Iustin Pop | single step, only via multiple operations as detailed in |
517 | c71a1a3d | Iustin Pop | :ref:`instance-relocation-label`. |
518 | c71a1a3d | Iustin Pop | |
519 | ffa6869f | Iustin Pop | Failing over an instance |
520 | ffa6869f | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~ |
521 | ffa6869f | Iustin Pop | |
522 | ffa6869f | Iustin Pop | If an instance is built in highly available mode you can at any time |
523 | ffa6869f | Iustin Pop | fail it over to its secondary node, even if the primary has somehow |
524 | ffa6869f | Iustin Pop | failed and it's not up anymore. Doing it is really easy, on the master |
525 | ffa6869f | Iustin Pop | node you can just run:: |
526 | ffa6869f | Iustin Pop | |
527 | 73225861 | Iustin Pop | $ gnt-instance failover %INSTANCE_NAME% |
528 | ffa6869f | Iustin Pop | |
529 | ffa6869f | Iustin Pop | That's it. After the command completes the secondary node is now the |
530 | c71a1a3d | Iustin Pop | primary, and vice-versa. |
531 | ffa6869f | Iustin Pop | |
532 | d4fcd298 | Guido Trotter | The instance will be started with an amount of memory between its |
533 | d4fcd298 | Guido Trotter | ``maxmem`` and its ``minmem`` value, depending on the free memory on its |
534 | d4fcd298 | Guido Trotter | target node, or the operation will fail if that's not possible. See |
535 | d4fcd298 | Guido Trotter | :ref:`instance-startup-label` for details. |
536 | d4fcd298 | Guido Trotter | |
537 | 7ed400f0 | Stratos Psomadakis | If the instance's disk template is of type rbd, then you can specify |
538 | 7ed400f0 | Stratos Psomadakis | the target node (which can be any node) explicitly, or specify an |
539 | 7ed400f0 | Stratos Psomadakis | iallocator plugin. If you omit both, the default iallocator will be |
540 | 7ed400f0 | Stratos Psomadakis | used to determine the target node:: |
541 | 7ed400f0 | Stratos Psomadakis | |
542 | 73225861 | Iustin Pop | $ gnt-instance failover -n %TARGET_NODE% %INSTANCE_NAME% |
543 | 7ed400f0 | Stratos Psomadakis | |
544 | ffa6869f | Iustin Pop | Live migrating an instance |
545 | ffa6869f | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
546 | ffa6869f | Iustin Pop | |
547 | c71a1a3d | Iustin Pop | If an instance is built in highly available mode, it currently runs and |
548 | 7e2f08ba | Helga Velroyen | both its nodes are running fine, you can migrate it over to its |
549 | c71a1a3d | Iustin Pop | secondary node, without downtime. On the master node you need to run:: |
550 | ffa6869f | Iustin Pop | |
551 | 73225861 | Iustin Pop | $ gnt-instance migrate %INSTANCE_NAME% |
552 | ffa6869f | Iustin Pop | |
553 | c71a1a3d | Iustin Pop | The current load on the instance and its memory size will influence how |
554 | c71a1a3d | Iustin Pop | long the migration will take. In any case, for both KVM and Xen |
555 | c71a1a3d | Iustin Pop | hypervisors, the migration will be transparent to the instance. |
556 | ffa6869f | Iustin Pop | |
557 | d4fcd298 | Guido Trotter | If the destination node has less memory than the instance's current |
558 | d4fcd298 | Guido Trotter | runtime memory, but at least the instance's minimum memory available |
559 | d4fcd298 | Guido Trotter | Ganeti will automatically reduce the instance runtime memory before |
560 | d4fcd298 | Guido Trotter | migrating it, unless the ``--no-runtime-changes`` option is passed, in |
561 | d4fcd298 | Guido Trotter | which case the target node should have at least the instance's current |
562 | d4fcd298 | Guido Trotter | runtime memory free. |
563 | d4fcd298 | Guido Trotter | |
564 | 7ed400f0 | Stratos Psomadakis | If the instance's disk template is of type rbd, then you can specify |
565 | 7ed400f0 | Stratos Psomadakis | the target node (which can be any node) explicitly, or specify an |
566 | 7ed400f0 | Stratos Psomadakis | iallocator plugin. If you omit both, the default iallocator will be |
567 | 7ed400f0 | Stratos Psomadakis | used to determine the target node:: |
568 | 7ed400f0 | Stratos Psomadakis | |
569 | 73225861 | Iustin Pop | $ gnt-instance migrate -n %TARGET_NODE% %INSTANCE_NAME% |
570 | 7ed400f0 | Stratos Psomadakis | |
571 | c71a1a3d | Iustin Pop | Moving an instance (offline) |
572 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
573 | ffa6869f | Iustin Pop | |
574 | c71a1a3d | Iustin Pop | If an instance has not been create as mirrored, then the only way to |
575 | c71a1a3d | Iustin Pop | change its primary node is to execute the move command:: |
576 | ffa6869f | Iustin Pop | |
577 | 73225861 | Iustin Pop | $ gnt-instance move -n %NEW_NODE% %INSTANCE% |
578 | ffa6869f | Iustin Pop | |
579 | c71a1a3d | Iustin Pop | This has a few prerequisites: |
580 | ffa6869f | Iustin Pop | |
581 | c71a1a3d | Iustin Pop | - the instance must be stopped |
582 | c71a1a3d | Iustin Pop | - its current primary node must be on-line and healthy |
583 | c71a1a3d | Iustin Pop | - the disks of the instance must not have any errors |
584 | ffa6869f | Iustin Pop | |
585 | c71a1a3d | Iustin Pop | Since this operation actually copies the data from the old node to the |
586 | c71a1a3d | Iustin Pop | new node, expect it to take proportional to the size of the instance's |
587 | c71a1a3d | Iustin Pop | disks and the speed of both the nodes' I/O system and their networking. |
588 | ffa6869f | Iustin Pop | |
589 | c71a1a3d | Iustin Pop | Disk operations |
590 | c71a1a3d | Iustin Pop | +++++++++++++++ |
591 | ffa6869f | Iustin Pop | |
592 | c71a1a3d | Iustin Pop | Disk failures are a common cause of errors in any server |
593 | c71a1a3d | Iustin Pop | deployment. Ganeti offers protection from single-node failure if your |
594 | c71a1a3d | Iustin Pop | instances were created in HA mode, and it also offers ways to restore |
595 | c71a1a3d | Iustin Pop | redundancy after a failure. |
596 | ffa6869f | Iustin Pop | |
597 | c71a1a3d | Iustin Pop | Preparing for disk operations |
598 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
599 | ffa6869f | Iustin Pop | |
600 | c71a1a3d | Iustin Pop | It is important to note that for Ganeti to be able to do any disk |
601 | 5e360222 | Michele Tartara | operation, the Linux machines on top of which Ganeti runs must be |
602 | 5e360222 | Michele Tartara | consistent; for LVM, this means that the LVM commands must not return |
603 | 5e360222 | Michele Tartara | failures; it is common that after a complete disk failure, any LVM |
604 | 5e360222 | Michele Tartara | command aborts with an error similar to:: |
605 | ffa6869f | Iustin Pop | |
606 | 73225861 | Iustin Pop | $ vgs |
607 | c71a1a3d | Iustin Pop | /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error |
608 | 73225861 | Iustin Pop | /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output error |
609 | c71a1a3d | Iustin Pop | /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error |
610 | 73225861 | Iustin Pop | Couldn't find device with uuid 't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'. |
611 | c71a1a3d | Iustin Pop | Couldn't find all physical volumes for volume group xenvg. |
612 | ffa6869f | Iustin Pop | |
613 | c71a1a3d | Iustin Pop | Before restoring an instance's disks to healthy status, it's needed to |
614 | c71a1a3d | Iustin Pop | fix the volume group used by Ganeti so that we can actually create and |
615 | c71a1a3d | Iustin Pop | manage the logical volumes. This is usually done in a multi-step |
616 | c71a1a3d | Iustin Pop | process: |
617 | ffa6869f | Iustin Pop | |
618 | c71a1a3d | Iustin Pop | #. first, if the disk is completely gone and LVM commands exit with |
619 | c71a1a3d | Iustin Pop | โCouldn't find device with uuidโฆโ then you need to run the command:: |
620 | c71a1a3d | Iustin Pop | |
621 | 73225861 | Iustin Pop | $ vgreduce --removemissing %VOLUME_GROUP% |
622 | c71a1a3d | Iustin Pop | |
623 | c71a1a3d | Iustin Pop | #. after the above command, the LVM commands should be executing |
624 | c71a1a3d | Iustin Pop | normally (warnings are normal, but the commands will not fail |
625 | c71a1a3d | Iustin Pop | completely). |
626 | c71a1a3d | Iustin Pop | |
627 | c71a1a3d | Iustin Pop | #. if the failed disk is still visible in the output of the ``pvs`` |
628 | c71a1a3d | Iustin Pop | command, you need to deactivate it from allocations by running:: |
629 | c71a1a3d | Iustin Pop | |
630 | 73225861 | Iustin Pop | $ pvs -x n /dev/%DISK% |
631 | ffa6869f | Iustin Pop | |
632 | c71a1a3d | Iustin Pop | At this point, the volume group should be consistent and any bad |
633 | c71a1a3d | Iustin Pop | physical volumes should not longer be available for allocation. |
634 | c71a1a3d | Iustin Pop | |
635 | c71a1a3d | Iustin Pop | Note that since version 2.1 Ganeti provides some commands to automate |
636 | c71a1a3d | Iustin Pop | these two operations, see :ref:`storage-units-label`. |
637 | c71a1a3d | Iustin Pop | |
638 | c71a1a3d | Iustin Pop | .. _rest-redundancy-label: |
639 | c71a1a3d | Iustin Pop | |
640 | c71a1a3d | Iustin Pop | Restoring redundancy for DRBD-based instances |
641 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
642 | c71a1a3d | Iustin Pop | |
643 | c71a1a3d | Iustin Pop | A DRBD instance has two nodes, and the storage on one of them has |
644 | c71a1a3d | Iustin Pop | failed. Depending on which node (primary or secondary) has failed, you |
645 | c71a1a3d | Iustin Pop | have three options at hand: |
646 | c71a1a3d | Iustin Pop | |
647 | c71a1a3d | Iustin Pop | - if the storage on the primary node has failed, you need to re-create |
648 | c71a1a3d | Iustin Pop | the disks on it |
649 | c71a1a3d | Iustin Pop | - if the storage on the secondary node has failed, you can either |
650 | c71a1a3d | Iustin Pop | re-create the disks on it or change the secondary and recreate |
651 | c71a1a3d | Iustin Pop | redundancy on the new secondary node |
652 | c71a1a3d | Iustin Pop | |
653 | c71a1a3d | Iustin Pop | Of course, at any point it's possible to force re-creation of disks even |
654 | c71a1a3d | Iustin Pop | though everything is already fine. |
655 | c71a1a3d | Iustin Pop | |
656 | c71a1a3d | Iustin Pop | For all three cases, the ``replace-disks`` operation can be used:: |
657 | c71a1a3d | Iustin Pop | |
658 | c71a1a3d | Iustin Pop | # re-create disks on the primary node |
659 | 73225861 | Iustin Pop | $ gnt-instance replace-disks -p %INSTANCE_NAME% |
660 | c71a1a3d | Iustin Pop | # re-create disks on the current secondary |
661 | 73225861 | Iustin Pop | $ gnt-instance replace-disks -s %INSTANCE_NAME% |
662 | c71a1a3d | Iustin Pop | # change the secondary node, via manual specification |
663 | 73225861 | Iustin Pop | $ gnt-instance replace-disks -n %NODE% %INSTANCE_NAME% |
664 | c71a1a3d | Iustin Pop | # change the secondary node, via an iallocator script |
665 | 73225861 | Iustin Pop | $ gnt-instance replace-disks -I %SCRIPT% %INSTANCE_NAME% |
666 | c71a1a3d | Iustin Pop | # since Ganeti 2.1: automatically fix the primary or secondary node |
667 | 73225861 | Iustin Pop | $ gnt-instance replace-disks -a %INSTANCE_NAME% |
668 | c71a1a3d | Iustin Pop | |
669 | c71a1a3d | Iustin Pop | Since the process involves copying all data from the working node to the |
670 | c71a1a3d | Iustin Pop | target node, it will take a while, depending on the instance's disk |
671 | 1cdc9dbb | Bernardo Dal Seno | size, node I/O system and network speed. But it is (barring any network |
672 | c71a1a3d | Iustin Pop | interruption) completely transparent for the instance. |
673 | c71a1a3d | Iustin Pop | |
674 | c71a1a3d | Iustin Pop | Re-creating disks for non-redundant instances |
675 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
676 | c71a1a3d | Iustin Pop | |
677 | c71a1a3d | Iustin Pop | .. versionadded:: 2.1 |
678 | c71a1a3d | Iustin Pop | |
679 | c71a1a3d | Iustin Pop | For non-redundant instances, there isn't a copy (except backups) to |
680 | c71a1a3d | Iustin Pop | re-create the disks. But it's possible to at-least re-create empty |
681 | c71a1a3d | Iustin Pop | disks, after which a reinstall can be run, via the ``recreate-disks`` |
682 | c71a1a3d | Iustin Pop | command:: |
683 | c71a1a3d | Iustin Pop | |
684 | 73225861 | Iustin Pop | $ gnt-instance recreate-disks %INSTANCE% |
685 | c71a1a3d | Iustin Pop | |
686 | 38db4e7c | Adam Ingrassia | Note that this will fail if the disks already exists. The instance can |
687 | 38db4e7c | Adam Ingrassia | be assigned to new nodes automatically by specifying an iallocator |
688 | 38db4e7c | Adam Ingrassia | through the ``--iallocator`` option. |
689 | c71a1a3d | Iustin Pop | |
690 | bbf74a76 | Iustin Pop | Conversion of an instance's disk type |
691 | bbf74a76 | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
692 | bbf74a76 | Iustin Pop | |
693 | bbf74a76 | Iustin Pop | It is possible to convert between a non-redundant instance of type |
694 | bbf74a76 | Iustin Pop | ``plain`` (LVM storage) and redundant ``drbd`` via the ``gnt-instance |
695 | bbf74a76 | Iustin Pop | modify`` command:: |
696 | bbf74a76 | Iustin Pop | |
697 | bbf74a76 | Iustin Pop | # start with a non-redundant instance |
698 | 73225861 | Iustin Pop | $ gnt-instance add -t plain ... %INSTANCE% |
699 | bbf74a76 | Iustin Pop | |
700 | bbf74a76 | Iustin Pop | # later convert it to redundant |
701 | 73225861 | Iustin Pop | $ gnt-instance stop %INSTANCE% |
702 | 73225861 | Iustin Pop | $ gnt-instance modify -t drbd -n %NEW_SECONDARY% %INSTANCE% |
703 | 73225861 | Iustin Pop | $ gnt-instance start %INSTANCE% |
704 | bbf74a76 | Iustin Pop | |
705 | bbf74a76 | Iustin Pop | # and convert it back |
706 | 73225861 | Iustin Pop | $ gnt-instance stop %INSTANCE% |
707 | 73225861 | Iustin Pop | $ gnt-instance modify -t plain %INSTANCE% |
708 | 73225861 | Iustin Pop | $ gnt-instance start %INSTANCE% |
709 | bbf74a76 | Iustin Pop | |
710 | bbf74a76 | Iustin Pop | The conversion must be done while the instance is stopped, and |
711 | bbf74a76 | Iustin Pop | converting from plain to drbd template presents a small risk, especially |
712 | bbf74a76 | Iustin Pop | if the instance has multiple disks and/or if one node fails during the |
713 | bbf74a76 | Iustin Pop | conversion procedure). As such, it's recommended (as always) to make |
714 | bbf74a76 | Iustin Pop | sure that downtime for manual recovery is acceptable and that the |
715 | bbf74a76 | Iustin Pop | instance has up-to-date backups. |
716 | bbf74a76 | Iustin Pop | |
717 | c71a1a3d | Iustin Pop | Debugging instances |
718 | c71a1a3d | Iustin Pop | +++++++++++++++++++ |
719 | ffa6869f | Iustin Pop | |
720 | ffa6869f | Iustin Pop | Accessing an instance's disks |
721 | ffa6869f | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
722 | ffa6869f | Iustin Pop | |
723 | c71a1a3d | Iustin Pop | From an instance's primary node you can have access to its disks. Never |
724 | ffa6869f | Iustin Pop | ever mount the underlying logical volume manually on a fault tolerant |
725 | c71a1a3d | Iustin Pop | instance, or will break replication and your data will be |
726 | c71a1a3d | Iustin Pop | inconsistent. The correct way to access an instance's disks is to run |
727 | c71a1a3d | Iustin Pop | (on the master node, as usual) the command:: |
728 | c71a1a3d | Iustin Pop | |
729 | 73225861 | Iustin Pop | $ gnt-instance activate-disks %INSTANCE% |
730 | c71a1a3d | Iustin Pop | |
731 | c71a1a3d | Iustin Pop | And then, *on the primary node of the instance*, access the device that |
732 | c71a1a3d | Iustin Pop | gets created. For example, you could mount the given disks, then edit |
733 | c71a1a3d | Iustin Pop | files on the filesystem, etc. |
734 | c71a1a3d | Iustin Pop | |
735 | c71a1a3d | Iustin Pop | Note that with partitioned disks (as opposed to whole-disk filesystems), |
736 | c71a1a3d | Iustin Pop | you will need to use a tool like :manpage:`kpartx(8)`:: |
737 | ffa6869f | Iustin Pop | |
738 | 73225861 | Iustin Pop | # on node1 |
739 | 73225861 | Iustin Pop | $ gnt-instance activate-disks %instance1% |
740 | 73225861 | Iustin Pop | node3:disk/0:โฆ |
741 | 73225861 | Iustin Pop | $ ssh node3 |
742 | 73225861 | Iustin Pop | # on node 3 |
743 | 73225861 | Iustin Pop | $ kpartx -l /dev/โฆ |
744 | 73225861 | Iustin Pop | $ kpartx -a /dev/โฆ |
745 | 73225861 | Iustin Pop | $ mount /dev/mapper/โฆ /mnt/ |
746 | c71a1a3d | Iustin Pop | # edit files under mnt as desired |
747 | 73225861 | Iustin Pop | $ umount /mnt/ |
748 | 73225861 | Iustin Pop | $ kpartx -d /dev/โฆ |
749 | 73225861 | Iustin Pop | $ exit |
750 | 73225861 | Iustin Pop | # back to node 1 |
751 | ffa6869f | Iustin Pop | |
752 | c71a1a3d | Iustin Pop | After you've finished you can deactivate them with the deactivate-disks |
753 | c71a1a3d | Iustin Pop | command, which works in the same way:: |
754 | c71a1a3d | Iustin Pop | |
755 | 73225861 | Iustin Pop | $ gnt-instance deactivate-disks %INSTANCE% |
756 | c71a1a3d | Iustin Pop | |
757 | c71a1a3d | Iustin Pop | Note that if any process started by you is still using the disks, the |
758 | c71a1a3d | Iustin Pop | above command will error out, and you **must** cleanup and ensure that |
759 | c71a1a3d | Iustin Pop | the above command runs successfully before you start the instance, |
760 | c71a1a3d | Iustin Pop | otherwise the instance will suffer corruption. |
761 | ffa6869f | Iustin Pop | |
762 | ffa6869f | Iustin Pop | Accessing an instance's console |
763 | fd07c6b3 | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
764 | ffa6869f | Iustin Pop | |
765 | ffa6869f | Iustin Pop | The command to access a running instance's console is:: |
766 | ffa6869f | Iustin Pop | |
767 | 73225861 | Iustin Pop | $ gnt-instance console %INSTANCE_NAME% |
768 | ffa6869f | Iustin Pop | |
769 | c71a1a3d | Iustin Pop | Use the console normally and then type ``^]`` when done, to exit. |
770 | c71a1a3d | Iustin Pop | |
771 | c71a1a3d | Iustin Pop | Other instance operations |
772 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++ |
773 | c71a1a3d | Iustin Pop | |
774 | c71a1a3d | Iustin Pop | Reboot |
775 | c71a1a3d | Iustin Pop | ~~~~~~ |
776 | ffa6869f | Iustin Pop | |
777 | c71a1a3d | Iustin Pop | There is a wrapper command for rebooting instances:: |
778 | ffa6869f | Iustin Pop | |
779 | 73225861 | Iustin Pop | $ gnt-instance reboot %instance2% |
780 | c71a1a3d | Iustin Pop | |
781 | c71a1a3d | Iustin Pop | By default, this does the equivalent of shutting down and then starting |
782 | c71a1a3d | Iustin Pop | the instance, but it accepts parameters to perform a soft-reboot (via |
783 | c71a1a3d | Iustin Pop | the hypervisor), a hard reboot (hypervisor shutdown and then startup) or |
784 | c71a1a3d | Iustin Pop | a full one (the default, which also de-configures and then configures |
785 | c71a1a3d | Iustin Pop | again the disks of the instance). |
786 | c71a1a3d | Iustin Pop | |
787 | c71a1a3d | Iustin Pop | Instance OS definitions debugging |
788 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++++++++++ |
789 | c71a1a3d | Iustin Pop | |
790 | c71a1a3d | Iustin Pop | Should you have any problems with instance operating systems the command |
791 | c71a1a3d | Iustin Pop | to see a complete status for all your nodes is:: |
792 | ffa6869f | Iustin Pop | |
793 | 73225861 | Iustin Pop | $ gnt-os diagnose |
794 | ffa6869f | Iustin Pop | |
795 | c71a1a3d | Iustin Pop | .. _instance-relocation-label: |
796 | ffa6869f | Iustin Pop | |
797 | c71a1a3d | Iustin Pop | Instance relocation |
798 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~ |
799 | ffa6869f | Iustin Pop | |
800 | c71a1a3d | Iustin Pop | While it is not possible to move an instance from nodes ``(A, B)`` to |
801 | c71a1a3d | Iustin Pop | nodes ``(C, D)`` in a single move, it is possible to do so in a few |
802 | c71a1a3d | Iustin Pop | steps:: |
803 | ffa6869f | Iustin Pop | |
804 | c71a1a3d | Iustin Pop | # instance is located on A, B |
805 | fe0d94d8 | Guido Trotter | $ gnt-instance replace-disks -n %nodeC% %instance1% |
806 | c71a1a3d | Iustin Pop | # instance has moved from (A, B) to (A, C) |
807 | c71a1a3d | Iustin Pop | # we now flip the primary/secondary nodes |
808 | 73225861 | Iustin Pop | $ gnt-instance migrate %instance1% |
809 | c71a1a3d | Iustin Pop | # instance lives on (C, A) |
810 | c71a1a3d | Iustin Pop | # we can then change A to D via: |
811 | fe0d94d8 | Guido Trotter | $ gnt-instance replace-disks -n %nodeD% %instance1% |
812 | 56c9a709 | Iustin Pop | |
813 | c71a1a3d | Iustin Pop | Which brings it into the final configuration of ``(C, D)``. Note that we |
814 | c71a1a3d | Iustin Pop | needed to do two replace-disks operation (two copies of the instance |
815 | c71a1a3d | Iustin Pop | disks), because we needed to get rid of both the original nodes (A and |
816 | c71a1a3d | Iustin Pop | B). |
817 | c71a1a3d | Iustin Pop | |
818 | c71a1a3d | Iustin Pop | Node operations |
819 | c71a1a3d | Iustin Pop | --------------- |
820 | c71a1a3d | Iustin Pop | |
821 | c71a1a3d | Iustin Pop | There are much fewer node operations available than for instances, but |
822 | c71a1a3d | Iustin Pop | they are equivalently important for maintaining a healthy cluster. |
823 | c71a1a3d | Iustin Pop | |
824 | c71a1a3d | Iustin Pop | Add/readd |
825 | c71a1a3d | Iustin Pop | +++++++++ |
826 | c71a1a3d | Iustin Pop | |
827 | c71a1a3d | Iustin Pop | It is at any time possible to extend the cluster with one more node, by |
828 | c71a1a3d | Iustin Pop | using the node add operation:: |
829 | c71a1a3d | Iustin Pop | |
830 | 73225861 | Iustin Pop | $ gnt-node add %NEW_NODE% |
831 | c71a1a3d | Iustin Pop | |
832 | c71a1a3d | Iustin Pop | If the cluster has a replication network defined, then you need to pass |
833 | c71a1a3d | Iustin Pop | the ``-s REPLICATION_IP`` parameter to this option. |
834 | c71a1a3d | Iustin Pop | |
835 | c71a1a3d | Iustin Pop | A variation of this command can be used to re-configure a node if its |
836 | c71a1a3d | Iustin Pop | Ganeti configuration is broken, for example if it has been reinstalled |
837 | c71a1a3d | Iustin Pop | by mistake:: |
838 | c71a1a3d | Iustin Pop | |
839 | 73225861 | Iustin Pop | $ gnt-node add --readd %EXISTING_NODE% |
840 | c71a1a3d | Iustin Pop | |
841 | c71a1a3d | Iustin Pop | This will reinitialise the node as if it's been newly added, but while |
842 | c71a1a3d | Iustin Pop | keeping its existing configuration in the cluster (primary/secondary IP, |
843 | c71a1a3d | Iustin Pop | etc.), in other words you won't need to use ``-s`` here. |
844 | c71a1a3d | Iustin Pop | |
845 | c71a1a3d | Iustin Pop | Changing the node role |
846 | c71a1a3d | Iustin Pop | ++++++++++++++++++++++ |
847 | c71a1a3d | Iustin Pop | |
848 | c71a1a3d | Iustin Pop | A node can be in different roles, as explained in the |
849 | c71a1a3d | Iustin Pop | :ref:`terminology-label` section. Promoting a node to the master role is |
850 | c71a1a3d | Iustin Pop | special, while the other roles are handled all via a single command. |
851 | c71a1a3d | Iustin Pop | |
852 | c71a1a3d | Iustin Pop | Failing over the master node |
853 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
854 | c71a1a3d | Iustin Pop | |
855 | c71a1a3d | Iustin Pop | If you want to promote a different node to the master role (for whatever |
856 | c71a1a3d | Iustin Pop | reason), run on any other master-candidate node the command:: |
857 | c71a1a3d | Iustin Pop | |
858 | 73225861 | Iustin Pop | $ gnt-cluster master-failover |
859 | c71a1a3d | Iustin Pop | |
860 | c71a1a3d | Iustin Pop | and the node you ran it on is now the new master. In case you try to run |
861 | c71a1a3d | Iustin Pop | this on a non master-candidate node, you will get an error telling you |
862 | c71a1a3d | Iustin Pop | which nodes are valid. |
863 | c71a1a3d | Iustin Pop | |
864 | c71a1a3d | Iustin Pop | Changing between the other roles |
865 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
866 | c71a1a3d | Iustin Pop | |
867 | c71a1a3d | Iustin Pop | The ``gnt-node modify`` command can be used to select a new role:: |
868 | c71a1a3d | Iustin Pop | |
869 | c71a1a3d | Iustin Pop | # change to master candidate |
870 | 73225861 | Iustin Pop | $ gnt-node modify -C yes %NODE% |
871 | c71a1a3d | Iustin Pop | # change to drained status |
872 | 73225861 | Iustin Pop | $ gnt-node modify -D yes %NODE% |
873 | c71a1a3d | Iustin Pop | # change to offline status |
874 | 73225861 | Iustin Pop | $ gnt-node modify -O yes %NODE% |
875 | c71a1a3d | Iustin Pop | # change to regular mode (reset all flags) |
876 | 73225861 | Iustin Pop | $ gnt-node modify -O no -D no -C no %NODE% |
877 | c71a1a3d | Iustin Pop | |
878 | c71a1a3d | Iustin Pop | Note that the cluster requires that at any point in time, a certain |
879 | c71a1a3d | Iustin Pop | number of nodes are master candidates, so changing from master candidate |
880 | c71a1a3d | Iustin Pop | to other roles might fail. It is recommended to either force the |
881 | c71a1a3d | Iustin Pop | operation (via the ``--force`` option) or first change the number of |
882 | c71a1a3d | Iustin Pop | master candidates in the cluster - see :ref:`cluster-config-label`. |
883 | c71a1a3d | Iustin Pop | |
884 | c71a1a3d | Iustin Pop | Evacuating nodes |
885 | c71a1a3d | Iustin Pop | ++++++++++++++++ |
886 | c71a1a3d | Iustin Pop | |
887 | c71a1a3d | Iustin Pop | There are two steps of moving instances off a node: |
888 | c71a1a3d | Iustin Pop | |
889 | c71a1a3d | Iustin Pop | - moving the primary instances (actually converting them into secondary |
890 | c71a1a3d | Iustin Pop | instances) |
891 | c71a1a3d | Iustin Pop | - moving the secondary instances (including any instances converted in |
892 | c71a1a3d | Iustin Pop | the step above) |
893 | c71a1a3d | Iustin Pop | |
894 | c71a1a3d | Iustin Pop | Primary instance conversion |
895 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
896 | c71a1a3d | Iustin Pop | |
897 | c71a1a3d | Iustin Pop | For this step, you can use either individual instance move |
898 | c71a1a3d | Iustin Pop | commands (as seen in :ref:`instance-change-primary-label`) or the bulk |
899 | c71a1a3d | Iustin Pop | per-node versions; these are:: |
900 | c71a1a3d | Iustin Pop | |
901 | 73225861 | Iustin Pop | $ gnt-node migrate %NODE% |
902 | 73225861 | Iustin Pop | $ gnt-node evacuate -s %NODE% |
903 | c71a1a3d | Iustin Pop | |
904 | c71a1a3d | Iustin Pop | Note that the instance โmoveโ command doesn't currently have a node |
905 | c71a1a3d | Iustin Pop | equivalent. |
906 | c71a1a3d | Iustin Pop | |
907 | c71a1a3d | Iustin Pop | Both these commands, or the equivalent per-instance command, will make |
908 | c71a1a3d | Iustin Pop | this node the secondary node for the respective instances, whereas their |
909 | c71a1a3d | Iustin Pop | current secondary node will become primary. Note that it is not possible |
910 | c71a1a3d | Iustin Pop | to change in one step the primary node to another node as primary, while |
911 | c71a1a3d | Iustin Pop | keeping the same secondary node. |
912 | c71a1a3d | Iustin Pop | |
913 | c71a1a3d | Iustin Pop | Secondary instance evacuation |
914 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
915 | c71a1a3d | Iustin Pop | |
916 | c71a1a3d | Iustin Pop | For the evacuation of secondary instances, a command called |
917 | c71a1a3d | Iustin Pop | :command:`gnt-node evacuate` is provided and its syntax is:: |
918 | c71a1a3d | Iustin Pop | |
919 | 73225861 | Iustin Pop | $ gnt-node evacuate -I %IALLOCATOR_SCRIPT% %NODE% |
920 | 73225861 | Iustin Pop | $ gnt-node evacuate -n %DESTINATION_NODE% %NODE% |
921 | c71a1a3d | Iustin Pop | |
922 | c71a1a3d | Iustin Pop | The first version will compute the new secondary for each instance in |
923 | c71a1a3d | Iustin Pop | turn using the given iallocator script, whereas the second one will |
924 | c71a1a3d | Iustin Pop | simply move all instances to DESTINATION_NODE. |
925 | c71a1a3d | Iustin Pop | |
926 | c71a1a3d | Iustin Pop | Removal |
927 | c71a1a3d | Iustin Pop | +++++++ |
928 | c71a1a3d | Iustin Pop | |
929 | c71a1a3d | Iustin Pop | Once a node no longer has any instances (neither primary nor secondary), |
930 | c71a1a3d | Iustin Pop | it's easy to remove it from the cluster:: |
931 | c71a1a3d | Iustin Pop | |
932 | 73225861 | Iustin Pop | $ gnt-node remove %NODE_NAME% |
933 | c71a1a3d | Iustin Pop | |
934 | c71a1a3d | Iustin Pop | This will deconfigure the node, stop the ganeti daemons on it and leave |
935 | c71a1a3d | Iustin Pop | it hopefully like before it joined to the cluster. |
936 | c71a1a3d | Iustin Pop | |
937 | 79829d23 | Guido Trotter | Replication network changes |
938 | 79829d23 | Guido Trotter | +++++++++++++++++++++++++++ |
939 | 79829d23 | Guido Trotter | |
940 | 79829d23 | Guido Trotter | The :command:`gnt-node modify -s` command can be used to change the |
941 | 79829d23 | Guido Trotter | secondary IP of a node. This operation can only be performed if: |
942 | 79829d23 | Guido Trotter | |
943 | 79829d23 | Guido Trotter | - No instance is active on the target node |
944 | 79829d23 | Guido Trotter | - The new target IP is reachable from the master's secondary IP |
945 | 79829d23 | Guido Trotter | |
946 | 79829d23 | Guido Trotter | Also this operation will not allow to change a node from single-homed |
947 | 79829d23 | Guido Trotter | (same primary and secondary ip) to multi-homed (separate replication |
948 | 79829d23 | Guido Trotter | network) or vice versa, unless: |
949 | 79829d23 | Guido Trotter | |
950 | 79829d23 | Guido Trotter | - The target node is the master node and `--force` is passed. |
951 | 79829d23 | Guido Trotter | - The target cluster is single-homed and the new primary ip is a change |
952 | 79829d23 | Guido Trotter | to single homed for a particular node. |
953 | 79829d23 | Guido Trotter | - The target cluster is multi-homed and the new primary ip is a change |
954 | 79829d23 | Guido Trotter | to multi homed for a particular node. |
955 | 79829d23 | Guido Trotter | |
956 | 79829d23 | Guido Trotter | For example to do a single-homed to multi-homed conversion:: |
957 | 79829d23 | Guido Trotter | |
958 | 79829d23 | Guido Trotter | $ gnt-node modify --force -s %SECONDARY_IP% %MASTER_NAME% |
959 | 79829d23 | Guido Trotter | $ gnt-node modify -s %SECONDARY_IP% %NODE1_NAME% |
960 | 79829d23 | Guido Trotter | $ gnt-node modify -s %SECONDARY_IP% %NODE2_NAME% |
961 | 79829d23 | Guido Trotter | $ gnt-node modify -s %SECONDARY_IP% %NODE3_NAME% |
962 | 79829d23 | Guido Trotter | ... |
963 | 79829d23 | Guido Trotter | |
964 | 79829d23 | Guido Trotter | The same commands can be used for multi-homed to single-homed except the |
965 | 79829d23 | Guido Trotter | secondary IPs should be the same as the primaries for each node, for |
966 | 79829d23 | Guido Trotter | that case. |
967 | 79829d23 | Guido Trotter | |
968 | c71a1a3d | Iustin Pop | Storage handling |
969 | c71a1a3d | Iustin Pop | ++++++++++++++++ |
970 | c71a1a3d | Iustin Pop | |
971 | c71a1a3d | Iustin Pop | When using LVM (either standalone or with DRBD), it can become tedious |
972 | c71a1a3d | Iustin Pop | to debug and fix it in case of errors. Furthermore, even file-based |
973 | c71a1a3d | Iustin Pop | storage can become complicated to handle manually on many hosts. Ganeti |
974 | c71a1a3d | Iustin Pop | provides a couple of commands to help with automation. |
975 | c71a1a3d | Iustin Pop | |
976 | c71a1a3d | Iustin Pop | Logical volumes |
977 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~ |
978 | c71a1a3d | Iustin Pop | |
979 | c71a1a3d | Iustin Pop | This is a command specific to LVM handling. It allows listing the |
980 | c71a1a3d | Iustin Pop | logical volumes on a given node or on all nodes and their association to |
981 | c71a1a3d | Iustin Pop | instances via the ``volumes`` command:: |
982 | c71a1a3d | Iustin Pop | |
983 | 73225861 | Iustin Pop | $ gnt-node volumes |
984 | c71a1a3d | Iustin Pop | Node PhysDev VG Name Size Instance |
985 | c71a1a3d | Iustin Pop | node1 /dev/sdb1 xenvg e61fbc97-โฆ.disk0 512M instance17 |
986 | c71a1a3d | Iustin Pop | node1 /dev/sdb1 xenvg ebd1a7d1-โฆ.disk0 512M instance19 |
987 | c71a1a3d | Iustin Pop | node2 /dev/sdb1 xenvg 0af08a3d-โฆ.disk0 512M instance20 |
988 | c71a1a3d | Iustin Pop | node2 /dev/sdb1 xenvg cc012285-โฆ.disk0 512M instance16 |
989 | c71a1a3d | Iustin Pop | node2 /dev/sdb1 xenvg f0fac192-โฆ.disk0 512M instance18 |
990 | c71a1a3d | Iustin Pop | |
991 | c71a1a3d | Iustin Pop | The above command maps each logical volume to a volume group and |
992 | c71a1a3d | Iustin Pop | underlying physical volume and (possibly) to an instance. |
993 | c71a1a3d | Iustin Pop | |
994 | c71a1a3d | Iustin Pop | .. _storage-units-label: |
995 | c71a1a3d | Iustin Pop | |
996 | c71a1a3d | Iustin Pop | Generalized storage handling |
997 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
998 | c71a1a3d | Iustin Pop | |
999 | c71a1a3d | Iustin Pop | .. versionadded:: 2.1 |
1000 | c71a1a3d | Iustin Pop | |
1001 | c71a1a3d | Iustin Pop | Starting with Ganeti 2.1, a new storage framework has been implemented |
1002 | c71a1a3d | Iustin Pop | that tries to abstract the handling of the storage type the cluster |
1003 | c71a1a3d | Iustin Pop | uses. |
1004 | c71a1a3d | Iustin Pop | |
1005 | c71a1a3d | Iustin Pop | First is listing the backend storage and their space situation:: |
1006 | c71a1a3d | Iustin Pop | |
1007 | 73225861 | Iustin Pop | $ gnt-node list-storage |
1008 | c71a1a3d | Iustin Pop | Node Name Size Used Free |
1009 | c71a1a3d | Iustin Pop | node1 /dev/sda7 673.8G 0M 673.8G |
1010 | c71a1a3d | Iustin Pop | node1 /dev/sdb1 698.6G 1.5G 697.1G |
1011 | c71a1a3d | Iustin Pop | node2 /dev/sda7 673.8G 0M 673.8G |
1012 | c71a1a3d | Iustin Pop | node2 /dev/sdb1 698.6G 1.0G 697.6G |
1013 | c71a1a3d | Iustin Pop | |
1014 | c71a1a3d | Iustin Pop | The default is to list LVM physical volumes. It's also possible to list |
1015 | c71a1a3d | Iustin Pop | the LVM volume groups:: |
1016 | c71a1a3d | Iustin Pop | |
1017 | 73225861 | Iustin Pop | $ gnt-node list-storage -t lvm-vg |
1018 | c71a1a3d | Iustin Pop | Node Name Size |
1019 | c71a1a3d | Iustin Pop | node1 xenvg 1.3T |
1020 | c71a1a3d | Iustin Pop | node2 xenvg 1.3T |
1021 | c71a1a3d | Iustin Pop | |
1022 | c71a1a3d | Iustin Pop | Next is repairing storage units, which is currently only implemented for |
1023 | c71a1a3d | Iustin Pop | volume groups and does the equivalent of ``vgreduce --removemissing``:: |
1024 | c71a1a3d | Iustin Pop | |
1025 | 73225861 | Iustin Pop | $ gnt-node repair-storage %node2% lvm-vg xenvg |
1026 | c71a1a3d | Iustin Pop | Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ... |
1027 | c71a1a3d | Iustin Pop | |
1028 | c71a1a3d | Iustin Pop | Last is the modification of volume properties, which is (again) only |
1029 | c71a1a3d | Iustin Pop | implemented for LVM physical volumes and allows toggling the |
1030 | c71a1a3d | Iustin Pop | ``allocatable`` value:: |
1031 | c71a1a3d | Iustin Pop | |
1032 | 73225861 | Iustin Pop | $ gnt-node modify-storage --allocatable=no %node2% lvm-pv /dev/%sdb1% |
1033 | c71a1a3d | Iustin Pop | |
1034 | c71a1a3d | Iustin Pop | Use of the storage commands |
1035 | 56c9a709 | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
1036 | 56c9a709 | Iustin Pop | |
1037 | c71a1a3d | Iustin Pop | All these commands are needed when recovering a node from a disk |
1038 | c71a1a3d | Iustin Pop | failure: |
1039 | c71a1a3d | Iustin Pop | |
1040 | c71a1a3d | Iustin Pop | - first, we need to recover from complete LVM failure (due to missing |
1041 | c71a1a3d | Iustin Pop | disk), by running the ``repair-storage`` command |
1042 | c71a1a3d | Iustin Pop | - second, we need to change allocation on any partially-broken disk |
1043 | c71a1a3d | Iustin Pop | (i.e. LVM still sees it, but it has bad blocks) by running |
1044 | c71a1a3d | Iustin Pop | ``modify-storage`` |
1045 | c71a1a3d | Iustin Pop | - then we can evacuate the instances as needed |
1046 | 56c9a709 | Iustin Pop | |
1047 | 56c9a709 | Iustin Pop | |
1048 | c71a1a3d | Iustin Pop | Cluster operations |
1049 | c71a1a3d | Iustin Pop | ------------------ |
1050 | c71a1a3d | Iustin Pop | |
1051 | c71a1a3d | Iustin Pop | Beside the cluster initialisation command (which is detailed in the |
1052 | c71a1a3d | Iustin Pop | :doc:`install` document) and the master failover command which is |
1053 | c71a1a3d | Iustin Pop | explained under node handling, there are a couple of other cluster |
1054 | c71a1a3d | Iustin Pop | operations available. |
1055 | c71a1a3d | Iustin Pop | |
1056 | c71a1a3d | Iustin Pop | .. _cluster-config-label: |
1057 | c71a1a3d | Iustin Pop | |
1058 | c71a1a3d | Iustin Pop | Standard operations |
1059 | c71a1a3d | Iustin Pop | +++++++++++++++++++ |
1060 | c71a1a3d | Iustin Pop | |
1061 | c71a1a3d | Iustin Pop | One of the few commands that can be run on any node (not only the |
1062 | c71a1a3d | Iustin Pop | master) is the ``getmaster`` command:: |
1063 | c71a1a3d | Iustin Pop | |
1064 | 73225861 | Iustin Pop | # on node2 |
1065 | 73225861 | Iustin Pop | $ gnt-cluster getmaster |
1066 | c71a1a3d | Iustin Pop | node1.example.com |
1067 | c71a1a3d | Iustin Pop | |
1068 | c71a1a3d | Iustin Pop | It is possible to query and change global cluster parameters via the |
1069 | c71a1a3d | Iustin Pop | ``info`` and ``modify`` commands:: |
1070 | c71a1a3d | Iustin Pop | |
1071 | 73225861 | Iustin Pop | $ gnt-cluster info |
1072 | c71a1a3d | Iustin Pop | Cluster name: cluster.example.com |
1073 | c71a1a3d | Iustin Pop | Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c |
1074 | c71a1a3d | Iustin Pop | Creation time: 2009-09-25 05:04:15 |
1075 | c71a1a3d | Iustin Pop | Modification time: 2009-10-18 22:11:47 |
1076 | c71a1a3d | Iustin Pop | Master node: node1.example.com |
1077 | c71a1a3d | Iustin Pop | Architecture (this node): 64bit (x86_64) |
1078 | c71a1a3d | Iustin Pop | โฆ |
1079 | c71a1a3d | Iustin Pop | Tags: foo |
1080 | c71a1a3d | Iustin Pop | Default hypervisor: xen-pvm |
1081 | c71a1a3d | Iustin Pop | Enabled hypervisors: xen-pvm |
1082 | c71a1a3d | Iustin Pop | Hypervisor parameters: |
1083 | c71a1a3d | Iustin Pop | - xen-pvm: |
1084 | c71a1a3d | Iustin Pop | root_path: /dev/sda1 |
1085 | c71a1a3d | Iustin Pop | โฆ |
1086 | c71a1a3d | Iustin Pop | Cluster parameters: |
1087 | c71a1a3d | Iustin Pop | - candidate pool size: 10 |
1088 | c71a1a3d | Iustin Pop | โฆ |
1089 | c71a1a3d | Iustin Pop | Default instance parameters: |
1090 | c71a1a3d | Iustin Pop | - default: |
1091 | c71a1a3d | Iustin Pop | memory: 128 |
1092 | c71a1a3d | Iustin Pop | โฆ |
1093 | c71a1a3d | Iustin Pop | Default nic parameters: |
1094 | c71a1a3d | Iustin Pop | - default: |
1095 | c71a1a3d | Iustin Pop | link: xen-br0 |
1096 | c71a1a3d | Iustin Pop | โฆ |
1097 | c71a1a3d | Iustin Pop | |
1098 | c71a1a3d | Iustin Pop | There various parameters above can be changed via the ``modify`` |
1099 | c71a1a3d | Iustin Pop | commands as follows: |
1100 | c71a1a3d | Iustin Pop | |
1101 | c71a1a3d | Iustin Pop | - the hypervisor parameters can be changed via ``modify -H |
1102 | c71a1a3d | Iustin Pop | xen-pvm:root_path=โฆ``, and so on for other hypervisors/key/values |
1103 | c71a1a3d | Iustin Pop | - the "default instance parameters" are changeable via ``modify -B |
1104 | c71a1a3d | Iustin Pop | parameter=valueโฆ`` syntax |
1105 | c71a1a3d | Iustin Pop | - the cluster parameters are changeable via separate options to the |
1106 | c71a1a3d | Iustin Pop | modify command (e.g. ``--candidate-pool-size``, etc.) |
1107 | c71a1a3d | Iustin Pop | |
1108 | c71a1a3d | Iustin Pop | For detailed option list see the :manpage:`gnt-cluster(8)` man page. |
1109 | c71a1a3d | Iustin Pop | |
1110 | c71a1a3d | Iustin Pop | The cluster version can be obtained via the ``version`` command:: |
1111 | 73225861 | Iustin Pop | $ gnt-cluster version |
1112 | c71a1a3d | Iustin Pop | Software version: 2.1.0 |
1113 | c71a1a3d | Iustin Pop | Internode protocol: 20 |
1114 | c71a1a3d | Iustin Pop | Configuration format: 2010000 |
1115 | c71a1a3d | Iustin Pop | OS api version: 15 |
1116 | c71a1a3d | Iustin Pop | Export interface: 0 |
1117 | c71a1a3d | Iustin Pop | |
1118 | c71a1a3d | Iustin Pop | This is not very useful except when debugging Ganeti. |
1119 | c71a1a3d | Iustin Pop | |
1120 | c71a1a3d | Iustin Pop | Global node commands |
1121 | c71a1a3d | Iustin Pop | ++++++++++++++++++++ |
1122 | c71a1a3d | Iustin Pop | |
1123 | c71a1a3d | Iustin Pop | There are two commands provided for replicating files to all nodes of a |
1124 | c71a1a3d | Iustin Pop | cluster and for running commands on all the nodes:: |
1125 | c71a1a3d | Iustin Pop | |
1126 | 73225861 | Iustin Pop | $ gnt-cluster copyfile %/path/to/file% |
1127 | 73225861 | Iustin Pop | $ gnt-cluster command %ls -l /path/to/file% |
1128 | c71a1a3d | Iustin Pop | |
1129 | c71a1a3d | Iustin Pop | These are simple wrappers over scp/ssh and more advanced usage can be |
1130 | c71a1a3d | Iustin Pop | obtained using :manpage:`dsh(1)` and similar commands. But they are |
1131 | c71a1a3d | Iustin Pop | useful to update an OS script from the master node, for example. |
1132 | c71a1a3d | Iustin Pop | |
1133 | c71a1a3d | Iustin Pop | Cluster verification |
1134 | c71a1a3d | Iustin Pop | ++++++++++++++++++++ |
1135 | c71a1a3d | Iustin Pop | |
1136 | c71a1a3d | Iustin Pop | There are three commands that relate to global cluster checks. The first |
1137 | c71a1a3d | Iustin Pop | one is ``verify`` which gives an overview on the cluster state, |
1138 | c71a1a3d | Iustin Pop | highlighting any issues. In normal operation, this command should return |
1139 | c71a1a3d | Iustin Pop | no ``ERROR`` messages:: |
1140 | c71a1a3d | Iustin Pop | |
1141 | 73225861 | Iustin Pop | $ gnt-cluster verify |
1142 | c71a1a3d | Iustin Pop | Sun Oct 25 23:08:58 2009 * Verifying global settings |
1143 | c71a1a3d | Iustin Pop | Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes) |
1144 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying node status |
1145 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying instance status |
1146 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying orphan volumes |
1147 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying remaining instances |
1148 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy |
1149 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Other Notes |
1150 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 - NOTICE: 5 non-redundant instance(s) found. |
1151 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Hooks Results |
1152 | c71a1a3d | Iustin Pop | |
1153 | c71a1a3d | Iustin Pop | The second command is ``verify-disks``, which checks that the instance's |
1154 | c71a1a3d | Iustin Pop | disks have the correct status based on the desired instance state |
1155 | c71a1a3d | Iustin Pop | (up/down):: |
1156 | c71a1a3d | Iustin Pop | |
1157 | 73225861 | Iustin Pop | $ gnt-cluster verify-disks |
1158 | c71a1a3d | Iustin Pop | |
1159 | c71a1a3d | Iustin Pop | Note that this command will show no output when disks are healthy. |
1160 | c71a1a3d | Iustin Pop | |
1161 | c71a1a3d | Iustin Pop | The last command is used to repair any discrepancies in Ganeti's |
1162 | c71a1a3d | Iustin Pop | recorded disk size and the actual disk size (disk size information is |
1163 | c71a1a3d | Iustin Pop | needed for proper activation and growth of DRBD-based disks):: |
1164 | c71a1a3d | Iustin Pop | |
1165 | 73225861 | Iustin Pop | $ gnt-cluster repair-disk-sizes |
1166 | c71a1a3d | Iustin Pop | Sun Oct 25 23:13:16 2009 - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048 |
1167 | c71a1a3d | Iustin Pop | Sun Oct 25 23:13:17 2009 - WARNING: Invalid result from node node4, ignoring node results |
1168 | c71a1a3d | Iustin Pop | |
1169 | c71a1a3d | Iustin Pop | The above shows one instance having wrong disk size, and a node which |
1170 | c71a1a3d | Iustin Pop | returned invalid data, and thus we ignored all primary instances of that |
1171 | c71a1a3d | Iustin Pop | node. |
1172 | c71a1a3d | Iustin Pop | |
1173 | c71a1a3d | Iustin Pop | Configuration redistribution |
1174 | c71a1a3d | Iustin Pop | ++++++++++++++++++++++++++++ |
1175 | c71a1a3d | Iustin Pop | |
1176 | c71a1a3d | Iustin Pop | If the verify command complains about file mismatches between the master |
1177 | c71a1a3d | Iustin Pop | and other nodes, due to some node problems or if you manually modified |
1178 | c71a1a3d | Iustin Pop | configuration files, you can force an push of the master configuration |
1179 | c71a1a3d | Iustin Pop | to all other nodes via the ``redist-conf`` command:: |
1180 | c71a1a3d | Iustin Pop | |
1181 | 73225861 | Iustin Pop | $ gnt-cluster redist-conf |
1182 | c71a1a3d | Iustin Pop | |
1183 | c71a1a3d | Iustin Pop | This command will be silent unless there are problems sending updates to |
1184 | c71a1a3d | Iustin Pop | the other nodes. |
1185 | c71a1a3d | Iustin Pop | |
1186 | c71a1a3d | Iustin Pop | |
1187 | c71a1a3d | Iustin Pop | Cluster renaming |
1188 | c71a1a3d | Iustin Pop | ++++++++++++++++ |
1189 | c71a1a3d | Iustin Pop | |
1190 | c71a1a3d | Iustin Pop | It is possible to rename a cluster, or to change its IP address, via the |
1191 | c71a1a3d | Iustin Pop | ``rename`` command. If only the IP has changed, you need to pass the |
1192 | c71a1a3d | Iustin Pop | current name and Ganeti will realise its IP has changed:: |
1193 | c71a1a3d | Iustin Pop | |
1194 | 73225861 | Iustin Pop | $ gnt-cluster rename %cluster.example.com% |
1195 | c71a1a3d | Iustin Pop | This will rename the cluster to 'cluster.example.com'. If |
1196 | c71a1a3d | Iustin Pop | you are connected over the network to the cluster name, the operation |
1197 | c71a1a3d | Iustin Pop | is very dangerous as the IP address will be removed from the node and |
1198 | c71a1a3d | Iustin Pop | the change may not go through. Continue? |
1199 | 73225861 | Iustin Pop | y/[n]/?: %y% |
1200 | c71a1a3d | Iustin Pop | Failure: prerequisites not met for this operation: |
1201 | c71a1a3d | Iustin Pop | Neither the name nor the IP address of the cluster has changed |
1202 | c71a1a3d | Iustin Pop | |
1203 | c71a1a3d | Iustin Pop | In the above output, neither value has changed since the cluster |
1204 | c71a1a3d | Iustin Pop | initialisation so the operation is not completed. |
1205 | c71a1a3d | Iustin Pop | |
1206 | c71a1a3d | Iustin Pop | Queue operations |
1207 | c71a1a3d | Iustin Pop | ++++++++++++++++ |
1208 | c71a1a3d | Iustin Pop | |
1209 | c71a1a3d | Iustin Pop | The job queue execution in Ganeti 2.0 and higher can be inspected, |
1210 | c71a1a3d | Iustin Pop | suspended and resumed via the ``queue`` command:: |
1211 | c71a1a3d | Iustin Pop | |
1212 | 73225861 | Iustin Pop | $ gnt-cluster queue info |
1213 | c71a1a3d | Iustin Pop | The drain flag is unset |
1214 | 73225861 | Iustin Pop | $ gnt-cluster queue drain |
1215 | 73225861 | Iustin Pop | $ gnt-instance stop %instance1% |
1216 | c71a1a3d | Iustin Pop | Failed to submit job for instance1: Job queue is drained, refusing job |
1217 | 73225861 | Iustin Pop | $ gnt-cluster queue info |
1218 | c71a1a3d | Iustin Pop | The drain flag is set |
1219 | 73225861 | Iustin Pop | $ gnt-cluster queue undrain |
1220 | c71a1a3d | Iustin Pop | |
1221 | c71a1a3d | Iustin Pop | This is most useful if you have an active cluster and you need to |
1222 | c71a1a3d | Iustin Pop | upgrade the Ganeti software, or simply restart the software on any node: |
1223 | c71a1a3d | Iustin Pop | |
1224 | c71a1a3d | Iustin Pop | #. suspend the queue via ``queue drain`` |
1225 | c71a1a3d | Iustin Pop | #. wait until there are no more running jobs via ``gnt-job list`` |
1226 | c71a1a3d | Iustin Pop | #. restart the master or another node, or upgrade the software |
1227 | c71a1a3d | Iustin Pop | #. resume the queue via ``queue undrain`` |
1228 | c71a1a3d | Iustin Pop | |
1229 | c71a1a3d | Iustin Pop | .. note:: this command only stores a local flag file, and if you |
1230 | c71a1a3d | Iustin Pop | failover the master, it will not have effect on the new master. |
1231 | c71a1a3d | Iustin Pop | |
1232 | c71a1a3d | Iustin Pop | |
1233 | c71a1a3d | Iustin Pop | Watcher control |
1234 | c71a1a3d | Iustin Pop | +++++++++++++++ |
1235 | c71a1a3d | Iustin Pop | |
1236 | 22ac4136 | Michael Hanselmann | The :manpage:`ganeti-watcher(8)` is a program, usually scheduled via |
1237 | c71a1a3d | Iustin Pop | ``cron``, that takes care of cluster maintenance operations (restarting |
1238 | c71a1a3d | Iustin Pop | downed instances, activating down DRBD disks, etc.). However, during |
1239 | c71a1a3d | Iustin Pop | maintenance and troubleshooting, this can get in your way; disabling it |
1240 | c71a1a3d | Iustin Pop | via commenting out the cron job is not so good as this can be |
1241 | c71a1a3d | Iustin Pop | forgotten. Thus there are some commands for automated control of the |
1242 | c71a1a3d | Iustin Pop | watcher: ``pause``, ``info`` and ``continue``:: |
1243 | c71a1a3d | Iustin Pop | |
1244 | 73225861 | Iustin Pop | $ gnt-cluster watcher info |
1245 | c71a1a3d | Iustin Pop | The watcher is not paused. |
1246 | 73225861 | Iustin Pop | $ gnt-cluster watcher pause %1h% |
1247 | c71a1a3d | Iustin Pop | The watcher is paused until Mon Oct 26 00:30:37 2009. |
1248 | 73225861 | Iustin Pop | $ gnt-cluster watcher info |
1249 | c71a1a3d | Iustin Pop | The watcher is paused until Mon Oct 26 00:30:37 2009. |
1250 | 73225861 | Iustin Pop | $ ganeti-watcher -d |
1251 | c71a1a3d | Iustin Pop | 2009-10-25 23:30:47,984: pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting |
1252 | 73225861 | Iustin Pop | $ gnt-cluster watcher continue |
1253 | c71a1a3d | Iustin Pop | The watcher is no longer paused. |
1254 | 73225861 | Iustin Pop | $ ganeti-watcher -d |
1255 | c71a1a3d | Iustin Pop | 2009-10-25 23:31:04,789: pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0 |
1256 | c71a1a3d | Iustin Pop | 2009-10-25 23:31:05,884: pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file |
1257 | c71a1a3d | Iustin Pop | 2009-10-25 23:31:06,061: pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file |
1258 | 73225861 | Iustin Pop | $ gnt-cluster watcher info |
1259 | c71a1a3d | Iustin Pop | The watcher is not paused. |
1260 | c71a1a3d | Iustin Pop | |
1261 | c71a1a3d | Iustin Pop | The exact details of the argument to the ``pause`` command are available |
1262 | c71a1a3d | Iustin Pop | in the manpage. |
1263 | c71a1a3d | Iustin Pop | |
1264 | c71a1a3d | Iustin Pop | .. note:: this command only stores a local flag file, and if you |
1265 | c71a1a3d | Iustin Pop | failover the master, it will not have effect on the new master. |
1266 | c71a1a3d | Iustin Pop | |
1267 | 6328fea3 | Iustin Pop | Node auto-maintenance |
1268 | 6328fea3 | Iustin Pop | +++++++++++++++++++++ |
1269 | 6328fea3 | Iustin Pop | |
1270 | 6328fea3 | Iustin Pop | If the cluster parameter ``maintain_node_health`` is enabled (see the |
1271 | 6328fea3 | Iustin Pop | manpage for :command:`gnt-cluster`, the init and modify subcommands), |
1272 | 6328fea3 | Iustin Pop | then the following will happen automatically: |
1273 | 6328fea3 | Iustin Pop | |
1274 | 6328fea3 | Iustin Pop | - the watcher will shutdown any instances running on offline nodes |
1275 | 6328fea3 | Iustin Pop | - the watcher will deactivate any DRBD devices on offline nodes |
1276 | 6328fea3 | Iustin Pop | |
1277 | 6328fea3 | Iustin Pop | In the future, more actions are planned, so only enable this parameter |
1278 | 6328fea3 | Iustin Pop | if the nodes are completely dedicated to Ganeti; otherwise it might be |
1279 | 6328fea3 | Iustin Pop | possible to lose data due to auto-maintenance actions. |
1280 | 6328fea3 | Iustin Pop | |
1281 | c71a1a3d | Iustin Pop | Removing a cluster entirely |
1282 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++++ |
1283 | c71a1a3d | Iustin Pop | |
1284 | c71a1a3d | Iustin Pop | The usual method to cleanup a cluster is to run ``gnt-cluster destroy`` |
1285 | c71a1a3d | Iustin Pop | however if the Ganeti installation is broken in any way then this will |
1286 | c71a1a3d | Iustin Pop | not run. |
1287 | c71a1a3d | Iustin Pop | |
1288 | c71a1a3d | Iustin Pop | It is possible in such a case to cleanup manually most if not all traces |
1289 | c71a1a3d | Iustin Pop | of a cluster installation by following these steps on all of the nodes: |
1290 | c71a1a3d | Iustin Pop | |
1291 | c71a1a3d | Iustin Pop | 1. Shutdown all instances. This depends on the virtualisation method |
1292 | c71a1a3d | Iustin Pop | used (Xen, KVM, etc.): |
1293 | 56c9a709 | Iustin Pop | |
1294 | 56c9a709 | Iustin Pop | - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0 |
1295 | 56c9a709 | Iustin Pop | instances |
1296 | 56c9a709 | Iustin Pop | - KVM: kill all the KVM processes |
1297 | 56c9a709 | Iustin Pop | - chroot: kill all processes under the chroot mountpoints |
1298 | 56c9a709 | Iustin Pop | |
1299 | c71a1a3d | Iustin Pop | 2. If using DRBD, shutdown all DRBD minors (which should by at this time |
1300 | c71a1a3d | Iustin Pop | no-longer in use by instances); on each node, run ``drbdsetup |
1301 | 56c9a709 | Iustin Pop | /dev/drbdN down`` for each active DRBD minor. |
1302 | 56c9a709 | Iustin Pop | |
1303 | c71a1a3d | Iustin Pop | 3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created |
1304 | c71a1a3d | Iustin Pop | logical volumes (and you are not sharing the volume group with the |
1305 | c71a1a3d | Iustin Pop | OS, for example), then simply running ``lvremove -f xenvg`` (replace |
1306 | c71a1a3d | Iustin Pop | 'xenvg' with your volume group name) should do the required cleanup. |
1307 | 56c9a709 | Iustin Pop | |
1308 | 56c9a709 | Iustin Pop | 4. If using file-based storage, remove recursively all files and |
1309 | 56c9a709 | Iustin Pop | directories under your file-storage directory: ``rm -rf |
1310 | c71a1a3d | Iustin Pop | /srv/ganeti/file-storage/*`` replacing the path with the correct path |
1311 | c71a1a3d | Iustin Pop | for your cluster. |
1312 | 56c9a709 | Iustin Pop | |
1313 | 56c9a709 | Iustin Pop | 5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any |
1314 | 56c9a709 | Iustin Pop | that remain alive (``pgrep ganeti`` and ``pkill ganeti``). |
1315 | 56c9a709 | Iustin Pop | |
1316 | 56c9a709 | Iustin Pop | 6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``), |
1317 | 56c9a709 | Iustin Pop | replacing the path with the correct path for your installation. |
1318 | 56c9a709 | Iustin Pop | |
1319 | 7ed400f0 | Stratos Psomadakis | 7. If using RBD, run ``rbd unmap /dev/rbdN`` to unmap the RBD disks. |
1320 | 7ed400f0 | Stratos Psomadakis | Then remove the RBD disk images used by Ganeti, identified by their |
1321 | 7ed400f0 | Stratos Psomadakis | UUIDs (``rbd rm uuid.rbd.diskN``). |
1322 | 7ed400f0 | Stratos Psomadakis | |
1323 | 56c9a709 | Iustin Pop | On the master node, remove the cluster from the master-netdev (usually |
1324 | c71a1a3d | Iustin Pop | ``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running |
1325 | c71a1a3d | Iustin Pop | ``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and |
1326 | c71a1a3d | Iustin Pop | network device name). |
1327 | 56c9a709 | Iustin Pop | |
1328 | 56c9a709 | Iustin Pop | At this point, the machines are ready for a cluster creation; in case |
1329 | c71a1a3d | Iustin Pop | you want to remove Ganeti completely, you need to also undo some of the |
1330 | c71a1a3d | Iustin Pop | SSH changes and log directories: |
1331 | 56c9a709 | Iustin Pop | |
1332 | 7faf5110 | Michael Hanselmann | - ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct |
1333 | 7faf5110 | Michael Hanselmann | paths) |
1334 | c71a1a3d | Iustin Pop | - remove from ``/root/.ssh`` the keys that Ganeti added (check the |
1335 | c71a1a3d | Iustin Pop | ``authorized_keys`` and ``id_dsa`` files) |
1336 | 56c9a709 | Iustin Pop | - regenerate the host's SSH keys (check the OpenSSH startup scripts) |
1337 | 56c9a709 | Iustin Pop | - uninstall Ganeti |
1338 | 56c9a709 | Iustin Pop | |
1339 | 56c9a709 | Iustin Pop | Otherwise, if you plan to re-create the cluster, you can just go ahead |
1340 | 56c9a709 | Iustin Pop | and rerun ``gnt-cluster init``. |
1341 | 558fd122 | Michael Hanselmann | |
1342 | ac0af025 | Michele Tartara | Monitoring the cluster |
1343 | ac0af025 | Michele Tartara | ---------------------- |
1344 | ac0af025 | Michele Tartara | |
1345 | ac0af025 | Michele Tartara | Starting with Ganeti 2.8, a monitoring daemon is available, providing |
1346 | ac0af025 | Michele Tartara | information about the status and the performance of the system. |
1347 | ac0af025 | Michele Tartara | |
1348 | ac0af025 | Michele Tartara | The monitoring daemon runs on every node, listening on TCP port 1815. Each |
1349 | ac0af025 | Michele Tartara | instance of the daemon provides information related to the node it is running |
1350 | ac0af025 | Michele Tartara | on. |
1351 | ac0af025 | Michele Tartara | |
1352 | ac0af025 | Michele Tartara | .. include:: monitoring-query-format.rst |
1353 | ac0af025 | Michele Tartara | |
1354 | c71a1a3d | Iustin Pop | Tags handling |
1355 | c71a1a3d | Iustin Pop | ------------- |
1356 | c71a1a3d | Iustin Pop | |
1357 | c71a1a3d | Iustin Pop | The tags handling (addition, removal, listing) is similar for all the |
1358 | c71a1a3d | Iustin Pop | objects that support it (instances, nodes, and the cluster). |
1359 | c71a1a3d | Iustin Pop | |
1360 | c71a1a3d | Iustin Pop | Limitations |
1361 | c71a1a3d | Iustin Pop | +++++++++++ |
1362 | c71a1a3d | Iustin Pop | |
1363 | c71a1a3d | Iustin Pop | Note that the set of characters present in a tag and the maximum tag |
1364 | c71a1a3d | Iustin Pop | length are restricted. Currently the maximum length is 128 characters, |
1365 | c71a1a3d | Iustin Pop | there can be at most 4096 tags per object, and the set of characters is |
1366 | bde65914 | Iustin Pop | comprised by alphanumeric characters and additionally ``.+*/:@-``. |
1367 | c71a1a3d | Iustin Pop | |
1368 | c71a1a3d | Iustin Pop | Operations |
1369 | c71a1a3d | Iustin Pop | ++++++++++ |
1370 | c71a1a3d | Iustin Pop | |
1371 | c71a1a3d | Iustin Pop | Tags can be added via ``add-tags``:: |
1372 | c71a1a3d | Iustin Pop | |
1373 | 73225861 | Iustin Pop | $ gnt-instance add-tags %INSTANCE% %a% %b% %c% |
1374 | 73225861 | Iustin Pop | $ gnt-node add-tags %INSTANCE% %a% %b% %c% |
1375 | 73225861 | Iustin Pop | $ gnt-cluster add-tags %a% %b% %c% |
1376 | c71a1a3d | Iustin Pop | |
1377 | c71a1a3d | Iustin Pop | |
1378 | c71a1a3d | Iustin Pop | The above commands add three tags to an instance, to a node and to the |
1379 | c71a1a3d | Iustin Pop | cluster. Note that the cluster command only takes tags as arguments, |
1380 | c71a1a3d | Iustin Pop | whereas the node and instance commands first required the node and |
1381 | c71a1a3d | Iustin Pop | instance name. |
1382 | c71a1a3d | Iustin Pop | |
1383 | c71a1a3d | Iustin Pop | Tags can also be added from a file, via the ``--from=FILENAME`` |
1384 | c71a1a3d | Iustin Pop | argument. The file is expected to contain one tag per line. |
1385 | c71a1a3d | Iustin Pop | |
1386 | c71a1a3d | Iustin Pop | Tags can also be remove via a syntax very similar to the add one:: |
1387 | c71a1a3d | Iustin Pop | |
1388 | 73225861 | Iustin Pop | $ gnt-instance remove-tags %INSTANCE% %a% %b% %c% |
1389 | c71a1a3d | Iustin Pop | |
1390 | c71a1a3d | Iustin Pop | And listed via:: |
1391 | c71a1a3d | Iustin Pop | |
1392 | 73225861 | Iustin Pop | $ gnt-instance list-tags |
1393 | 73225861 | Iustin Pop | $ gnt-node list-tags |
1394 | 73225861 | Iustin Pop | $ gnt-cluster list-tags |
1395 | c71a1a3d | Iustin Pop | |
1396 | c71a1a3d | Iustin Pop | Global tag search |
1397 | c71a1a3d | Iustin Pop | +++++++++++++++++ |
1398 | c71a1a3d | Iustin Pop | |
1399 | c71a1a3d | Iustin Pop | It is also possible to execute a global search on the all tags defined |
1400 | c71a1a3d | Iustin Pop | in the cluster configuration, via a cluster command:: |
1401 | c71a1a3d | Iustin Pop | |
1402 | 73225861 | Iustin Pop | $ gnt-cluster search-tags %REGEXP% |
1403 | c71a1a3d | Iustin Pop | |
1404 | c71a1a3d | Iustin Pop | The parameter expected is a regular expression (see |
1405 | c71a1a3d | Iustin Pop | :manpage:`regex(7)`). This will return all tags that match the search, |
1406 | c71a1a3d | Iustin Pop | together with the object they are defined in (the names being show in a |
1407 | c71a1a3d | Iustin Pop | hierarchical kind of way):: |
1408 | c71a1a3d | Iustin Pop | |
1409 | 73225861 | Iustin Pop | $ gnt-cluster search-tags %o% |
1410 | c71a1a3d | Iustin Pop | /cluster foo |
1411 | c71a1a3d | Iustin Pop | /instances/instance1 owner:bar |
1412 | c71a1a3d | Iustin Pop | |
1413 | ee414f1c | Michele Tartara | Autorepair |
1414 | ee414f1c | Michele Tartara | ---------- |
1415 | ee414f1c | Michele Tartara | |
1416 | ee414f1c | Michele Tartara | The tool ``harep`` can be used to automatically fix some problems that are |
1417 | ee414f1c | Michele Tartara | present in the cluster. |
1418 | ee414f1c | Michele Tartara | |
1419 | ee414f1c | Michele Tartara | It is mainly meant to be regularly and automatically executed |
1420 | ee414f1c | Michele Tartara | as a cron job. This is quite evident by considering that, when executed, it does |
1421 | ee414f1c | Michele Tartara | not immediately fix all the issues of the instances of the cluster, but it |
1422 | ee414f1c | Michele Tartara | cycles the instances through a series of states, one at every ``harep`` |
1423 | ee414f1c | Michele Tartara | execution. Every state performs a step towards the resolution of the problem. |
1424 | ee414f1c | Michele Tartara | This process goes on until the instance is brought back to the healthy state, |
1425 | ee414f1c | Michele Tartara | or the tool realizes that it is not able to fix the instance, and |
1426 | ee414f1c | Michele Tartara | therefore marks it as in failure state. |
1427 | ee414f1c | Michele Tartara | |
1428 | ee414f1c | Michele Tartara | Allowing harep to act on the cluster |
1429 | ee414f1c | Michele Tartara | ++++++++++++++++++++++++++++++++++++ |
1430 | ee414f1c | Michele Tartara | |
1431 | ee414f1c | Michele Tartara | By default, ``harep`` checks the status of the cluster but it is not allowed to |
1432 | ee414f1c | Michele Tartara | perform any modification. Modification must be explicitly allowed by an |
1433 | ee414f1c | Michele Tartara | appropriate use of tags. Tagging can be applied at various levels, and can |
1434 | ee414f1c | Michele Tartara | enable different kinds of autorepair, as hereafter described. |
1435 | ee414f1c | Michele Tartara | |
1436 | ee414f1c | Michele Tartara | All the tags that authorize ``harep`` to perform modifications follow this |
1437 | ee414f1c | Michele Tartara | syntax:: |
1438 | ee414f1c | Michele Tartara | |
1439 | ee414f1c | Michele Tartara | ganeti:watcher:autorepair:<type> |
1440 | ee414f1c | Michele Tartara | |
1441 | ee414f1c | Michele Tartara | where ``<type>`` indicates the kind of intervention that can be performed. Every |
1442 | ee414f1c | Michele Tartara | possible value of ``<type>`` includes at least all the authorization of the |
1443 | ee414f1c | Michele Tartara | previous one, plus its own. The possible values, in increasing order of |
1444 | ee414f1c | Michele Tartara | severity, are: |
1445 | ee414f1c | Michele Tartara | |
1446 | ee414f1c | Michele Tartara | - ``fix-storage`` allows a disk replacement or another operation that |
1447 | ee414f1c | Michele Tartara | fixes the instance backend storage without affecting the instance |
1448 | ee414f1c | Michele Tartara | itself. This can for example recover from a broken drbd secondary, but |
1449 | ee414f1c | Michele Tartara | risks data loss if something is wrong on the primary but the secondary |
1450 | ee414f1c | Michele Tartara | was somehow recoverable. |
1451 | ee414f1c | Michele Tartara | - ``migrate`` allows an instance migration. This can recover from a |
1452 | ee414f1c | Michele Tartara | drained primary, but can cause an instance crash in some cases (bugs). |
1453 | ee414f1c | Michele Tartara | - ``failover`` allows instance reboot on the secondary. This can recover |
1454 | ee414f1c | Michele Tartara | from an offline primary, but the instance will lose its running state. |
1455 | ee414f1c | Michele Tartara | - ``reinstall`` allows disks to be recreated and an instance to be |
1456 | ee414f1c | Michele Tartara | reinstalled. This can recover from primary&secondary both being |
1457 | ee414f1c | Michele Tartara | offline, or from an offline primary in the case of non-redundant |
1458 | ee414f1c | Michele Tartara | instances. It causes data loss. |
1459 | ee414f1c | Michele Tartara | |
1460 | ee414f1c | Michele Tartara | These autorepair tags can be applied to a cluster, a nodegroup or an instance, |
1461 | ee414f1c | Michele Tartara | and will act where they are applied and to everything in the entities sub-tree |
1462 | ee414f1c | Michele Tartara | (e.g. a tag applied to a nodegroup will apply to all the instances contained in |
1463 | ee414f1c | Michele Tartara | that nodegroup, but not to the rest of the cluster). |
1464 | ee414f1c | Michele Tartara | |
1465 | ee414f1c | Michele Tartara | If there are multiple ``ganeti:watcher:autorepair:<type>`` tags in an |
1466 | ee414f1c | Michele Tartara | object (cluster, node group or instance), the least destructive tag |
1467 | ee414f1c | Michele Tartara | takes precedence. When multiplicity happens across objects, the nearest |
1468 | ee414f1c | Michele Tartara | tag wins. For example, if in a cluster with two instances, *I1* and |
1469 | ee414f1c | Michele Tartara | *I2*, *I1* has ``failover``, and the cluster itself has both |
1470 | ee414f1c | Michele Tartara | ``fix-storage`` and ``reinstall``, *I1* will end up with ``failover`` |
1471 | ee414f1c | Michele Tartara | and *I2* with ``fix-storage``. |
1472 | ee414f1c | Michele Tartara | |
1473 | ee414f1c | Michele Tartara | Limiting harep |
1474 | ee414f1c | Michele Tartara | ++++++++++++++ |
1475 | ee414f1c | Michele Tartara | |
1476 | ee414f1c | Michele Tartara | Sometimes it is useful to stop harep from performing its task temporarily, |
1477 | ee414f1c | Michele Tartara | and it is useful to be able to do so without distrupting its configuration, that |
1478 | ee414f1c | Michele Tartara | is, without removing the authorization tags. In order to do this, suspend tags |
1479 | ee414f1c | Michele Tartara | are provided. |
1480 | ee414f1c | Michele Tartara | |
1481 | ee414f1c | Michele Tartara | Suspend tags can be added to cluster, nodegroup or instances, and act on the |
1482 | ee414f1c | Michele Tartara | entire entities sub-tree. No operation will be performed by ``harep`` on the |
1483 | ee414f1c | Michele Tartara | instances protected by a suspend tag. Their syntax is as follows:: |
1484 | ee414f1c | Michele Tartara | |
1485 | ee414f1c | Michele Tartara | ganeti:watcher:autorepair:suspend[:<timestamp>] |
1486 | ee414f1c | Michele Tartara | |
1487 | ee414f1c | Michele Tartara | If there are multiple suspend tags in an object, the form without timestamp |
1488 | ee414f1c | Michele Tartara | takes precedence (permanent suspension); or, if all object tags have a |
1489 | ee414f1c | Michele Tartara | timestamp, the one with the highest timestamp. |
1490 | ee414f1c | Michele Tartara | |
1491 | ee414f1c | Michele Tartara | Tags with a timestamp will be automatically removed when the time indicated by |
1492 | ee414f1c | Michele Tartara | the timestamp is passed. Indefinite suspension tags have to be removed manually. |
1493 | ee414f1c | Michele Tartara | |
1494 | ee414f1c | Michele Tartara | Result reporting |
1495 | ee414f1c | Michele Tartara | ++++++++++++++++ |
1496 | ee414f1c | Michele Tartara | |
1497 | ee414f1c | Michele Tartara | Harep will report about the result of its actions both through its CLI, and by |
1498 | ee414f1c | Michele Tartara | adding tags to the instances it operated on. Such tags will follow the syntax |
1499 | ee414f1c | Michele Tartara | hereby described:: |
1500 | ee414f1c | Michele Tartara | |
1501 | ee414f1c | Michele Tartara | ganeti:watcher:autorepair:result:<type>:<id>:<timestamp>:<result>:<jobs> |
1502 | ee414f1c | Michele Tartara | |
1503 | ee414f1c | Michele Tartara | If this tag is present a repair of type ``type`` has been performed on |
1504 | ee414f1c | Michele Tartara | the instance and has been completed by ``timestamp``. The result is |
1505 | ee414f1c | Michele Tartara | either ``success``, ``failure`` or ``enoperm``, and jobs is a |
1506 | ee414f1c | Michele Tartara | *+*-separated list of jobs that were executed for this repair. |
1507 | ee414f1c | Michele Tartara | |
1508 | ee414f1c | Michele Tartara | An ``enoperm`` result is an error state due to permission problems. It |
1509 | ee414f1c | Michele Tartara | is returned when the repair cannot proceed because it would require to perform |
1510 | ee414f1c | Michele Tartara | an operation that is not allowed by the ``ganeti:watcher:autorepair:<type>`` tag |
1511 | ee414f1c | Michele Tartara | that is defining the instance autorepair permissions. |
1512 | ee414f1c | Michele Tartara | |
1513 | ee414f1c | Michele Tartara | NB: if an instance repair ends up in a failure state, it will not be touched |
1514 | ee414f1c | Michele Tartara | again by ``harep`` until it has been manually fixed by the system administrator |
1515 | ee414f1c | Michele Tartara | and the ``ganeti:watcher:autorepair:result:failure:*`` tag has been manually |
1516 | ee414f1c | Michele Tartara | removed. |
1517 | c71a1a3d | Iustin Pop | |
1518 | c71a1a3d | Iustin Pop | Job operations |
1519 | c71a1a3d | Iustin Pop | -------------- |
1520 | c71a1a3d | Iustin Pop | |
1521 | c71a1a3d | Iustin Pop | The various jobs submitted by the instance/node/cluster commands can be |
1522 | c71a1a3d | Iustin Pop | examined, canceled and archived by various invocations of the |
1523 | c71a1a3d | Iustin Pop | ``gnt-job`` command. |
1524 | c71a1a3d | Iustin Pop | |
1525 | c71a1a3d | Iustin Pop | First is the job list command:: |
1526 | c71a1a3d | Iustin Pop | |
1527 | 73225861 | Iustin Pop | $ gnt-job list |
1528 | c71a1a3d | Iustin Pop | 17771 success INSTANCE_QUERY_DATA |
1529 | c71a1a3d | Iustin Pop | 17773 success CLUSTER_VERIFY_DISKS |
1530 | c71a1a3d | Iustin Pop | 17775 success CLUSTER_REPAIR_DISK_SIZES |
1531 | c71a1a3d | Iustin Pop | 17776 error CLUSTER_RENAME(cluster.example.com) |
1532 | c71a1a3d | Iustin Pop | 17780 success CLUSTER_REDIST_CONF |
1533 | c71a1a3d | Iustin Pop | 17792 success INSTANCE_REBOOT(instance1.example.com) |
1534 | c71a1a3d | Iustin Pop | |
1535 | c71a1a3d | Iustin Pop | More detailed information about a job can be found via the ``info`` |
1536 | c71a1a3d | Iustin Pop | command:: |
1537 | c71a1a3d | Iustin Pop | |
1538 | 73225861 | Iustin Pop | $ gnt-job info %17776% |
1539 | c71a1a3d | Iustin Pop | Job ID: 17776 |
1540 | c71a1a3d | Iustin Pop | Status: error |
1541 | c71a1a3d | Iustin Pop | Received: 2009-10-25 23:18:02.180569 |
1542 | c71a1a3d | Iustin Pop | Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s) |
1543 | c71a1a3d | Iustin Pop | Processing end: 2009-10-25 23:18:02.279743 (delta 0.079408s) |
1544 | c71a1a3d | Iustin Pop | Total processing time: 0.099174 seconds |
1545 | c71a1a3d | Iustin Pop | Opcodes: |
1546 | c71a1a3d | Iustin Pop | OP_CLUSTER_RENAME |
1547 | c71a1a3d | Iustin Pop | Status: error |
1548 | c71a1a3d | Iustin Pop | Processing start: 2009-10-25 23:18:02.200335 |
1549 | c71a1a3d | Iustin Pop | Processing end: 2009-10-25 23:18:02.252282 |
1550 | c71a1a3d | Iustin Pop | Input fields: |
1551 | c71a1a3d | Iustin Pop | name: cluster.example.com |
1552 | c71a1a3d | Iustin Pop | Result: |
1553 | c71a1a3d | Iustin Pop | OpPrereqError |
1554 | c71a1a3d | Iustin Pop | [Neither the name nor the IP address of the cluster has changed] |
1555 | c71a1a3d | Iustin Pop | Execution log: |
1556 | c71a1a3d | Iustin Pop | |
1557 | c71a1a3d | Iustin Pop | During the execution of a job, it's possible to follow the output of a |
1558 | c71a1a3d | Iustin Pop | job, similar to the log that one get from the ``gnt-`` commands, via the |
1559 | c71a1a3d | Iustin Pop | watch command:: |
1560 | c71a1a3d | Iustin Pop | |
1561 | 73225861 | Iustin Pop | $ gnt-instance add --submit โฆ %instance1% |
1562 | c71a1a3d | Iustin Pop | JobID: 17818 |
1563 | 73225861 | Iustin Pop | $ gnt-job watch %17818% |
1564 | c71a1a3d | Iustin Pop | Output from job 17818 follows |
1565 | c71a1a3d | Iustin Pop | ----------------------------- |
1566 | c71a1a3d | Iustin Pop | Mon Oct 26 00:22:48 2009 - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2 |
1567 | c71a1a3d | Iustin Pop | Mon Oct 26 00:22:49 2009 * creating instance disks... |
1568 | c71a1a3d | Iustin Pop | Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config |
1569 | c71a1a3d | Iustin Pop | Mon Oct 26 00:22:52 2009 - INFO: Waiting for instance instance1 to sync disks. |
1570 | c71a1a3d | Iustin Pop | โฆ |
1571 | e0897adf | Michael Hanselmann | Mon Oct 26 00:23:03 2009 creating os for instance instance1 on node node1 |
1572 | c71a1a3d | Iustin Pop | Mon Oct 26 00:23:03 2009 * running the instance OS create scripts... |
1573 | c71a1a3d | Iustin Pop | Mon Oct 26 00:23:13 2009 * starting instance... |
1574 | 73225861 | Iustin Pop | $ |
1575 | c71a1a3d | Iustin Pop | |
1576 | c71a1a3d | Iustin Pop | This is useful if you need to follow a job's progress from multiple |
1577 | c71a1a3d | Iustin Pop | terminals. |
1578 | c71a1a3d | Iustin Pop | |
1579 | c71a1a3d | Iustin Pop | A job that has not yet started to run can be canceled:: |
1580 | c71a1a3d | Iustin Pop | |
1581 | 73225861 | Iustin Pop | $ gnt-job cancel %17810% |
1582 | c71a1a3d | Iustin Pop | |
1583 | c71a1a3d | Iustin Pop | But not one that has already started execution:: |
1584 | c71a1a3d | Iustin Pop | |
1585 | 73225861 | Iustin Pop | $ gnt-job cancel %17805% |
1586 | c71a1a3d | Iustin Pop | Job 17805 is no longer waiting in the queue |
1587 | c71a1a3d | Iustin Pop | |
1588 | c71a1a3d | Iustin Pop | There are two queues for jobs: the *current* and the *archive* |
1589 | c71a1a3d | Iustin Pop | queue. Jobs are initially submitted to the current queue, and they stay |
1590 | c71a1a3d | Iustin Pop | in that queue until they have finished execution (either successfully or |
1591 | 89907375 | Michael Hanselmann | not). At that point, they can be moved into the archive queue using e.g. |
1592 | 89907375 | Michael Hanselmann | ``gnt-job autoarchive all``. The ``ganeti-watcher`` script will do this |
1593 | 89907375 | Michael Hanselmann | automatically 6 hours after a job is finished. The ``ganeti-cleaner`` |
1594 | 89907375 | Michael Hanselmann | script will then remove archived the jobs from the archive directory |
1595 | c71a1a3d | Iustin Pop | after three weeks. |
1596 | c71a1a3d | Iustin Pop | |
1597 | 89907375 | Michael Hanselmann | Note that ``gnt-job list`` only shows jobs in the current queue. |
1598 | 89907375 | Michael Hanselmann | Archived jobs can be viewed using ``gnt-job info <id>``. |
1599 | c71a1a3d | Iustin Pop | |
1600 | bde65914 | Iustin Pop | Special Ganeti deployments |
1601 | bde65914 | Iustin Pop | -------------------------- |
1602 | bde65914 | Iustin Pop | |
1603 | bde65914 | Iustin Pop | Since Ganeti 2.4, it is possible to extend the Ganeti deployment with |
1604 | bde65914 | Iustin Pop | two custom scenarios: Ganeti inside Ganeti and multi-site model. |
1605 | bde65914 | Iustin Pop | |
1606 | bde65914 | Iustin Pop | Running Ganeti under Ganeti |
1607 | bde65914 | Iustin Pop | +++++++++++++++++++++++++++ |
1608 | bde65914 | Iustin Pop | |
1609 | bde65914 | Iustin Pop | It is sometimes useful to be able to use a Ganeti instance as a Ganeti |
1610 | bde65914 | Iustin Pop | node (part of another cluster, usually). One example scenario is two |
1611 | bde65914 | Iustin Pop | small clusters, where we want to have an additional master candidate |
1612 | bde65914 | Iustin Pop | that holds the cluster configuration and can be used for helping with |
1613 | bde65914 | Iustin Pop | the master voting process. |
1614 | bde65914 | Iustin Pop | |
1615 | bde65914 | Iustin Pop | However, these Ganeti instance should not host instances themselves, and |
1616 | bde65914 | Iustin Pop | should not be considered in the normal capacity planning, evacuation |
1617 | bde65914 | Iustin Pop | strategies, etc. In order to accomplish this, mark these nodes as |
1618 | bde65914 | Iustin Pop | non-``vm_capable``:: |
1619 | bde65914 | Iustin Pop | |
1620 | 73225861 | Iustin Pop | $ gnt-node modify --vm-capable=no %node3% |
1621 | bde65914 | Iustin Pop | |
1622 | bde65914 | Iustin Pop | The vm_capable status can be listed as usual via ``gnt-node list``:: |
1623 | bde65914 | Iustin Pop | |
1624 | 73225861 | Iustin Pop | $ gnt-node list -oname,vm_capable |
1625 | bde65914 | Iustin Pop | Node VMCapable |
1626 | bde65914 | Iustin Pop | node1 Y |
1627 | bde65914 | Iustin Pop | node2 Y |
1628 | bde65914 | Iustin Pop | node3 N |
1629 | bde65914 | Iustin Pop | |
1630 | bde65914 | Iustin Pop | When this flag is set, the cluster will not do any operations that |
1631 | bde65914 | Iustin Pop | relate to instances on such nodes, e.g. hypervisor operations, |
1632 | bde65914 | Iustin Pop | disk-related operations, etc. Basically they will just keep the ssconf |
1633 | bde65914 | Iustin Pop | files, and if master candidates the full configuration. |
1634 | bde65914 | Iustin Pop | |
1635 | bde65914 | Iustin Pop | Multi-site model |
1636 | bde65914 | Iustin Pop | ++++++++++++++++ |
1637 | bde65914 | Iustin Pop | |
1638 | bde65914 | Iustin Pop | If Ganeti is deployed in multi-site model, with each site being a node |
1639 | bde65914 | Iustin Pop | group (so that instances are not relocated across the WAN by mistake), |
1640 | bde65914 | Iustin Pop | it is conceivable that either the WAN latency is high or that some sites |
1641 | bde65914 | Iustin Pop | have a lower reliability than others. In this case, it doesn't make |
1642 | bde65914 | Iustin Pop | sense to replicate the job information across all sites (or even outside |
1643 | bde65914 | Iustin Pop | of a โcentralโ node group), so it should be possible to restrict which |
1644 | bde65914 | Iustin Pop | nodes can become master candidates via the auto-promotion algorithm. |
1645 | bde65914 | Iustin Pop | |
1646 | bde65914 | Iustin Pop | Ganeti 2.4 introduces for this purpose a new ``master_capable`` flag, |
1647 | bde65914 | Iustin Pop | which (when unset) prevents nodes from being marked as master |
1648 | bde65914 | Iustin Pop | candidates, either manually or automatically. |
1649 | bde65914 | Iustin Pop | |
1650 | bde65914 | Iustin Pop | As usual, the node modify operation can change this flag:: |
1651 | bde65914 | Iustin Pop | |
1652 | 73225861 | Iustin Pop | $ gnt-node modify --auto-promote --master-capable=no %node3% |
1653 | bde65914 | Iustin Pop | Fri Jan 7 06:23:07 2011 - INFO: Demoting from master candidate |
1654 | bde65914 | Iustin Pop | Fri Jan 7 06:23:08 2011 - INFO: Promoted nodes to master candidate role: node4 |
1655 | bde65914 | Iustin Pop | Modified node node3 |
1656 | bde65914 | Iustin Pop | - master_capable -> False |
1657 | bde65914 | Iustin Pop | - master_candidate -> False |
1658 | bde65914 | Iustin Pop | |
1659 | bde65914 | Iustin Pop | And the node list operation will list this flag:: |
1660 | bde65914 | Iustin Pop | |
1661 | 73225861 | Iustin Pop | $ gnt-node list -oname,master_capable %node1% %node2% %node3% |
1662 | bde65914 | Iustin Pop | Node MasterCapable |
1663 | bde65914 | Iustin Pop | node1 Y |
1664 | bde65914 | Iustin Pop | node2 Y |
1665 | bde65914 | Iustin Pop | node3 N |
1666 | bde65914 | Iustin Pop | |
1667 | bde65914 | Iustin Pop | Note that marking a node both not ``vm_capable`` and not |
1668 | bde65914 | Iustin Pop | ``master_capable`` makes the node practically unusable from Ganeti's |
1669 | bde65914 | Iustin Pop | point of view. Hence these two flags should be used probably in |
1670 | bde65914 | Iustin Pop | contrast: some nodes will be only master candidates (master_capable but |
1671 | bde65914 | Iustin Pop | not vm_capable), and other nodes will only hold instances (vm_capable |
1672 | bde65914 | Iustin Pop | but not master_capable). |
1673 | bde65914 | Iustin Pop | |
1674 | bde65914 | Iustin Pop | |
1675 | c71a1a3d | Iustin Pop | Ganeti tools |
1676 | c71a1a3d | Iustin Pop | ------------ |
1677 | c71a1a3d | Iustin Pop | |
1678 | c71a1a3d | Iustin Pop | Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided |
1679 | c71a1a3d | Iustin Pop | and installed in ``$prefix/sbin`` at install time, there are a couple of |
1680 | c71a1a3d | Iustin Pop | other tools installed which are used seldom but can be helpful in some |
1681 | c71a1a3d | Iustin Pop | cases. |
1682 | c71a1a3d | Iustin Pop | |
1683 | c71a1a3d | Iustin Pop | lvmstrap |
1684 | c71a1a3d | Iustin Pop | ++++++++ |
1685 | c71a1a3d | Iustin Pop | |
1686 | c71a1a3d | Iustin Pop | The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section, |
1687 | c71a1a3d | Iustin Pop | has two modes of operation: |
1688 | c71a1a3d | Iustin Pop | |
1689 | c71a1a3d | Iustin Pop | - ``diskinfo`` shows the discovered disks on the system and their status |
1690 | c71a1a3d | Iustin Pop | - ``create`` takes all not-in-use disks and creates a volume group out |
1691 | c71a1a3d | Iustin Pop | of them |
1692 | c71a1a3d | Iustin Pop | |
1693 | c71a1a3d | Iustin Pop | .. warning:: The ``create`` argument to this command causes data-loss! |
1694 | c71a1a3d | Iustin Pop | |
1695 | c71a1a3d | Iustin Pop | cfgupgrade |
1696 | c71a1a3d | Iustin Pop | ++++++++++ |
1697 | c71a1a3d | Iustin Pop | |
1698 | c71a1a3d | Iustin Pop | The ``cfgupgrade`` tools is used to upgrade between major (and minor) |
1699 | 91fb0d18 | Bernardo Dal Seno | Ganeti versions, and to roll back. Point-releases are usually |
1700 | 91fb0d18 | Bernardo Dal Seno | transparent for the admin. |
1701 | c71a1a3d | Iustin Pop | |
1702 | c71a1a3d | Iustin Pop | More information about the upgrade procedure is listed on the wiki at |
1703 | c71a1a3d | Iustin Pop | http://code.google.com/p/ganeti/wiki/UpgradeNotes. |
1704 | c71a1a3d | Iustin Pop | |
1705 | b5672ea0 | Iustin Pop | There is also a script designed to upgrade from Ganeti 1.2 to 2.0, |
1706 | b5672ea0 | Iustin Pop | called ``cfgupgrade12``. |
1707 | b5672ea0 | Iustin Pop | |
1708 | c71a1a3d | Iustin Pop | cfgshell |
1709 | c71a1a3d | Iustin Pop | ++++++++ |
1710 | c71a1a3d | Iustin Pop | |
1711 | c71a1a3d | Iustin Pop | .. note:: This command is not actively maintained; make sure you backup |
1712 | c71a1a3d | Iustin Pop | your configuration before using it |
1713 | c71a1a3d | Iustin Pop | |
1714 | c71a1a3d | Iustin Pop | This can be used as an alternative to direct editing of the |
1715 | c71a1a3d | Iustin Pop | main configuration file if Ganeti has a bug and prevents you, for |
1716 | c71a1a3d | Iustin Pop | example, from removing an instance or a node from the configuration |
1717 | c71a1a3d | Iustin Pop | file. |
1718 | c71a1a3d | Iustin Pop | |
1719 | c71a1a3d | Iustin Pop | .. _burnin-label: |
1720 | c71a1a3d | Iustin Pop | |
1721 | c71a1a3d | Iustin Pop | burnin |
1722 | c71a1a3d | Iustin Pop | ++++++ |
1723 | c71a1a3d | Iustin Pop | |
1724 | c71a1a3d | Iustin Pop | .. warning:: This command will erase existing instances if given as |
1725 | c71a1a3d | Iustin Pop | arguments! |
1726 | c71a1a3d | Iustin Pop | |
1727 | c71a1a3d | Iustin Pop | This tool is used to exercise either the hardware of machines or |
1728 | c71a1a3d | Iustin Pop | alternatively the Ganeti software. It is safe to run on an existing |
1729 | c71a1a3d | Iustin Pop | cluster **as long as you don't pass it existing instance names**. |
1730 | c71a1a3d | Iustin Pop | |
1731 | c71a1a3d | Iustin Pop | The command will, by default, execute a comprehensive set of operations |
1732 | c71a1a3d | Iustin Pop | against a list of instances, these being: |
1733 | c71a1a3d | Iustin Pop | |
1734 | c71a1a3d | Iustin Pop | - creation |
1735 | c71a1a3d | Iustin Pop | - disk replacement (for redundant instances) |
1736 | c71a1a3d | Iustin Pop | - failover and migration (for redundant instances) |
1737 | c71a1a3d | Iustin Pop | - move (for non-redundant instances) |
1738 | c71a1a3d | Iustin Pop | - disk growth |
1739 | c71a1a3d | Iustin Pop | - add disks, remove disk |
1740 | c71a1a3d | Iustin Pop | - add NICs, remove NICs |
1741 | c71a1a3d | Iustin Pop | - export and then import |
1742 | c71a1a3d | Iustin Pop | - rename |
1743 | c71a1a3d | Iustin Pop | - reboot |
1744 | c71a1a3d | Iustin Pop | - shutdown/startup |
1745 | c71a1a3d | Iustin Pop | - and finally removal of the test instances |
1746 | c71a1a3d | Iustin Pop | |
1747 | c71a1a3d | Iustin Pop | Executing all these operations will test that the hardware performs |
1748 | c71a1a3d | Iustin Pop | well: the creation, disk replace, disk add and disk growth will exercise |
1749 | c71a1a3d | Iustin Pop | the storage and network; the migrate command will test the memory of the |
1750 | c71a1a3d | Iustin Pop | systems. Depending on the passed options, it can also test that the |
1751 | c71a1a3d | Iustin Pop | instance OS definitions are executing properly the rename, import and |
1752 | c71a1a3d | Iustin Pop | export operations. |
1753 | c71a1a3d | Iustin Pop | |
1754 | ea5fd476 | Iustin Pop | sanitize-config |
1755 | ea5fd476 | Iustin Pop | +++++++++++++++ |
1756 | ea5fd476 | Iustin Pop | |
1757 | ea5fd476 | Iustin Pop | This tool takes the Ganeti configuration and outputs a "sanitized" |
1758 | ea5fd476 | Iustin Pop | version, by randomizing or clearing: |
1759 | ea5fd476 | Iustin Pop | |
1760 | ea5fd476 | Iustin Pop | - DRBD secrets and cluster public key (always) |
1761 | ea5fd476 | Iustin Pop | - host names (optional) |
1762 | ea5fd476 | Iustin Pop | - IPs (optional) |
1763 | ea5fd476 | Iustin Pop | - OS names (optional) |
1764 | ea5fd476 | Iustin Pop | - LV names (optional, only useful for very old clusters which still have |
1765 | ea5fd476 | Iustin Pop | instances whose LVs are based on the instance name) |
1766 | ea5fd476 | Iustin Pop | |
1767 | ea5fd476 | Iustin Pop | By default, all optional items are activated except the LV name |
1768 | ea5fd476 | Iustin Pop | randomization. When passing ``--no-randomization``, which disables the |
1769 | ea5fd476 | Iustin Pop | optional items (i.e. just the DRBD secrets and cluster public keys are |
1770 | ea5fd476 | Iustin Pop | randomized), the resulting file can be used as a safety copy of the |
1771 | ea5fd476 | Iustin Pop | cluster config - while not trivial, the layout of the cluster can be |
1772 | ea5fd476 | Iustin Pop | recreated from it and if the instance disks have not been lost it |
1773 | ea5fd476 | Iustin Pop | permits recovery from the loss of all master candidates. |
1774 | ea5fd476 | Iustin Pop | |
1775 | e0897adf | Michael Hanselmann | move-instance |
1776 | e0897adf | Michael Hanselmann | +++++++++++++ |
1777 | e0897adf | Michael Hanselmann | |
1778 | e0897adf | Michael Hanselmann | See :doc:`separate documentation for move-instance <move-instance>`. |
1779 | e0897adf | Michael Hanselmann | |
1780 | 60af7882 | Michele Tartara | users-setup |
1781 | 60af7882 | Michele Tartara | +++++++++++ |
1782 | 60af7882 | Michele Tartara | |
1783 | 60af7882 | Michele Tartara | Ganeti can either be run entirely as root, or with every daemon running as |
1784 | 60af7882 | Michele Tartara | its own specific user (if the parameters ``--with-user-prefix`` and/or |
1785 | 60af7882 | Michele Tartara | ``--with-group-prefix`` have been specified at ``./configure``-time). |
1786 | 60af7882 | Michele Tartara | |
1787 | 60af7882 | Michele Tartara | In case split users are activated, they are required to exist on the system, |
1788 | 60af7882 | Michele Tartara | and they need to belong to the proper groups in order for the access |
1789 | 60af7882 | Michele Tartara | permissions to files and programs to be correct. |
1790 | 60af7882 | Michele Tartara | |
1791 | 60af7882 | Michele Tartara | The ``users-setup`` tool, when run, takes care of setting up the proper |
1792 | 60af7882 | Michele Tartara | users and groups. |
1793 | 60af7882 | Michele Tartara | |
1794 | 60af7882 | Michele Tartara | The tool does not accept any parameter, and requires root permissions to run. |
1795 | 60af7882 | Michele Tartara | |
1796 | e0897adf | Michael Hanselmann | .. TODO: document cluster-merge tool |
1797 | e0897adf | Michael Hanselmann | |
1798 | ea5fd476 | Iustin Pop | |
1799 | c71a1a3d | Iustin Pop | Other Ganeti projects |
1800 | c71a1a3d | Iustin Pop | --------------------- |
1801 | c71a1a3d | Iustin Pop | |
1802 | 1ebe6dbd | Iustin Pop | Below is a list (which might not be up-to-date) of additional projects |
1803 | 1ebe6dbd | Iustin Pop | that can be useful in a Ganeti deployment. They can be downloaded from |
1804 | 1ebe6dbd | Iustin Pop | the project site (http://code.google.com/p/ganeti/) and the repositories |
1805 | 1ebe6dbd | Iustin Pop | are also on the project git site (http://git.ganeti.org). |
1806 | c71a1a3d | Iustin Pop | |
1807 | c71a1a3d | Iustin Pop | NBMA tools |
1808 | c71a1a3d | Iustin Pop | ++++++++++ |
1809 | c71a1a3d | Iustin Pop | |
1810 | c71a1a3d | Iustin Pop | The ``ganeti-nbma`` software is designed to allow instances to live on a |
1811 | c71a1a3d | Iustin Pop | separate, virtual network from the nodes, and in an environment where |
1812 | c71a1a3d | Iustin Pop | nodes are not guaranteed to be able to reach each other via multicasting |
1813 | c71a1a3d | Iustin Pop | or broadcasting. For more information see the README in the source |
1814 | c71a1a3d | Iustin Pop | archive. |
1815 | c71a1a3d | Iustin Pop | |
1816 | c71a1a3d | Iustin Pop | ganeti-htools |
1817 | c71a1a3d | Iustin Pop | +++++++++++++ |
1818 | c71a1a3d | Iustin Pop | |
1819 | 1ebe6dbd | Iustin Pop | Before Ganeti version 2.5, this was a standalone project; since that |
1820 | 1ebe6dbd | Iustin Pop | version it is integrated into the Ganeti codebase (see |
1821 | 1ebe6dbd | Iustin Pop | :doc:`install-quick` for instructions on how to enable it). If you run |
1822 | 1ebe6dbd | Iustin Pop | an older Ganeti version, you will have to download and build it |
1823 | 1ebe6dbd | Iustin Pop | separately. |
1824 | c71a1a3d | Iustin Pop | |
1825 | c71a1a3d | Iustin Pop | For more information and installation instructions, see the README file |
1826 | c71a1a3d | Iustin Pop | in the source archive. |
1827 | c71a1a3d | Iustin Pop | |
1828 | 558fd122 | Michael Hanselmann | .. vim: set textwidth=72 : |
1829 | c71a1a3d | Iustin Pop | .. Local Variables: |
1830 | c71a1a3d | Iustin Pop | .. mode: rst |
1831 | c71a1a3d | Iustin Pop | .. fill-column: 72 |
1832 | c71a1a3d | Iustin Pop | .. End: |