## root / doc / admin.rst @ d2baa21d

History | View | Annotate | Download (45.4 kB)

1 | ffa6869f | Iustin Pop | Ganeti administrator's guide |
---|---|---|---|

2 | ffa6869f | Iustin Pop | ============================ |

3 | ffa6869f | Iustin Pop | |

4 | fd07c6b3 | Iustin Pop | Documents Ganeti version |version| |

5 | ffa6869f | Iustin Pop | |

6 | ffa6869f | Iustin Pop | .. contents:: |

7 | ffa6869f | Iustin Pop | |

8 | c71a1a3d | Iustin Pop | .. highlight:: text |

9 | c71a1a3d | Iustin Pop | |

10 | ffa6869f | Iustin Pop | Introduction |

11 | ffa6869f | Iustin Pop | ------------ |

12 | ffa6869f | Iustin Pop | |

13 | c71a1a3d | Iustin Pop | Ganeti is a virtualization cluster management software. You are expected |

14 | c71a1a3d | Iustin Pop | to be a system administrator familiar with your Linux distribution and |

15 | c71a1a3d | Iustin Pop | the Xen or KVM virtualization environments before using it. |

16 | ffa6869f | Iustin Pop | |

17 | ffa6869f | Iustin Pop | The various components of Ganeti all have man pages and interactive |

18 | c71a1a3d | Iustin Pop | help. This manual though will help you getting familiar with the system |

19 | c71a1a3d | Iustin Pop | by explaining the most common operations, grouped by related use. |

20 | ffa6869f | Iustin Pop | |

21 | ffa6869f | Iustin Pop | After a terminology glossary and a section on the prerequisites needed |

22 | c71a1a3d | Iustin Pop | to use this manual, the rest of this document is divided in sections |

23 | c71a1a3d | Iustin Pop | for the different targets that a command affects: instance, nodes, etc. |

24 | ffa6869f | Iustin Pop | |

25 | c71a1a3d | Iustin Pop | .. _terminology-label: |

26 | ffa6869f | Iustin Pop | |

27 | ffa6869f | Iustin Pop | Ganeti terminology |

28 | c71a1a3d | Iustin Pop | ++++++++++++++++++ |

29 | ffa6869f | Iustin Pop | |

30 | c71a1a3d | Iustin Pop | This section provides a small introduction to Ganeti terminology, which |

31 | c71a1a3d | Iustin Pop | might be useful when reading the rest of the document. |

32 | ffa6869f | Iustin Pop | |

33 | ffa6869f | Iustin Pop | Cluster |

34 | c71a1a3d | Iustin Pop | ~~~~~~~ |

35 | ffa6869f | Iustin Pop | |

36 | c71a1a3d | Iustin Pop | A set of machines (nodes) that cooperate to offer a coherent, highly |

37 | c71a1a3d | Iustin Pop | available virtualization service under a single administration domain. |

38 | ffa6869f | Iustin Pop | |

39 | c71a1a3d | Iustin Pop | Node |

40 | c71a1a3d | Iustin Pop | ~~~~ |

41 | c71a1a3d | Iustin Pop | |

42 | c71a1a3d | Iustin Pop | A physical machine which is member of a cluster. Nodes are the basic |

43 | c71a1a3d | Iustin Pop | cluster infrastructure, and they don't need to be fault tolerant in |

44 | c71a1a3d | Iustin Pop | order to achieve high availability for instances. |

45 | c71a1a3d | Iustin Pop | |

46 | c71a1a3d | Iustin Pop | Node can be added and removed (if they host no instances) at will from |

47 | c71a1a3d | Iustin Pop | the cluster. In a HA cluster and only with HA instances, the loss of any |

48 | c71a1a3d | Iustin Pop | single node will not cause disk data loss for any instance; of course, |

49 | c71a1a3d | Iustin Pop | a node crash will cause the crash of the its primary instances. |

50 | c71a1a3d | Iustin Pop | |

51 | c71a1a3d | Iustin Pop | A node belonging to a cluster can be in one of the following roles at a |

52 | c71a1a3d | Iustin Pop | given time: |

53 | c71a1a3d | Iustin Pop | |

54 | c71a1a3d | Iustin Pop | - *master* node, which is the node from which the cluster is controlled |

55 | c71a1a3d | Iustin Pop | - *master candidate* node, only nodes in this role have the full cluster |

56 | c71a1a3d | Iustin Pop | configuration and knowledge, and only master candidates can become the |

57 | c71a1a3d | Iustin Pop | master node |

58 | c71a1a3d | Iustin Pop | - *regular* node, which is the state in which most nodes will be on |

59 | c71a1a3d | Iustin Pop | bigger clusters (>20 nodes) |

60 | c71a1a3d | Iustin Pop | - *drained* node, nodes in this state are functioning normally but the |

61 | c71a1a3d | Iustin Pop | cannot receive new instances; the intention is that nodes in this role |

62 | c71a1a3d | Iustin Pop | have some issue and they are being evacuated for hardware repairs |

63 | c71a1a3d | Iustin Pop | - *offline* node, in which there is a record in the cluster |

64 | c71a1a3d | Iustin Pop | configuration about the node, but the daemons on the master node will |

65 | c71a1a3d | Iustin Pop | not talk to this node; any instances declared as having an offline |

66 | c71a1a3d | Iustin Pop | node as either primary or secondary will be flagged as an error in the |

67 | c71a1a3d | Iustin Pop | cluster verify operation |

68 | c71a1a3d | Iustin Pop | |

69 | c71a1a3d | Iustin Pop | Depending on the role, each node will run a set of daemons: |

70 | c71a1a3d | Iustin Pop | |

71 | c71a1a3d | Iustin Pop | - the :command:`ganeti-noded` daemon, which control the manipulation of |

72 | c71a1a3d | Iustin Pop | this node's hardware resources; it runs on all nodes which are in a |

73 | c71a1a3d | Iustin Pop | cluster |

74 | c71a1a3d | Iustin Pop | - the :command:`ganeti-confd` daemon (Ganeti 2.1+) which runs on all |

75 | c71a1a3d | Iustin Pop | nodes, but is only functional on master candidate nodes |

76 | c71a1a3d | Iustin Pop | - the :command:`ganeti-rapi` daemon which runs on the master node and |

77 | c71a1a3d | Iustin Pop | offers an HTTP-based API for the cluster |

78 | c71a1a3d | Iustin Pop | - the :command:`ganeti-masterd` daemon which runs on the master node and |

79 | c71a1a3d | Iustin Pop | allows control of the cluster |

80 | ffa6869f | Iustin Pop | |

81 | ffa6869f | Iustin Pop | Instance |

82 | c71a1a3d | Iustin Pop | ~~~~~~~~ |

83 | ffa6869f | Iustin Pop | |

84 | c71a1a3d | Iustin Pop | A virtual machine which runs on a cluster. It can be a fault tolerant, |

85 | c71a1a3d | Iustin Pop | highly available entity. |

86 | ffa6869f | Iustin Pop | |

87 | c71a1a3d | Iustin Pop | An instance has various parameters, which are classified in three |

88 | c71a1a3d | Iustin Pop | categories: hypervisor related-parameters (called ``hvparams``), general |

89 | c71a1a3d | Iustin Pop | parameters (called ``beparams``) and per network-card parameters (called |

90 | c71a1a3d | Iustin Pop | ``nicparams``). All these parameters can be modified either at instance |

91 | c71a1a3d | Iustin Pop | level or via defaults at cluster level. |

92 | ffa6869f | Iustin Pop | |

93 | c71a1a3d | Iustin Pop | Disk template |

94 | ffa6869f | Iustin Pop | ~~~~~~~~~~~~~ |

95 | ffa6869f | Iustin Pop | |

96 | c71a1a3d | Iustin Pop | The are multiple options for the storage provided to an instance; while |

97 | c71a1a3d | Iustin Pop | the instance sees the same virtual drive in all cases, the node-level |

98 | c71a1a3d | Iustin Pop | configuration varies between them. |

99 | ffa6869f | Iustin Pop | |

100 | c71a1a3d | Iustin Pop | There are four disk templates you can choose from: |

101 | ffa6869f | Iustin Pop | |

102 | c71a1a3d | Iustin Pop | diskless |

103 | c71a1a3d | Iustin Pop | The instance has no disks. Only used for special purpose operating |

104 | c71a1a3d | Iustin Pop | systems or for testing. |

105 | c71a1a3d | Iustin Pop | |

106 | c71a1a3d | Iustin Pop | file |

107 | c71a1a3d | Iustin Pop | The instance will use plain files as backend for its disks. No |

108 | c71a1a3d | Iustin Pop | redundancy is provided, and this is somewhat more difficult to |

109 | c71a1a3d | Iustin Pop | configure for high performance. |

110 | c71a1a3d | Iustin Pop | |

111 | c71a1a3d | Iustin Pop | plain |

112 | c71a1a3d | Iustin Pop | The instance will use LVM devices as backend for its disks. No |

113 | c71a1a3d | Iustin Pop | redundancy is provided. |

114 | c71a1a3d | Iustin Pop | |

115 | c71a1a3d | Iustin Pop | drbd |

116 | c71a1a3d | Iustin Pop | .. note:: This is only valid for multi-node clusters using DRBD 8.0+ |

117 | c71a1a3d | Iustin Pop | |

118 | c71a1a3d | Iustin Pop | A mirror is set between the local node and a remote one, which must be |

119 | c71a1a3d | Iustin Pop | specified with the second value of the --node option. Use this option |

120 | c71a1a3d | Iustin Pop | to obtain a highly available instance that can be failed over to a |

121 | c71a1a3d | Iustin Pop | remote node should the primary one fail. |

122 | c71a1a3d | Iustin Pop | |

123 | c71a1a3d | Iustin Pop | IAllocator |

124 | c71a1a3d | Iustin Pop | ~~~~~~~~~~ |

125 | c71a1a3d | Iustin Pop | |

126 | c71a1a3d | Iustin Pop | A framework for using external (user-provided) scripts to compute the |

127 | c71a1a3d | Iustin Pop | placement of instances on the cluster nodes. This eliminates the need to |

128 | c71a1a3d | Iustin Pop | manually specify nodes in instance add, instance moves, node evacuate, |

129 | c71a1a3d | Iustin Pop | etc. |

130 | c71a1a3d | Iustin Pop | |

131 | c71a1a3d | Iustin Pop | In order for Ganeti to be able to use these scripts, they must be place |

132 | c71a1a3d | Iustin Pop | in the iallocator directory (usually ``lib/ganeti/iallocators`` under |

133 | c71a1a3d | Iustin Pop | the installation prefix, e.g. ``/usr/local``). |

134 | c71a1a3d | Iustin Pop | |

135 | c71a1a3d | Iustin Pop | “Primary” and “secondary” concepts |

136 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

137 | c71a1a3d | Iustin Pop | |

138 | c71a1a3d | Iustin Pop | An instance has a primary and depending on the disk configuration, might |

139 | c71a1a3d | Iustin Pop | also have a secondary node. The instance always runs on the primary node |

140 | c71a1a3d | Iustin Pop | and only uses its secondary node for disk replication. |

141 | c71a1a3d | Iustin Pop | |

142 | c71a1a3d | Iustin Pop | Similarly, the term of primary and secondary instances when talking |

143 | c71a1a3d | Iustin Pop | about a node refers to the set of instances having the given node as |

144 | c71a1a3d | Iustin Pop | primary, respectively secondary. |

145 | c71a1a3d | Iustin Pop | |

146 | c71a1a3d | Iustin Pop | Tags |

147 | c71a1a3d | Iustin Pop | ~~~~ |

148 | c71a1a3d | Iustin Pop | |

149 | c71a1a3d | Iustin Pop | Tags are short strings that can be attached to either to cluster itself, |

150 | c71a1a3d | Iustin Pop | or to nodes or instances. They are useful as a very simplistic |

151 | c71a1a3d | Iustin Pop | information store for helping with cluster administration, for example |

152 | c71a1a3d | Iustin Pop | by attaching owner information to each instance after it's created:: |

153 | ffa6869f | Iustin Pop | |

154 | c71a1a3d | Iustin Pop | gnt-instance add … instance1 |

155 | c71a1a3d | Iustin Pop | gnt-instance add-tags instance1 owner:user2 |

156 | c71a1a3d | Iustin Pop | |

157 | c71a1a3d | Iustin Pop | And then by listing each instance and its tags, this information could |

158 | c71a1a3d | Iustin Pop | be used for contacting the users of each instance. |

159 | c71a1a3d | Iustin Pop | |

160 | c71a1a3d | Iustin Pop | Jobs and OpCodes |

161 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~ |

162 | c71a1a3d | Iustin Pop | |

163 | c71a1a3d | Iustin Pop | While not directly visible by an end-user, it's useful to know that a |

164 | c71a1a3d | Iustin Pop | basic cluster operation (e.g. starting an instance) is represented |

165 | c71a1a3d | Iustin Pop | internall by Ganeti as an *OpCode* (abbreviation from operation |

166 | c71a1a3d | Iustin Pop | code). These OpCodes are executed as part of a *Job*. The OpCodes in a |

167 | c71a1a3d | Iustin Pop | single Job are processed serially by Ganeti, but different Jobs will be |

168 | c71a1a3d | Iustin Pop | processed (depending on resource availability) in parallel. |

169 | c71a1a3d | Iustin Pop | |

170 | c71a1a3d | Iustin Pop | For example, shutting down the entire cluster can be done by running the |

171 | c71a1a3d | Iustin Pop | command ``gnt-instance shutdown --all``, which will submit for each |

172 | c71a1a3d | Iustin Pop | instance a separate job containing the “shutdown instance” OpCode. |

173 | c71a1a3d | Iustin Pop | |

174 | c71a1a3d | Iustin Pop | |

175 | c71a1a3d | Iustin Pop | Prerequisites |

176 | c71a1a3d | Iustin Pop | +++++++++++++ |

177 | c71a1a3d | Iustin Pop | |

178 | c71a1a3d | Iustin Pop | You need to have your Ganeti cluster installed and configured before you |

179 | c71a1a3d | Iustin Pop | try any of the commands in this document. Please follow the |

180 | c71a1a3d | Iustin Pop | :doc:`install` for instructions on how to do that. |

181 | c71a1a3d | Iustin Pop | |

182 | c71a1a3d | Iustin Pop | Instance management |

183 | c71a1a3d | Iustin Pop | ------------------- |

184 | c71a1a3d | Iustin Pop | |

185 | c71a1a3d | Iustin Pop | Adding an instance |

186 | c71a1a3d | Iustin Pop | ++++++++++++++++++ |

187 | c71a1a3d | Iustin Pop | |

188 | c71a1a3d | Iustin Pop | The add operation might seem complex due to the many parameters it |

189 | c71a1a3d | Iustin Pop | accepts, but once you have understood the (few) required parameters and |

190 | c71a1a3d | Iustin Pop | the customisation capabilities you will see it is an easy operation. |

191 | c71a1a3d | Iustin Pop | |

192 | c71a1a3d | Iustin Pop | The add operation requires at minimum five parameters: |

193 | c71a1a3d | Iustin Pop | |

194 | c71a1a3d | Iustin Pop | - the OS for the instance |

195 | c71a1a3d | Iustin Pop | - the disk template |

196 | c71a1a3d | Iustin Pop | - the disk count and size |

197 | c71a1a3d | Iustin Pop | - the node specification or alternatively the iallocator to use |

198 | c71a1a3d | Iustin Pop | - and finally the instance name |

199 | c71a1a3d | Iustin Pop | |

200 | c71a1a3d | Iustin Pop | The OS for the instance must be visible in the output of the command |

201 | c71a1a3d | Iustin Pop | ``gnt-os list`` and specifies which guest OS to install on the instance. |

202 | c71a1a3d | Iustin Pop | |

203 | c71a1a3d | Iustin Pop | The disk template specifies what kind of storage to use as backend for |

204 | c71a1a3d | Iustin Pop | the (virtual) disks presented to the instance; note that for instances |

205 | c71a1a3d | Iustin Pop | with multiple virtual disks, they all must be of the same type. |

206 | c71a1a3d | Iustin Pop | |

207 | c71a1a3d | Iustin Pop | The node(s) on which the instance will run can be given either manually, |

208 | c71a1a3d | Iustin Pop | via the ``-n`` option, or computed automatically by Ganeti, if you have |

209 | c71a1a3d | Iustin Pop | installed any iallocator script. |

210 | c71a1a3d | Iustin Pop | |

211 | c71a1a3d | Iustin Pop | With the above parameters in mind, the command is:: |

212 | ffa6869f | Iustin Pop | |

213 | ffa6869f | Iustin Pop | gnt-instance add \ |

214 | c71a1a3d | Iustin Pop | -n TARGET_NODE:SECONDARY_NODE \ |

215 | c71a1a3d | Iustin Pop | -o OS_TYPE \ |

216 | c71a1a3d | Iustin Pop | -t DISK_TEMPLATE -s DISK_SIZE \ |

217 | ffa6869f | Iustin Pop | INSTANCE_NAME |

218 | ffa6869f | Iustin Pop | |

219 | ffa6869f | Iustin Pop | The instance name must be resolvable (e.g. exist in DNS) and usually |

220 | c71a1a3d | Iustin Pop | points to an address in the same subnet as the cluster itself. |

221 | ffa6869f | Iustin Pop | |

222 | c71a1a3d | Iustin Pop | The above command has the minimum required options; other options you |

223 | c71a1a3d | Iustin Pop | can give include, among others: |

224 | ffa6869f | Iustin Pop | |

225 | ffa6869f | Iustin Pop | - The memory size (``-B memory``) |

226 | ffa6869f | Iustin Pop | |

227 | ffa6869f | Iustin Pop | - The number of virtual CPUs (``-B vcpus``) |

228 | ffa6869f | Iustin Pop | |

229 | ffa6869f | Iustin Pop | - Arguments for the NICs of the instance; by default, a single-NIC |

230 | ffa6869f | Iustin Pop | instance is created. The IP and/or bridge of the NIC can be changed |

231 | ffa6869f | Iustin Pop | via ``--nic 0:ip=IP,bridge=BRIDGE`` |

232 | ffa6869f | Iustin Pop | |

233 | c71a1a3d | Iustin Pop | See the manpage for gnt-instance for the detailed option list. |

234 | ffa6869f | Iustin Pop | |

235 | c71a1a3d | Iustin Pop | For example if you want to create an highly available instance, with a |

236 | c71a1a3d | Iustin Pop | single disk of 50GB and the default memory size, having primary node |

237 | c71a1a3d | Iustin Pop | ``node1`` and secondary node ``node3``, use the following command:: |

238 | ffa6869f | Iustin Pop | |

239 | c71a1a3d | Iustin Pop | gnt-instance add -n node1:node3 -o debootstrap -t drbd \ |

240 | c71a1a3d | Iustin Pop | instance1 |

241 | ffa6869f | Iustin Pop | |

242 | c71a1a3d | Iustin Pop | There is a also a command for batch instance creation from a |

243 | c71a1a3d | Iustin Pop | specification file, see the ``batch-create`` operation in the |

244 | c71a1a3d | Iustin Pop | gnt-instance manual page. |

245 | ffa6869f | Iustin Pop | |

246 | c71a1a3d | Iustin Pop | Regular instance operations |

247 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++++ |

248 | ffa6869f | Iustin Pop | |

249 | c71a1a3d | Iustin Pop | Removal |

250 | c71a1a3d | Iustin Pop | ~~~~~~~ |

251 | ffa6869f | Iustin Pop | |

252 | c71a1a3d | Iustin Pop | Removing an instance is even easier than creating one. This operation is |

253 | c71a1a3d | Iustin Pop | irreversible and destroys all the contents of your instance. Use with |

254 | c71a1a3d | Iustin Pop | care:: |

255 | ffa6869f | Iustin Pop | |

256 | ffa6869f | Iustin Pop | gnt-instance remove INSTANCE_NAME |

257 | ffa6869f | Iustin Pop | |

258 | c71a1a3d | Iustin Pop | Startup/shutdown |

259 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~ |

260 | ffa6869f | Iustin Pop | |

261 | ffa6869f | Iustin Pop | Instances are automatically started at instance creation time. To |

262 | ffa6869f | Iustin Pop | manually start one which is currently stopped you can run:: |

263 | ffa6869f | Iustin Pop | |

264 | ffa6869f | Iustin Pop | gnt-instance startup INSTANCE_NAME |

265 | ffa6869f | Iustin Pop | |

266 | ffa6869f | Iustin Pop | While the command to stop one is:: |

267 | ffa6869f | Iustin Pop | |

268 | ffa6869f | Iustin Pop | gnt-instance shutdown INSTANCE_NAME |

269 | ffa6869f | Iustin Pop | |

270 | c71a1a3d | Iustin Pop | .. warning:: Do not use the Xen or KVM commands directly to stop |

271 | c71a1a3d | Iustin Pop | instances. If you run for example ``xm shutdown`` or ``xm destroy`` |

272 | c71a1a3d | Iustin Pop | on an instance Ganeti will automatically restart it (via the |

273 | c71a1a3d | Iustin Pop | :command:`ganeti-watcher` command which is launched via cron). |

274 | c71a1a3d | Iustin Pop | |

275 | c71a1a3d | Iustin Pop | Querying instances |

276 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~ |

277 | c71a1a3d | Iustin Pop | |

278 | c71a1a3d | Iustin Pop | There are two ways to get information about instances: listing |

279 | c71a1a3d | Iustin Pop | instances, which does a tabular output containing a given set of fields |

280 | c71a1a3d | Iustin Pop | about each instance, and querying detailed information about a set of |

281 | c71a1a3d | Iustin Pop | instances. |

282 | c71a1a3d | Iustin Pop | |

283 | ffa6869f | Iustin Pop | The command to see all the instances configured and their status is:: |

284 | ffa6869f | Iustin Pop | |

285 | ffa6869f | Iustin Pop | gnt-instance list |

286 | ffa6869f | Iustin Pop | |

287 | c71a1a3d | Iustin Pop | The command can return a custom set of information when using the ``-o`` |

288 | c71a1a3d | Iustin Pop | option (as always, check the manpage for a detailed specification). Each |

289 | c71a1a3d | Iustin Pop | instance will be represented on a line, thus making it easy to parse |

290 | c71a1a3d | Iustin Pop | this output via the usual shell utilities (grep, sed, etc.). |

291 | c71a1a3d | Iustin Pop | |

292 | c71a1a3d | Iustin Pop | To get more detailed information about an instance, you can run:: |

293 | c71a1a3d | Iustin Pop | |

294 | c71a1a3d | Iustin Pop | gnt-instance info INSTANCE |

295 | c71a1a3d | Iustin Pop | |

296 | c71a1a3d | Iustin Pop | which will give a multi-line block of information about the instance, |

297 | c71a1a3d | Iustin Pop | it's hardware resources (especially its disks and their redundancy |

298 | c71a1a3d | Iustin Pop | status), etc. This is harder to parse and is more expensive than the |

299 | c71a1a3d | Iustin Pop | list operation, but returns much more detailed information. |

300 | ffa6869f | Iustin Pop | |

301 | ffa6869f | Iustin Pop | |

302 | c71a1a3d | Iustin Pop | Export/Import |

303 | c71a1a3d | Iustin Pop | +++++++++++++ |

304 | c71a1a3d | Iustin Pop | |

305 | c71a1a3d | Iustin Pop | You can create a snapshot of an instance disk and its Ganeti |

306 | ffa6869f | Iustin Pop | configuration, which then you can backup, or import into another |

307 | ffa6869f | Iustin Pop | cluster. The way to export an instance is:: |

308 | ffa6869f | Iustin Pop | |

309 | ffa6869f | Iustin Pop | gnt-backup export -n TARGET_NODE INSTANCE_NAME |

310 | ffa6869f | Iustin Pop | |

311 | c71a1a3d | Iustin Pop | |

312 | ffa6869f | Iustin Pop | The target node can be any node in the cluster with enough space under |

313 | c71a1a3d | Iustin Pop | ``/srv/ganeti`` to hold the instance image. Use the ``--noshutdown`` |

314 | c71a1a3d | Iustin Pop | option to snapshot an instance without rebooting it. Note that Ganeti |

315 | c71a1a3d | Iustin Pop | only keeps one snapshot for an instance - any previous snapshot of the |

316 | c71a1a3d | Iustin Pop | same instance existing cluster-wide under ``/srv/ganeti`` will be |

317 | c71a1a3d | Iustin Pop | removed by this operation: if you want to keep them, you need to move |

318 | c71a1a3d | Iustin Pop | them out of the Ganeti exports directory. |

319 | ffa6869f | Iustin Pop | |

320 | c71a1a3d | Iustin Pop | Importing an instance is similar to creating a new one, but additionally |

321 | c71a1a3d | Iustin Pop | one must specify the location of the snapshot. The command is:: |

322 | ffa6869f | Iustin Pop | |

323 | ffa6869f | Iustin Pop | gnt-backup import -n TARGET_NODE -t DISK_TEMPLATE \ |

324 | ffa6869f | Iustin Pop | --src-node=NODE --src-dir=DIR INSTANCE_NAME |

325 | ffa6869f | Iustin Pop | |

326 | fd07c6b3 | Iustin Pop | Most of the options available for the command :command:`gnt-instance |

327 | fd07c6b3 | Iustin Pop | add` are supported here too. |

328 | ffa6869f | Iustin Pop | |

329 | c71a1a3d | Iustin Pop | Instance HA features |

330 | c71a1a3d | Iustin Pop | -------------------- |

331 | ffa6869f | Iustin Pop | |

332 | ffa6869f | Iustin Pop | .. note:: This section only applies to multi-node clusters |

333 | ffa6869f | Iustin Pop | |

334 | c71a1a3d | Iustin Pop | .. _instance-change-primary-label: |

335 | c71a1a3d | Iustin Pop | |

336 | c71a1a3d | Iustin Pop | Changing the primary node |

337 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++ |

338 | c71a1a3d | Iustin Pop | |

339 | c71a1a3d | Iustin Pop | There are three ways to exchange an instance's primary and secondary |

340 | c71a1a3d | Iustin Pop | nodes; the right one to choose depends on how the instance has been |

341 | c71a1a3d | Iustin Pop | created and the status of its current primary node. See |

342 | c71a1a3d | Iustin Pop | :ref:`rest-redundancy-label` for information on changing the secondary |

343 | c71a1a3d | Iustin Pop | node. Note that it's only possible to change the primary node to the |

344 | c71a1a3d | Iustin Pop | secondary and vice-versa; a direct change of the primary node with a |

345 | c71a1a3d | Iustin Pop | third node, while keeping the current secondary is not possible in a |

346 | c71a1a3d | Iustin Pop | single step, only via multiple operations as detailed in |

347 | c71a1a3d | Iustin Pop | :ref:`instance-relocation-label`. |

348 | c71a1a3d | Iustin Pop | |

349 | ffa6869f | Iustin Pop | Failing over an instance |

350 | ffa6869f | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~ |

351 | ffa6869f | Iustin Pop | |

352 | ffa6869f | Iustin Pop | If an instance is built in highly available mode you can at any time |

353 | ffa6869f | Iustin Pop | fail it over to its secondary node, even if the primary has somehow |

354 | ffa6869f | Iustin Pop | failed and it's not up anymore. Doing it is really easy, on the master |

355 | ffa6869f | Iustin Pop | node you can just run:: |

356 | ffa6869f | Iustin Pop | |

357 | ffa6869f | Iustin Pop | gnt-instance failover INSTANCE_NAME |

358 | ffa6869f | Iustin Pop | |

359 | ffa6869f | Iustin Pop | That's it. After the command completes the secondary node is now the |

360 | c71a1a3d | Iustin Pop | primary, and vice-versa. |

361 | ffa6869f | Iustin Pop | |

362 | ffa6869f | Iustin Pop | Live migrating an instance |

363 | ffa6869f | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |

364 | ffa6869f | Iustin Pop | |

365 | c71a1a3d | Iustin Pop | If an instance is built in highly available mode, it currently runs and |

366 | c71a1a3d | Iustin Pop | both its nodes are running fine, you can at migrate it over to its |

367 | c71a1a3d | Iustin Pop | secondary node, without downtime. On the master node you need to run:: |

368 | ffa6869f | Iustin Pop | |

369 | ffa6869f | Iustin Pop | gnt-instance migrate INSTANCE_NAME |

370 | ffa6869f | Iustin Pop | |

371 | c71a1a3d | Iustin Pop | The current load on the instance and its memory size will influence how |

372 | c71a1a3d | Iustin Pop | long the migration will take. In any case, for both KVM and Xen |

373 | c71a1a3d | Iustin Pop | hypervisors, the migration will be transparent to the instance. |

374 | ffa6869f | Iustin Pop | |

375 | c71a1a3d | Iustin Pop | Moving an instance (offline) |

376 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

377 | ffa6869f | Iustin Pop | |

378 | c71a1a3d | Iustin Pop | If an instance has not been create as mirrored, then the only way to |

379 | c71a1a3d | Iustin Pop | change its primary node is to execute the move command:: |

380 | ffa6869f | Iustin Pop | |

381 | c71a1a3d | Iustin Pop | gnt-instance move -n NEW_NODE INSTANCE |

382 | ffa6869f | Iustin Pop | |

383 | c71a1a3d | Iustin Pop | This has a few prerequisites: |

384 | ffa6869f | Iustin Pop | |

385 | c71a1a3d | Iustin Pop | - the instance must be stopped |

386 | c71a1a3d | Iustin Pop | - its current primary node must be on-line and healthy |

387 | c71a1a3d | Iustin Pop | - the disks of the instance must not have any errors |

388 | ffa6869f | Iustin Pop | |

389 | c71a1a3d | Iustin Pop | Since this operation actually copies the data from the old node to the |

390 | c71a1a3d | Iustin Pop | new node, expect it to take proportional to the size of the instance's |

391 | c71a1a3d | Iustin Pop | disks and the speed of both the nodes' I/O system and their networking. |

392 | ffa6869f | Iustin Pop | |

393 | c71a1a3d | Iustin Pop | Disk operations |

394 | c71a1a3d | Iustin Pop | +++++++++++++++ |

395 | ffa6869f | Iustin Pop | |

396 | c71a1a3d | Iustin Pop | Disk failures are a common cause of errors in any server |

397 | c71a1a3d | Iustin Pop | deployment. Ganeti offers protection from single-node failure if your |

398 | c71a1a3d | Iustin Pop | instances were created in HA mode, and it also offers ways to restore |

399 | c71a1a3d | Iustin Pop | redundancy after a failure. |

400 | ffa6869f | Iustin Pop | |

401 | c71a1a3d | Iustin Pop | Preparing for disk operations |

402 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

403 | ffa6869f | Iustin Pop | |

404 | c71a1a3d | Iustin Pop | It is important to note that for Ganeti to be able to do any disk |

405 | c71a1a3d | Iustin Pop | operation, the Linux machines on top of which Ganeti must be consistent; |

406 | c71a1a3d | Iustin Pop | for LVM, this means that the LVM commands must not return failures; it |

407 | c71a1a3d | Iustin Pop | is common that after a complete disk failure, any LVM command aborts |

408 | c71a1a3d | Iustin Pop | with an error similar to:: |

409 | ffa6869f | Iustin Pop | |

410 | c71a1a3d | Iustin Pop | # vgs |

411 | c71a1a3d | Iustin Pop | /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error |

412 | c71a1a3d | Iustin Pop | /dev/sdb1: read failed after 0 of 4096 at 750153695232: Input/output |

413 | c71a1a3d | Iustin Pop | error |

414 | c71a1a3d | Iustin Pop | /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error |

415 | c71a1a3d | Iustin Pop | Couldn't find device with uuid |

416 | c71a1a3d | Iustin Pop | 't30jmN-4Rcf-Fr5e-CURS-pawt-z0jU-m1TgeJ'. |

417 | c71a1a3d | Iustin Pop | Couldn't find all physical volumes for volume group xenvg. |

418 | ffa6869f | Iustin Pop | |

419 | c71a1a3d | Iustin Pop | Before restoring an instance's disks to healthy status, it's needed to |

420 | c71a1a3d | Iustin Pop | fix the volume group used by Ganeti so that we can actually create and |

421 | c71a1a3d | Iustin Pop | manage the logical volumes. This is usually done in a multi-step |

422 | c71a1a3d | Iustin Pop | process: |

423 | ffa6869f | Iustin Pop | |

424 | c71a1a3d | Iustin Pop | #. first, if the disk is completely gone and LVM commands exit with |

425 | c71a1a3d | Iustin Pop | “Couldn't find device with uuid…” then you need to run the command:: |

426 | c71a1a3d | Iustin Pop | |

427 | c71a1a3d | Iustin Pop | vgreduce --removemissing VOLUME_GROUP |

428 | c71a1a3d | Iustin Pop | |

429 | c71a1a3d | Iustin Pop | #. after the above command, the LVM commands should be executing |

430 | c71a1a3d | Iustin Pop | normally (warnings are normal, but the commands will not fail |

431 | c71a1a3d | Iustin Pop | completely). |

432 | c71a1a3d | Iustin Pop | |

433 | c71a1a3d | Iustin Pop | #. if the failed disk is still visible in the output of the ``pvs`` |

434 | c71a1a3d | Iustin Pop | command, you need to deactivate it from allocations by running:: |

435 | c71a1a3d | Iustin Pop | |

436 | c71a1a3d | Iustin Pop | pvs -x n /dev/DISK |

437 | ffa6869f | Iustin Pop | |

438 | c71a1a3d | Iustin Pop | At this point, the volume group should be consistent and any bad |

439 | c71a1a3d | Iustin Pop | physical volumes should not longer be available for allocation. |

440 | c71a1a3d | Iustin Pop | |

441 | c71a1a3d | Iustin Pop | Note that since version 2.1 Ganeti provides some commands to automate |

442 | c71a1a3d | Iustin Pop | these two operations, see :ref:`storage-units-label`. |

443 | c71a1a3d | Iustin Pop | |

444 | c71a1a3d | Iustin Pop | .. _rest-redundancy-label: |

445 | c71a1a3d | Iustin Pop | |

446 | c71a1a3d | Iustin Pop | Restoring redundancy for DRBD-based instances |

447 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

448 | c71a1a3d | Iustin Pop | |

449 | c71a1a3d | Iustin Pop | A DRBD instance has two nodes, and the storage on one of them has |

450 | c71a1a3d | Iustin Pop | failed. Depending on which node (primary or secondary) has failed, you |

451 | c71a1a3d | Iustin Pop | have three options at hand: |

452 | c71a1a3d | Iustin Pop | |

453 | c71a1a3d | Iustin Pop | - if the storage on the primary node has failed, you need to re-create |

454 | c71a1a3d | Iustin Pop | the disks on it |

455 | c71a1a3d | Iustin Pop | - if the storage on the secondary node has failed, you can either |

456 | c71a1a3d | Iustin Pop | re-create the disks on it or change the secondary and recreate |

457 | c71a1a3d | Iustin Pop | redundancy on the new secondary node |

458 | c71a1a3d | Iustin Pop | |

459 | c71a1a3d | Iustin Pop | Of course, at any point it's possible to force re-creation of disks even |

460 | c71a1a3d | Iustin Pop | though everything is already fine. |

461 | c71a1a3d | Iustin Pop | |

462 | c71a1a3d | Iustin Pop | For all three cases, the ``replace-disks`` operation can be used:: |

463 | c71a1a3d | Iustin Pop | |

464 | c71a1a3d | Iustin Pop | # re-create disks on the primary node |

465 | c71a1a3d | Iustin Pop | gnt-instance replace-disks -p INSTANCE_NAME |

466 | c71a1a3d | Iustin Pop | # re-create disks on the current secondary |

467 | c71a1a3d | Iustin Pop | gnt-instance replace-disks -s INSTANCE_NAME |

468 | c71a1a3d | Iustin Pop | # change the secondary node, via manual specification |

469 | c71a1a3d | Iustin Pop | gnt-instance replace-disks -n NODE INSTANCE_NAME |

470 | c71a1a3d | Iustin Pop | # change the secondary node, via an iallocator script |

471 | c71a1a3d | Iustin Pop | gnt-instance replace-disks -I SCRIPT INSTANCE_NAME |

472 | c71a1a3d | Iustin Pop | # since Ganeti 2.1: automatically fix the primary or secondary node |

473 | c71a1a3d | Iustin Pop | gnt-instance replace-disks -a INSTANCE_NAME |

474 | c71a1a3d | Iustin Pop | |

475 | c71a1a3d | Iustin Pop | Since the process involves copying all data from the working node to the |

476 | c71a1a3d | Iustin Pop | target node, it will take a while, depending on the instance's disk |

477 | c71a1a3d | Iustin Pop | size, node I/O system and network speed. But it is (baring any network |

478 | c71a1a3d | Iustin Pop | interruption) completely transparent for the instance. |

479 | c71a1a3d | Iustin Pop | |

480 | c71a1a3d | Iustin Pop | Re-creating disks for non-redundant instances |

481 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

482 | c71a1a3d | Iustin Pop | |

483 | c71a1a3d | Iustin Pop | .. versionadded:: 2.1 |

484 | c71a1a3d | Iustin Pop | |

485 | c71a1a3d | Iustin Pop | For non-redundant instances, there isn't a copy (except backups) to |

486 | c71a1a3d | Iustin Pop | re-create the disks. But it's possible to at-least re-create empty |

487 | c71a1a3d | Iustin Pop | disks, after which a reinstall can be run, via the ``recreate-disks`` |

488 | c71a1a3d | Iustin Pop | command:: |

489 | c71a1a3d | Iustin Pop | |

490 | c71a1a3d | Iustin Pop | gnt-instance recreate-disks INSTANCE |

491 | c71a1a3d | Iustin Pop | |

492 | c71a1a3d | Iustin Pop | Note that this will fail if the disks already exists. |

493 | c71a1a3d | Iustin Pop | |

494 | c71a1a3d | Iustin Pop | Debugging instances |

495 | c71a1a3d | Iustin Pop | +++++++++++++++++++ |

496 | ffa6869f | Iustin Pop | |

497 | ffa6869f | Iustin Pop | Accessing an instance's disks |

498 | ffa6869f | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

499 | ffa6869f | Iustin Pop | |

500 | c71a1a3d | Iustin Pop | From an instance's primary node you can have access to its disks. Never |

501 | ffa6869f | Iustin Pop | ever mount the underlying logical volume manually on a fault tolerant |

502 | c71a1a3d | Iustin Pop | instance, or will break replication and your data will be |

503 | c71a1a3d | Iustin Pop | inconsistent. The correct way to access an instance's disks is to run |

504 | c71a1a3d | Iustin Pop | (on the master node, as usual) the command:: |

505 | c71a1a3d | Iustin Pop | |

506 | c71a1a3d | Iustin Pop | gnt-instance activate-disks INSTANCE |

507 | c71a1a3d | Iustin Pop | |

508 | c71a1a3d | Iustin Pop | And then, *on the primary node of the instance*, access the device that |

509 | c71a1a3d | Iustin Pop | gets created. For example, you could mount the given disks, then edit |

510 | c71a1a3d | Iustin Pop | files on the filesystem, etc. |

511 | c71a1a3d | Iustin Pop | |

512 | c71a1a3d | Iustin Pop | Note that with partitioned disks (as opposed to whole-disk filesystems), |

513 | c71a1a3d | Iustin Pop | you will need to use a tool like :manpage:`kpartx(8)`:: |

514 | ffa6869f | Iustin Pop | |

515 | c71a1a3d | Iustin Pop | node1# gnt-instance activate-disks instance1 |

516 | c71a1a3d | Iustin Pop | … |

517 | c71a1a3d | Iustin Pop | node1# ssh node3 |

518 | c71a1a3d | Iustin Pop | node3# kpartx -l /dev/… |

519 | c71a1a3d | Iustin Pop | node3# kpartx -a /dev/… |

520 | c71a1a3d | Iustin Pop | node3# mount /dev/mapper/… /mnt/ |

521 | c71a1a3d | Iustin Pop | # edit files under mnt as desired |

522 | c71a1a3d | Iustin Pop | node3# umount /mnt/ |

523 | c71a1a3d | Iustin Pop | node3# kpartx -d /dev/… |

524 | c71a1a3d | Iustin Pop | node3# exit |

525 | c71a1a3d | Iustin Pop | node1# |

526 | ffa6869f | Iustin Pop | |

527 | c71a1a3d | Iustin Pop | After you've finished you can deactivate them with the deactivate-disks |

528 | c71a1a3d | Iustin Pop | command, which works in the same way:: |

529 | c71a1a3d | Iustin Pop | |

530 | c71a1a3d | Iustin Pop | gnt-instance deactivate-disks INSTANCE |

531 | c71a1a3d | Iustin Pop | |

532 | c71a1a3d | Iustin Pop | Note that if any process started by you is still using the disks, the |

533 | c71a1a3d | Iustin Pop | above command will error out, and you **must** cleanup and ensure that |

534 | c71a1a3d | Iustin Pop | the above command runs successfully before you start the instance, |

535 | c71a1a3d | Iustin Pop | otherwise the instance will suffer corruption. |

536 | ffa6869f | Iustin Pop | |

537 | ffa6869f | Iustin Pop | Accessing an instance's console |

538 | fd07c6b3 | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

539 | ffa6869f | Iustin Pop | |

540 | ffa6869f | Iustin Pop | The command to access a running instance's console is:: |

541 | ffa6869f | Iustin Pop | |

542 | ffa6869f | Iustin Pop | gnt-instance console INSTANCE_NAME |

543 | ffa6869f | Iustin Pop | |

544 | c71a1a3d | Iustin Pop | Use the console normally and then type ``^]`` when done, to exit. |

545 | c71a1a3d | Iustin Pop | |

546 | c71a1a3d | Iustin Pop | Other instance operations |

547 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++ |

548 | c71a1a3d | Iustin Pop | |

549 | c71a1a3d | Iustin Pop | Reboot |

550 | c71a1a3d | Iustin Pop | ~~~~~~ |

551 | ffa6869f | Iustin Pop | |

552 | c71a1a3d | Iustin Pop | There is a wrapper command for rebooting instances:: |

553 | ffa6869f | Iustin Pop | |

554 | c71a1a3d | Iustin Pop | gnt-instance reboot instance2 |

555 | c71a1a3d | Iustin Pop | |

556 | c71a1a3d | Iustin Pop | By default, this does the equivalent of shutting down and then starting |

557 | c71a1a3d | Iustin Pop | the instance, but it accepts parameters to perform a soft-reboot (via |

558 | c71a1a3d | Iustin Pop | the hypervisor), a hard reboot (hypervisor shutdown and then startup) or |

559 | c71a1a3d | Iustin Pop | a full one (the default, which also de-configures and then configures |

560 | c71a1a3d | Iustin Pop | again the disks of the instance). |

561 | c71a1a3d | Iustin Pop | |

562 | c71a1a3d | Iustin Pop | Instance OS definitions debugging |

563 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++++++++++ |

564 | c71a1a3d | Iustin Pop | |

565 | c71a1a3d | Iustin Pop | Should you have any problems with instance operating systems the command |

566 | c71a1a3d | Iustin Pop | to see a complete status for all your nodes is:: |

567 | ffa6869f | Iustin Pop | |

568 | ffa6869f | Iustin Pop | gnt-os diagnose |

569 | ffa6869f | Iustin Pop | |

570 | c71a1a3d | Iustin Pop | .. _instance-relocation-label: |

571 | ffa6869f | Iustin Pop | |

572 | c71a1a3d | Iustin Pop | Instance relocation |

573 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~ |

574 | ffa6869f | Iustin Pop | |

575 | c71a1a3d | Iustin Pop | While it is not possible to move an instance from nodes ``(A, B)`` to |

576 | c71a1a3d | Iustin Pop | nodes ``(C, D)`` in a single move, it is possible to do so in a few |

577 | c71a1a3d | Iustin Pop | steps:: |

578 | ffa6869f | Iustin Pop | |

579 | c71a1a3d | Iustin Pop | # instance is located on A, B |

580 | c71a1a3d | Iustin Pop | node1# gnt-instance replace -n nodeC instance1 |

581 | c71a1a3d | Iustin Pop | # instance has moved from (A, B) to (A, C) |

582 | c71a1a3d | Iustin Pop | # we now flip the primary/secondary nodes |

583 | c71a1a3d | Iustin Pop | node1# gnt-instance migrate instance1 |

584 | c71a1a3d | Iustin Pop | # instance lives on (C, A) |

585 | c71a1a3d | Iustin Pop | # we can then change A to D via: |

586 | c71a1a3d | Iustin Pop | node1# gnt-instance replace -n nodeD instance1 |

587 | 56c9a709 | Iustin Pop | |

588 | c71a1a3d | Iustin Pop | Which brings it into the final configuration of ``(C, D)``. Note that we |

589 | c71a1a3d | Iustin Pop | needed to do two replace-disks operation (two copies of the instance |

590 | c71a1a3d | Iustin Pop | disks), because we needed to get rid of both the original nodes (A and |

591 | c71a1a3d | Iustin Pop | B). |

592 | c71a1a3d | Iustin Pop | |

593 | c71a1a3d | Iustin Pop | Node operations |

594 | c71a1a3d | Iustin Pop | --------------- |

595 | c71a1a3d | Iustin Pop | |

596 | c71a1a3d | Iustin Pop | There are much fewer node operations available than for instances, but |

597 | c71a1a3d | Iustin Pop | they are equivalently important for maintaining a healthy cluster. |

598 | c71a1a3d | Iustin Pop | |

599 | c71a1a3d | Iustin Pop | Add/readd |

600 | c71a1a3d | Iustin Pop | +++++++++ |

601 | c71a1a3d | Iustin Pop | |

602 | c71a1a3d | Iustin Pop | It is at any time possible to extend the cluster with one more node, by |

603 | c71a1a3d | Iustin Pop | using the node add operation:: |

604 | c71a1a3d | Iustin Pop | |

605 | c71a1a3d | Iustin Pop | gnt-node add NEW_NODE |

606 | c71a1a3d | Iustin Pop | |

607 | c71a1a3d | Iustin Pop | If the cluster has a replication network defined, then you need to pass |

608 | c71a1a3d | Iustin Pop | the ``-s REPLICATION_IP`` parameter to this option. |

609 | c71a1a3d | Iustin Pop | |

610 | c71a1a3d | Iustin Pop | A variation of this command can be used to re-configure a node if its |

611 | c71a1a3d | Iustin Pop | Ganeti configuration is broken, for example if it has been reinstalled |

612 | c71a1a3d | Iustin Pop | by mistake:: |

613 | c71a1a3d | Iustin Pop | |

614 | c71a1a3d | Iustin Pop | gnt-node add --readd EXISTING_NODE |

615 | c71a1a3d | Iustin Pop | |

616 | c71a1a3d | Iustin Pop | This will reinitialise the node as if it's been newly added, but while |

617 | c71a1a3d | Iustin Pop | keeping its existing configuration in the cluster (primary/secondary IP, |

618 | c71a1a3d | Iustin Pop | etc.), in other words you won't need to use ``-s`` here. |

619 | c71a1a3d | Iustin Pop | |

620 | c71a1a3d | Iustin Pop | Changing the node role |

621 | c71a1a3d | Iustin Pop | ++++++++++++++++++++++ |

622 | c71a1a3d | Iustin Pop | |

623 | c71a1a3d | Iustin Pop | A node can be in different roles, as explained in the |

624 | c71a1a3d | Iustin Pop | :ref:`terminology-label` section. Promoting a node to the master role is |

625 | c71a1a3d | Iustin Pop | special, while the other roles are handled all via a single command. |

626 | c71a1a3d | Iustin Pop | |

627 | c71a1a3d | Iustin Pop | Failing over the master node |

628 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

629 | c71a1a3d | Iustin Pop | |

630 | c71a1a3d | Iustin Pop | If you want to promote a different node to the master role (for whatever |

631 | c71a1a3d | Iustin Pop | reason), run on any other master-candidate node the command:: |

632 | c71a1a3d | Iustin Pop | |

633 | c71a1a3d | Iustin Pop | gnt-cluster masterfailover |

634 | c71a1a3d | Iustin Pop | |

635 | c71a1a3d | Iustin Pop | and the node you ran it on is now the new master. In case you try to run |

636 | c71a1a3d | Iustin Pop | this on a non master-candidate node, you will get an error telling you |

637 | c71a1a3d | Iustin Pop | which nodes are valid. |

638 | c71a1a3d | Iustin Pop | |

639 | c71a1a3d | Iustin Pop | Changing between the other roles |

640 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

641 | c71a1a3d | Iustin Pop | |

642 | c71a1a3d | Iustin Pop | The ``gnt-node modify`` command can be used to select a new role:: |

643 | c71a1a3d | Iustin Pop | |

644 | c71a1a3d | Iustin Pop | # change to master candidate |

645 | c71a1a3d | Iustin Pop | gnt-node modify -C yes NODE |

646 | c71a1a3d | Iustin Pop | # change to drained status |

647 | c71a1a3d | Iustin Pop | gnt-node modify -D yes NODE |

648 | c71a1a3d | Iustin Pop | # change to offline status |

649 | c71a1a3d | Iustin Pop | gnt-node modify -O yes NODE |

650 | c71a1a3d | Iustin Pop | # change to regular mode (reset all flags) |

651 | c71a1a3d | Iustin Pop | gnt-node modify -O no -D no -C no NODE |

652 | c71a1a3d | Iustin Pop | |

653 | c71a1a3d | Iustin Pop | Note that the cluster requires that at any point in time, a certain |

654 | c71a1a3d | Iustin Pop | number of nodes are master candidates, so changing from master candidate |

655 | c71a1a3d | Iustin Pop | to other roles might fail. It is recommended to either force the |

656 | c71a1a3d | Iustin Pop | operation (via the ``--force`` option) or first change the number of |

657 | c71a1a3d | Iustin Pop | master candidates in the cluster - see :ref:`cluster-config-label`. |

658 | c71a1a3d | Iustin Pop | |

659 | c71a1a3d | Iustin Pop | Evacuating nodes |

660 | c71a1a3d | Iustin Pop | ++++++++++++++++ |

661 | c71a1a3d | Iustin Pop | |

662 | c71a1a3d | Iustin Pop | There are two steps of moving instances off a node: |

663 | c71a1a3d | Iustin Pop | |

664 | c71a1a3d | Iustin Pop | - moving the primary instances (actually converting them into secondary |

665 | c71a1a3d | Iustin Pop | instances) |

666 | c71a1a3d | Iustin Pop | - moving the secondary instances (including any instances converted in |

667 | c71a1a3d | Iustin Pop | the step above) |

668 | c71a1a3d | Iustin Pop | |

669 | c71a1a3d | Iustin Pop | Primary instance conversion |

670 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

671 | c71a1a3d | Iustin Pop | |

672 | c71a1a3d | Iustin Pop | For this step, you can use either individual instance move |

673 | c71a1a3d | Iustin Pop | commands (as seen in :ref:`instance-change-primary-label`) or the bulk |

674 | c71a1a3d | Iustin Pop | per-node versions; these are:: |

675 | c71a1a3d | Iustin Pop | |

676 | c71a1a3d | Iustin Pop | gnt-node migrate NODE |

677 | c71a1a3d | Iustin Pop | gnt-node evacuate NODE |

678 | c71a1a3d | Iustin Pop | |

679 | c71a1a3d | Iustin Pop | Note that the instance “move” command doesn't currently have a node |

680 | c71a1a3d | Iustin Pop | equivalent. |

681 | c71a1a3d | Iustin Pop | |

682 | c71a1a3d | Iustin Pop | Both these commands, or the equivalent per-instance command, will make |

683 | c71a1a3d | Iustin Pop | this node the secondary node for the respective instances, whereas their |

684 | c71a1a3d | Iustin Pop | current secondary node will become primary. Note that it is not possible |

685 | c71a1a3d | Iustin Pop | to change in one step the primary node to another node as primary, while |

686 | c71a1a3d | Iustin Pop | keeping the same secondary node. |

687 | c71a1a3d | Iustin Pop | |

688 | c71a1a3d | Iustin Pop | Secondary instance evacuation |

689 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

690 | c71a1a3d | Iustin Pop | |

691 | c71a1a3d | Iustin Pop | For the evacuation of secondary instances, a command called |

692 | c71a1a3d | Iustin Pop | :command:`gnt-node evacuate` is provided and its syntax is:: |

693 | c71a1a3d | Iustin Pop | |

694 | c71a1a3d | Iustin Pop | gnt-node evacuate -I IALLOCATOR_SCRIPT NODE |

695 | c71a1a3d | Iustin Pop | gnt-node evacuate -n DESTINATION_NODE NODE |

696 | c71a1a3d | Iustin Pop | |

697 | c71a1a3d | Iustin Pop | The first version will compute the new secondary for each instance in |

698 | c71a1a3d | Iustin Pop | turn using the given iallocator script, whereas the second one will |

699 | c71a1a3d | Iustin Pop | simply move all instances to DESTINATION_NODE. |

700 | c71a1a3d | Iustin Pop | |

701 | c71a1a3d | Iustin Pop | Removal |

702 | c71a1a3d | Iustin Pop | +++++++ |

703 | c71a1a3d | Iustin Pop | |

704 | c71a1a3d | Iustin Pop | Once a node no longer has any instances (neither primary nor secondary), |

705 | c71a1a3d | Iustin Pop | it's easy to remove it from the cluster:: |

706 | c71a1a3d | Iustin Pop | |

707 | c71a1a3d | Iustin Pop | gnt-node remove NODE_NAME |

708 | c71a1a3d | Iustin Pop | |

709 | c71a1a3d | Iustin Pop | This will deconfigure the node, stop the ganeti daemons on it and leave |

710 | c71a1a3d | Iustin Pop | it hopefully like before it joined to the cluster. |

711 | c71a1a3d | Iustin Pop | |

712 | c71a1a3d | Iustin Pop | Storage handling |

713 | c71a1a3d | Iustin Pop | ++++++++++++++++ |

714 | c71a1a3d | Iustin Pop | |

715 | c71a1a3d | Iustin Pop | When using LVM (either standalone or with DRBD), it can become tedious |

716 | c71a1a3d | Iustin Pop | to debug and fix it in case of errors. Furthermore, even file-based |

717 | c71a1a3d | Iustin Pop | storage can become complicated to handle manually on many hosts. Ganeti |

718 | c71a1a3d | Iustin Pop | provides a couple of commands to help with automation. |

719 | c71a1a3d | Iustin Pop | |

720 | c71a1a3d | Iustin Pop | Logical volumes |

721 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~ |

722 | c71a1a3d | Iustin Pop | |

723 | c71a1a3d | Iustin Pop | This is a command specific to LVM handling. It allows listing the |

724 | c71a1a3d | Iustin Pop | logical volumes on a given node or on all nodes and their association to |

725 | c71a1a3d | Iustin Pop | instances via the ``volumes`` command:: |

726 | c71a1a3d | Iustin Pop | |

727 | c71a1a3d | Iustin Pop | node1# gnt-node volumes |

728 | c71a1a3d | Iustin Pop | Node PhysDev VG Name Size Instance |

729 | c71a1a3d | Iustin Pop | node1 /dev/sdb1 xenvg e61fbc97-….disk0 512M instance17 |

730 | c71a1a3d | Iustin Pop | node1 /dev/sdb1 xenvg ebd1a7d1-….disk0 512M instance19 |

731 | c71a1a3d | Iustin Pop | node2 /dev/sdb1 xenvg 0af08a3d-….disk0 512M instance20 |

732 | c71a1a3d | Iustin Pop | node2 /dev/sdb1 xenvg cc012285-….disk0 512M instance16 |

733 | c71a1a3d | Iustin Pop | node2 /dev/sdb1 xenvg f0fac192-….disk0 512M instance18 |

734 | c71a1a3d | Iustin Pop | |

735 | c71a1a3d | Iustin Pop | The above command maps each logical volume to a volume group and |

736 | c71a1a3d | Iustin Pop | underlying physical volume and (possibly) to an instance. |

737 | c71a1a3d | Iustin Pop | |

738 | c71a1a3d | Iustin Pop | .. _storage-units-label: |

739 | c71a1a3d | Iustin Pop | |

740 | c71a1a3d | Iustin Pop | Generalized storage handling |

741 | c71a1a3d | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

742 | c71a1a3d | Iustin Pop | |

743 | c71a1a3d | Iustin Pop | .. versionadded:: 2.1 |

744 | c71a1a3d | Iustin Pop | |

745 | c71a1a3d | Iustin Pop | Starting with Ganeti 2.1, a new storage framework has been implemented |

746 | c71a1a3d | Iustin Pop | that tries to abstract the handling of the storage type the cluster |

747 | c71a1a3d | Iustin Pop | uses. |

748 | c71a1a3d | Iustin Pop | |

749 | c71a1a3d | Iustin Pop | First is listing the backend storage and their space situation:: |

750 | c71a1a3d | Iustin Pop | |

751 | c71a1a3d | Iustin Pop | node1# gnt-node list-storage |

752 | c71a1a3d | Iustin Pop | Node Name Size Used Free |

753 | c71a1a3d | Iustin Pop | node1 /dev/sda7 673.8G 0M 673.8G |

754 | c71a1a3d | Iustin Pop | node1 /dev/sdb1 698.6G 1.5G 697.1G |

755 | c71a1a3d | Iustin Pop | node2 /dev/sda7 673.8G 0M 673.8G |

756 | c71a1a3d | Iustin Pop | node2 /dev/sdb1 698.6G 1.0G 697.6G |

757 | c71a1a3d | Iustin Pop | |

758 | c71a1a3d | Iustin Pop | The default is to list LVM physical volumes. It's also possible to list |

759 | c71a1a3d | Iustin Pop | the LVM volume groups:: |

760 | c71a1a3d | Iustin Pop | |

761 | c71a1a3d | Iustin Pop | node1# gnt-node list-storage -t lvm-vg |

762 | c71a1a3d | Iustin Pop | Node Name Size |

763 | c71a1a3d | Iustin Pop | node1 xenvg 1.3T |

764 | c71a1a3d | Iustin Pop | node2 xenvg 1.3T |

765 | c71a1a3d | Iustin Pop | |

766 | c71a1a3d | Iustin Pop | Next is repairing storage units, which is currently only implemented for |

767 | c71a1a3d | Iustin Pop | volume groups and does the equivalent of ``vgreduce --removemissing``:: |

768 | c71a1a3d | Iustin Pop | |

769 | c71a1a3d | Iustin Pop | node1# gnt-node repair-storage node2 lvm-vg xenvg |

770 | c71a1a3d | Iustin Pop | Sun Oct 25 22:21:45 2009 Repairing storage unit 'xenvg' on node2 ... |

771 | c71a1a3d | Iustin Pop | |

772 | c71a1a3d | Iustin Pop | Last is the modification of volume properties, which is (again) only |

773 | c71a1a3d | Iustin Pop | implemented for LVM physical volumes and allows toggling the |

774 | c71a1a3d | Iustin Pop | ``allocatable`` value:: |

775 | c71a1a3d | Iustin Pop | |

776 | c71a1a3d | Iustin Pop | node1# gnt-node modify-storage --allocatable=no node2 lvm-pv /dev/sdb1 |

777 | c71a1a3d | Iustin Pop | |

778 | c71a1a3d | Iustin Pop | Use of the storage commands |

779 | 56c9a709 | Iustin Pop | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |

780 | 56c9a709 | Iustin Pop | |

781 | c71a1a3d | Iustin Pop | All these commands are needed when recovering a node from a disk |

782 | c71a1a3d | Iustin Pop | failure: |

783 | c71a1a3d | Iustin Pop | |

784 | c71a1a3d | Iustin Pop | - first, we need to recover from complete LVM failure (due to missing |

785 | c71a1a3d | Iustin Pop | disk), by running the ``repair-storage`` command |

786 | c71a1a3d | Iustin Pop | - second, we need to change allocation on any partially-broken disk |

787 | c71a1a3d | Iustin Pop | (i.e. LVM still sees it, but it has bad blocks) by running |

788 | c71a1a3d | Iustin Pop | ``modify-storage`` |

789 | c71a1a3d | Iustin Pop | - then we can evacuate the instances as needed |

790 | 56c9a709 | Iustin Pop | |

791 | 56c9a709 | Iustin Pop | |

792 | c71a1a3d | Iustin Pop | Cluster operations |

793 | c71a1a3d | Iustin Pop | ------------------ |

794 | c71a1a3d | Iustin Pop | |

795 | c71a1a3d | Iustin Pop | Beside the cluster initialisation command (which is detailed in the |

796 | c71a1a3d | Iustin Pop | :doc:`install` document) and the master failover command which is |

797 | c71a1a3d | Iustin Pop | explained under node handling, there are a couple of other cluster |

798 | c71a1a3d | Iustin Pop | operations available. |

799 | c71a1a3d | Iustin Pop | |

800 | c71a1a3d | Iustin Pop | .. _cluster-config-label: |

801 | c71a1a3d | Iustin Pop | |

802 | c71a1a3d | Iustin Pop | Standard operations |

803 | c71a1a3d | Iustin Pop | +++++++++++++++++++ |

804 | c71a1a3d | Iustin Pop | |

805 | c71a1a3d | Iustin Pop | One of the few commands that can be run on any node (not only the |

806 | c71a1a3d | Iustin Pop | master) is the ``getmaster`` command:: |

807 | c71a1a3d | Iustin Pop | |

808 | c71a1a3d | Iustin Pop | node2# gnt-cluster getmaster |

809 | c71a1a3d | Iustin Pop | node1.example.com |

810 | c71a1a3d | Iustin Pop | node2# |

811 | c71a1a3d | Iustin Pop | |

812 | c71a1a3d | Iustin Pop | It is possible to query and change global cluster parameters via the |

813 | c71a1a3d | Iustin Pop | ``info`` and ``modify`` commands:: |

814 | c71a1a3d | Iustin Pop | |

815 | c71a1a3d | Iustin Pop | node1# gnt-cluster info |

816 | c71a1a3d | Iustin Pop | Cluster name: cluster.example.com |

817 | c71a1a3d | Iustin Pop | Cluster UUID: 07805e6f-f0af-4310-95f1-572862ee939c |

818 | c71a1a3d | Iustin Pop | Creation time: 2009-09-25 05:04:15 |

819 | c71a1a3d | Iustin Pop | Modification time: 2009-10-18 22:11:47 |

820 | c71a1a3d | Iustin Pop | Master node: node1.example.com |

821 | c71a1a3d | Iustin Pop | Architecture (this node): 64bit (x86_64) |

822 | c71a1a3d | Iustin Pop | … |

823 | c71a1a3d | Iustin Pop | Tags: foo |

824 | c71a1a3d | Iustin Pop | Default hypervisor: xen-pvm |

825 | c71a1a3d | Iustin Pop | Enabled hypervisors: xen-pvm |

826 | c71a1a3d | Iustin Pop | Hypervisor parameters: |

827 | c71a1a3d | Iustin Pop | - xen-pvm: |

828 | c71a1a3d | Iustin Pop | root_path: /dev/sda1 |

829 | c71a1a3d | Iustin Pop | … |

830 | c71a1a3d | Iustin Pop | Cluster parameters: |

831 | c71a1a3d | Iustin Pop | - candidate pool size: 10 |

832 | c71a1a3d | Iustin Pop | … |

833 | c71a1a3d | Iustin Pop | Default instance parameters: |

834 | c71a1a3d | Iustin Pop | - default: |

835 | c71a1a3d | Iustin Pop | memory: 128 |

836 | c71a1a3d | Iustin Pop | … |

837 | c71a1a3d | Iustin Pop | Default nic parameters: |

838 | c71a1a3d | Iustin Pop | - default: |

839 | c71a1a3d | Iustin Pop | link: xen-br0 |

840 | c71a1a3d | Iustin Pop | … |

841 | c71a1a3d | Iustin Pop | |

842 | c71a1a3d | Iustin Pop | There various parameters above can be changed via the ``modify`` |

843 | c71a1a3d | Iustin Pop | commands as follows: |

844 | c71a1a3d | Iustin Pop | |

845 | c71a1a3d | Iustin Pop | - the hypervisor parameters can be changed via ``modify -H |

846 | c71a1a3d | Iustin Pop | xen-pvm:root_path=…``, and so on for other hypervisors/key/values |

847 | c71a1a3d | Iustin Pop | - the "default instance parameters" are changeable via ``modify -B |

848 | c71a1a3d | Iustin Pop | parameter=value…`` syntax |

849 | c71a1a3d | Iustin Pop | - the cluster parameters are changeable via separate options to the |

850 | c71a1a3d | Iustin Pop | modify command (e.g. ``--candidate-pool-size``, etc.) |

851 | c71a1a3d | Iustin Pop | |

852 | c71a1a3d | Iustin Pop | For detailed option list see the :manpage:`gnt-cluster(8)` man page. |

853 | c71a1a3d | Iustin Pop | |

854 | c71a1a3d | Iustin Pop | The cluster version can be obtained via the ``version`` command:: |

855 | c71a1a3d | Iustin Pop | node1# gnt-cluster version |

856 | c71a1a3d | Iustin Pop | Software version: 2.1.0 |

857 | c71a1a3d | Iustin Pop | Internode protocol: 20 |

858 | c71a1a3d | Iustin Pop | Configuration format: 2010000 |

859 | c71a1a3d | Iustin Pop | OS api version: 15 |

860 | c71a1a3d | Iustin Pop | Export interface: 0 |

861 | c71a1a3d | Iustin Pop | |

862 | c71a1a3d | Iustin Pop | This is not very useful except when debugging Ganeti. |

863 | c71a1a3d | Iustin Pop | |

864 | c71a1a3d | Iustin Pop | Global node commands |

865 | c71a1a3d | Iustin Pop | ++++++++++++++++++++ |

866 | c71a1a3d | Iustin Pop | |

867 | c71a1a3d | Iustin Pop | There are two commands provided for replicating files to all nodes of a |

868 | c71a1a3d | Iustin Pop | cluster and for running commands on all the nodes:: |

869 | c71a1a3d | Iustin Pop | |

870 | c71a1a3d | Iustin Pop | node1# gnt-cluster copyfile /path/to/file |

871 | c71a1a3d | Iustin Pop | node1# gnt-cluster command ls -l /path/to/file |

872 | c71a1a3d | Iustin Pop | |

873 | c71a1a3d | Iustin Pop | These are simple wrappers over scp/ssh and more advanced usage can be |

874 | c71a1a3d | Iustin Pop | obtained using :manpage:`dsh(1)` and similar commands. But they are |

875 | c71a1a3d | Iustin Pop | useful to update an OS script from the master node, for example. |

876 | c71a1a3d | Iustin Pop | |

877 | c71a1a3d | Iustin Pop | Cluster verification |

878 | c71a1a3d | Iustin Pop | ++++++++++++++++++++ |

879 | c71a1a3d | Iustin Pop | |

880 | c71a1a3d | Iustin Pop | There are three commands that relate to global cluster checks. The first |

881 | c71a1a3d | Iustin Pop | one is ``verify`` which gives an overview on the cluster state, |

882 | c71a1a3d | Iustin Pop | highlighting any issues. In normal operation, this command should return |

883 | c71a1a3d | Iustin Pop | no ``ERROR`` messages:: |

884 | c71a1a3d | Iustin Pop | |

885 | c71a1a3d | Iustin Pop | node1# gnt-cluster verify |

886 | c71a1a3d | Iustin Pop | Sun Oct 25 23:08:58 2009 * Verifying global settings |

887 | c71a1a3d | Iustin Pop | Sun Oct 25 23:08:58 2009 * Gathering data (2 nodes) |

888 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying node status |

889 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying instance status |

890 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying orphan volumes |

891 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying remaining instances |

892 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Verifying N+1 Memory redundancy |

893 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Other Notes |

894 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 - NOTICE: 5 non-redundant instance(s) found. |

895 | c71a1a3d | Iustin Pop | Sun Oct 25 23:09:00 2009 * Hooks Results |

896 | c71a1a3d | Iustin Pop | |

897 | c71a1a3d | Iustin Pop | The second command is ``verify-disks``, which checks that the instance's |

898 | c71a1a3d | Iustin Pop | disks have the correct status based on the desired instance state |

899 | c71a1a3d | Iustin Pop | (up/down):: |

900 | c71a1a3d | Iustin Pop | |

901 | c71a1a3d | Iustin Pop | node1# gnt-cluster verify-disks |

902 | c71a1a3d | Iustin Pop | |

903 | c71a1a3d | Iustin Pop | Note that this command will show no output when disks are healthy. |

904 | c71a1a3d | Iustin Pop | |

905 | c71a1a3d | Iustin Pop | The last command is used to repair any discrepancies in Ganeti's |

906 | c71a1a3d | Iustin Pop | recorded disk size and the actual disk size (disk size information is |

907 | c71a1a3d | Iustin Pop | needed for proper activation and growth of DRBD-based disks):: |

908 | c71a1a3d | Iustin Pop | |

909 | c71a1a3d | Iustin Pop | node1# gnt-cluster repair-disk-sizes |

910 | c71a1a3d | Iustin Pop | Sun Oct 25 23:13:16 2009 - INFO: Disk 0 of instance instance1 has mismatched size, correcting: recorded 512, actual 2048 |

911 | c71a1a3d | Iustin Pop | Sun Oct 25 23:13:17 2009 - WARNING: Invalid result from node node4, ignoring node results |

912 | c71a1a3d | Iustin Pop | |

913 | c71a1a3d | Iustin Pop | The above shows one instance having wrong disk size, and a node which |

914 | c71a1a3d | Iustin Pop | returned invalid data, and thus we ignored all primary instances of that |

915 | c71a1a3d | Iustin Pop | node. |

916 | c71a1a3d | Iustin Pop | |

917 | c71a1a3d | Iustin Pop | Configuration redistribution |

918 | c71a1a3d | Iustin Pop | ++++++++++++++++++++++++++++ |

919 | c71a1a3d | Iustin Pop | |

920 | c71a1a3d | Iustin Pop | If the verify command complains about file mismatches between the master |

921 | c71a1a3d | Iustin Pop | and other nodes, due to some node problems or if you manually modified |

922 | c71a1a3d | Iustin Pop | configuration files, you can force an push of the master configuration |

923 | c71a1a3d | Iustin Pop | to all other nodes via the ``redist-conf`` command:: |

924 | c71a1a3d | Iustin Pop | |

925 | c71a1a3d | Iustin Pop | node1# gnt-cluster redist-conf |

926 | c71a1a3d | Iustin Pop | node1# |

927 | c71a1a3d | Iustin Pop | |

928 | c71a1a3d | Iustin Pop | This command will be silent unless there are problems sending updates to |

929 | c71a1a3d | Iustin Pop | the other nodes. |

930 | c71a1a3d | Iustin Pop | |

931 | c71a1a3d | Iustin Pop | |

932 | c71a1a3d | Iustin Pop | Cluster renaming |

933 | c71a1a3d | Iustin Pop | ++++++++++++++++ |

934 | c71a1a3d | Iustin Pop | |

935 | c71a1a3d | Iustin Pop | It is possible to rename a cluster, or to change its IP address, via the |

936 | c71a1a3d | Iustin Pop | ``rename`` command. If only the IP has changed, you need to pass the |

937 | c71a1a3d | Iustin Pop | current name and Ganeti will realise its IP has changed:: |

938 | c71a1a3d | Iustin Pop | |

939 | c71a1a3d | Iustin Pop | node1# gnt-cluster rename cluster.example.com |

940 | c71a1a3d | Iustin Pop | This will rename the cluster to 'cluster.example.com'. If |

941 | c71a1a3d | Iustin Pop | you are connected over the network to the cluster name, the operation |

942 | c71a1a3d | Iustin Pop | is very dangerous as the IP address will be removed from the node and |

943 | c71a1a3d | Iustin Pop | the change may not go through. Continue? |

944 | c71a1a3d | Iustin Pop | y/[n]/?: y |

945 | c71a1a3d | Iustin Pop | Failure: prerequisites not met for this operation: |

946 | c71a1a3d | Iustin Pop | Neither the name nor the IP address of the cluster has changed |

947 | c71a1a3d | Iustin Pop | |

948 | c71a1a3d | Iustin Pop | In the above output, neither value has changed since the cluster |

949 | c71a1a3d | Iustin Pop | initialisation so the operation is not completed. |

950 | c71a1a3d | Iustin Pop | |

951 | c71a1a3d | Iustin Pop | Queue operations |

952 | c71a1a3d | Iustin Pop | ++++++++++++++++ |

953 | c71a1a3d | Iustin Pop | |

954 | c71a1a3d | Iustin Pop | The job queue execution in Ganeti 2.0 and higher can be inspected, |

955 | c71a1a3d | Iustin Pop | suspended and resumed via the ``queue`` command:: |

956 | c71a1a3d | Iustin Pop | |

957 | c71a1a3d | Iustin Pop | node1~# gnt-cluster queue info |

958 | c71a1a3d | Iustin Pop | The drain flag is unset |

959 | c71a1a3d | Iustin Pop | node1~# gnt-cluster queue drain |

960 | c71a1a3d | Iustin Pop | node1~# gnt-instance stop instance1 |

961 | c71a1a3d | Iustin Pop | Failed to submit job for instance1: Job queue is drained, refusing job |

962 | c71a1a3d | Iustin Pop | node1~# gnt-cluster queue info |

963 | c71a1a3d | Iustin Pop | The drain flag is set |

964 | c71a1a3d | Iustin Pop | node1~# gnt-cluster queue undrain |

965 | c71a1a3d | Iustin Pop | |

966 | c71a1a3d | Iustin Pop | This is most useful if you have an active cluster and you need to |

967 | c71a1a3d | Iustin Pop | upgrade the Ganeti software, or simply restart the software on any node: |

968 | c71a1a3d | Iustin Pop | |

969 | c71a1a3d | Iustin Pop | #. suspend the queue via ``queue drain`` |

970 | c71a1a3d | Iustin Pop | #. wait until there are no more running jobs via ``gnt-job list`` |

971 | c71a1a3d | Iustin Pop | #. restart the master or another node, or upgrade the software |

972 | c71a1a3d | Iustin Pop | #. resume the queue via ``queue undrain`` |

973 | c71a1a3d | Iustin Pop | |

974 | c71a1a3d | Iustin Pop | .. note:: this command only stores a local flag file, and if you |

975 | c71a1a3d | Iustin Pop | failover the master, it will not have effect on the new master. |

976 | c71a1a3d | Iustin Pop | |

977 | c71a1a3d | Iustin Pop | |

978 | c71a1a3d | Iustin Pop | Watcher control |

979 | c71a1a3d | Iustin Pop | +++++++++++++++ |

980 | c71a1a3d | Iustin Pop | |

981 | c71a1a3d | Iustin Pop | The :manpage:`ganeti-watcher` is a program, usually scheduled via |

982 | c71a1a3d | Iustin Pop | ``cron``, that takes care of cluster maintenance operations (restarting |

983 | c71a1a3d | Iustin Pop | downed instances, activating down DRBD disks, etc.). However, during |

984 | c71a1a3d | Iustin Pop | maintenance and troubleshooting, this can get in your way; disabling it |

985 | c71a1a3d | Iustin Pop | via commenting out the cron job is not so good as this can be |

986 | c71a1a3d | Iustin Pop | forgotten. Thus there are some commands for automated control of the |

987 | c71a1a3d | Iustin Pop | watcher: ``pause``, ``info`` and ``continue``:: |

988 | c71a1a3d | Iustin Pop | |

989 | c71a1a3d | Iustin Pop | node1~# gnt-cluster watcher info |

990 | c71a1a3d | Iustin Pop | The watcher is not paused. |

991 | c71a1a3d | Iustin Pop | node1~# gnt-cluster watcher pause 1h |

992 | c71a1a3d | Iustin Pop | The watcher is paused until Mon Oct 26 00:30:37 2009. |

993 | c71a1a3d | Iustin Pop | node1~# gnt-cluster watcher info |

994 | c71a1a3d | Iustin Pop | The watcher is paused until Mon Oct 26 00:30:37 2009. |

995 | c71a1a3d | Iustin Pop | node1~# ganeti-watcher -d |

996 | c71a1a3d | Iustin Pop | 2009-10-25 23:30:47,984: pid=28867 ganeti-watcher:486 DEBUG Pause has been set, exiting |

997 | c71a1a3d | Iustin Pop | node1~# gnt-cluster watcher continue |

998 | c71a1a3d | Iustin Pop | The watcher is no longer paused. |

999 | c71a1a3d | Iustin Pop | node1~# ganeti-watcher -d |

1000 | c71a1a3d | Iustin Pop | 2009-10-25 23:31:04,789: pid=28976 ganeti-watcher:345 DEBUG Archived 0 jobs, left 0 |

1001 | c71a1a3d | Iustin Pop | 2009-10-25 23:31:05,884: pid=28976 ganeti-watcher:280 DEBUG Got data from cluster, writing instance status file |

1002 | c71a1a3d | Iustin Pop | 2009-10-25 23:31:06,061: pid=28976 ganeti-watcher:150 DEBUG Data didn't change, just touching status file |

1003 | c71a1a3d | Iustin Pop | node1~# gnt-cluster watcher info |

1004 | c71a1a3d | Iustin Pop | The watcher is not paused. |

1005 | c71a1a3d | Iustin Pop | node1~# |

1006 | c71a1a3d | Iustin Pop | |

1007 | c71a1a3d | Iustin Pop | The exact details of the argument to the ``pause`` command are available |

1008 | c71a1a3d | Iustin Pop | in the manpage. |

1009 | c71a1a3d | Iustin Pop | |

1010 | c71a1a3d | Iustin Pop | .. note:: this command only stores a local flag file, and if you |

1011 | c71a1a3d | Iustin Pop | failover the master, it will not have effect on the new master. |

1012 | c71a1a3d | Iustin Pop | |

1013 | c71a1a3d | Iustin Pop | Removing a cluster entirely |

1014 | c71a1a3d | Iustin Pop | +++++++++++++++++++++++++++ |

1015 | c71a1a3d | Iustin Pop | |

1016 | c71a1a3d | Iustin Pop | The usual method to cleanup a cluster is to run ``gnt-cluster destroy`` |

1017 | c71a1a3d | Iustin Pop | however if the Ganeti installation is broken in any way then this will |

1018 | c71a1a3d | Iustin Pop | not run. |

1019 | c71a1a3d | Iustin Pop | |

1020 | c71a1a3d | Iustin Pop | It is possible in such a case to cleanup manually most if not all traces |

1021 | c71a1a3d | Iustin Pop | of a cluster installation by following these steps on all of the nodes: |

1022 | c71a1a3d | Iustin Pop | |

1023 | c71a1a3d | Iustin Pop | 1. Shutdown all instances. This depends on the virtualisation method |

1024 | c71a1a3d | Iustin Pop | used (Xen, KVM, etc.): |

1025 | 56c9a709 | Iustin Pop | |

1026 | 56c9a709 | Iustin Pop | - Xen: run ``xm list`` and ``xm destroy`` on all the non-Domain-0 |

1027 | 56c9a709 | Iustin Pop | instances |

1028 | 56c9a709 | Iustin Pop | - KVM: kill all the KVM processes |

1029 | 56c9a709 | Iustin Pop | - chroot: kill all processes under the chroot mountpoints |

1030 | 56c9a709 | Iustin Pop | |

1031 | c71a1a3d | Iustin Pop | 2. If using DRBD, shutdown all DRBD minors (which should by at this time |

1032 | c71a1a3d | Iustin Pop | no-longer in use by instances); on each node, run ``drbdsetup |

1033 | 56c9a709 | Iustin Pop | /dev/drbdN down`` for each active DRBD minor. |

1034 | 56c9a709 | Iustin Pop | |

1035 | c71a1a3d | Iustin Pop | 3. If using LVM, cleanup the Ganeti volume group; if only Ganeti created |

1036 | c71a1a3d | Iustin Pop | logical volumes (and you are not sharing the volume group with the |

1037 | c71a1a3d | Iustin Pop | OS, for example), then simply running ``lvremove -f xenvg`` (replace |

1038 | c71a1a3d | Iustin Pop | 'xenvg' with your volume group name) should do the required cleanup. |

1039 | 56c9a709 | Iustin Pop | |

1040 | 56c9a709 | Iustin Pop | 4. If using file-based storage, remove recursively all files and |

1041 | 56c9a709 | Iustin Pop | directories under your file-storage directory: ``rm -rf |

1042 | c71a1a3d | Iustin Pop | /srv/ganeti/file-storage/*`` replacing the path with the correct path |

1043 | c71a1a3d | Iustin Pop | for your cluster. |

1044 | 56c9a709 | Iustin Pop | |

1045 | 56c9a709 | Iustin Pop | 5. Stop the ganeti daemons (``/etc/init.d/ganeti stop``) and kill any |

1046 | 56c9a709 | Iustin Pop | that remain alive (``pgrep ganeti`` and ``pkill ganeti``). |

1047 | 56c9a709 | Iustin Pop | |

1048 | 56c9a709 | Iustin Pop | 6. Remove the ganeti state directory (``rm -rf /var/lib/ganeti/*``), |

1049 | 56c9a709 | Iustin Pop | replacing the path with the correct path for your installation. |

1050 | 56c9a709 | Iustin Pop | |

1051 | 56c9a709 | Iustin Pop | On the master node, remove the cluster from the master-netdev (usually |

1052 | c71a1a3d | Iustin Pop | ``xen-br0`` for bridged mode, otherwise ``eth0`` or similar), by running |

1053 | c71a1a3d | Iustin Pop | ``ip a del $clusterip/32 dev xen-br0`` (use the correct cluster ip and |

1054 | c71a1a3d | Iustin Pop | network device name). |

1055 | 56c9a709 | Iustin Pop | |

1056 | 56c9a709 | Iustin Pop | At this point, the machines are ready for a cluster creation; in case |

1057 | c71a1a3d | Iustin Pop | you want to remove Ganeti completely, you need to also undo some of the |

1058 | c71a1a3d | Iustin Pop | SSH changes and log directories: |

1059 | 56c9a709 | Iustin Pop | |

1060 | 7faf5110 | Michael Hanselmann | - ``rm -rf /var/log/ganeti /srv/ganeti`` (replace with the correct |

1061 | 7faf5110 | Michael Hanselmann | paths) |

1062 | c71a1a3d | Iustin Pop | - remove from ``/root/.ssh`` the keys that Ganeti added (check the |

1063 | c71a1a3d | Iustin Pop | ``authorized_keys`` and ``id_dsa`` files) |

1064 | 56c9a709 | Iustin Pop | - regenerate the host's SSH keys (check the OpenSSH startup scripts) |

1065 | 56c9a709 | Iustin Pop | - uninstall Ganeti |

1066 | 56c9a709 | Iustin Pop | |

1067 | 56c9a709 | Iustin Pop | Otherwise, if you plan to re-create the cluster, you can just go ahead |

1068 | 56c9a709 | Iustin Pop | and rerun ``gnt-cluster init``. |

1069 | 558fd122 | Michael Hanselmann | |

1070 | c71a1a3d | Iustin Pop | Tags handling |

1071 | c71a1a3d | Iustin Pop | ------------- |

1072 | c71a1a3d | Iustin Pop | |

1073 | c71a1a3d | Iustin Pop | The tags handling (addition, removal, listing) is similar for all the |

1074 | c71a1a3d | Iustin Pop | objects that support it (instances, nodes, and the cluster). |

1075 | c71a1a3d | Iustin Pop | |

1076 | c71a1a3d | Iustin Pop | Limitations |

1077 | c71a1a3d | Iustin Pop | +++++++++++ |

1078 | c71a1a3d | Iustin Pop | |

1079 | c71a1a3d | Iustin Pop | Note that the set of characters present in a tag and the maximum tag |

1080 | c71a1a3d | Iustin Pop | length are restricted. Currently the maximum length is 128 characters, |

1081 | c71a1a3d | Iustin Pop | there can be at most 4096 tags per object, and the set of characters is |

1082 | c71a1a3d | Iustin Pop | comprised by alphanumeric characters and additionally ``.+*/:-``. |

1083 | c71a1a3d | Iustin Pop | |

1084 | c71a1a3d | Iustin Pop | Operations |

1085 | c71a1a3d | Iustin Pop | ++++++++++ |

1086 | c71a1a3d | Iustin Pop | |

1087 | c71a1a3d | Iustin Pop | Tags can be added via ``add-tags``:: |

1088 | c71a1a3d | Iustin Pop | |

1089 | c71a1a3d | Iustin Pop | gnt-instance add-tags INSTANCE a b c |

1090 | c71a1a3d | Iustin Pop | gnt-node add-tags INSTANCE a b c |

1091 | c71a1a3d | Iustin Pop | gnt-cluster add-tags a b c |

1092 | c71a1a3d | Iustin Pop | |

1093 | c71a1a3d | Iustin Pop | |

1094 | c71a1a3d | Iustin Pop | The above commands add three tags to an instance, to a node and to the |

1095 | c71a1a3d | Iustin Pop | cluster. Note that the cluster command only takes tags as arguments, |

1096 | c71a1a3d | Iustin Pop | whereas the node and instance commands first required the node and |

1097 | c71a1a3d | Iustin Pop | instance name. |

1098 | c71a1a3d | Iustin Pop | |

1099 | c71a1a3d | Iustin Pop | Tags can also be added from a file, via the ``--from=FILENAME`` |

1100 | c71a1a3d | Iustin Pop | argument. The file is expected to contain one tag per line. |

1101 | c71a1a3d | Iustin Pop | |

1102 | c71a1a3d | Iustin Pop | Tags can also be remove via a syntax very similar to the add one:: |

1103 | c71a1a3d | Iustin Pop | |

1104 | c71a1a3d | Iustin Pop | gnt-instance remove-tags INSTANCE a b c |

1105 | c71a1a3d | Iustin Pop | |

1106 | c71a1a3d | Iustin Pop | And listed via:: |

1107 | c71a1a3d | Iustin Pop | |

1108 | c71a1a3d | Iustin Pop | gnt-instance list-tags |

1109 | c71a1a3d | Iustin Pop | gnt-node list-tags |

1110 | c71a1a3d | Iustin Pop | gnt-cluster list-tags |

1111 | c71a1a3d | Iustin Pop | |

1112 | c71a1a3d | Iustin Pop | Global tag search |

1113 | c71a1a3d | Iustin Pop | +++++++++++++++++ |

1114 | c71a1a3d | Iustin Pop | |

1115 | c71a1a3d | Iustin Pop | It is also possible to execute a global search on the all tags defined |

1116 | c71a1a3d | Iustin Pop | in the cluster configuration, via a cluster command:: |

1117 | c71a1a3d | Iustin Pop | |

1118 | c71a1a3d | Iustin Pop | gnt-cluster search-tags REGEXP |

1119 | c71a1a3d | Iustin Pop | |

1120 | c71a1a3d | Iustin Pop | The parameter expected is a regular expression (see |

1121 | c71a1a3d | Iustin Pop | :manpage:`regex(7)`). This will return all tags that match the search, |

1122 | c71a1a3d | Iustin Pop | together with the object they are defined in (the names being show in a |

1123 | c71a1a3d | Iustin Pop | hierarchical kind of way):: |

1124 | c71a1a3d | Iustin Pop | |

1125 | c71a1a3d | Iustin Pop | node1# gnt-cluster search-tags o |

1126 | c71a1a3d | Iustin Pop | /cluster foo |

1127 | c71a1a3d | Iustin Pop | /instances/instance1 owner:bar |

1128 | c71a1a3d | Iustin Pop | |

1129 | c71a1a3d | Iustin Pop | |

1130 | c71a1a3d | Iustin Pop | Job operations |

1131 | c71a1a3d | Iustin Pop | -------------- |

1132 | c71a1a3d | Iustin Pop | |

1133 | c71a1a3d | Iustin Pop | The various jobs submitted by the instance/node/cluster commands can be |

1134 | c71a1a3d | Iustin Pop | examined, canceled and archived by various invocations of the |

1135 | c71a1a3d | Iustin Pop | ``gnt-job`` command. |

1136 | c71a1a3d | Iustin Pop | |

1137 | c71a1a3d | Iustin Pop | First is the job list command:: |

1138 | c71a1a3d | Iustin Pop | |

1139 | c71a1a3d | Iustin Pop | node1# gnt-job list |

1140 | c71a1a3d | Iustin Pop | 17771 success INSTANCE_QUERY_DATA |

1141 | c71a1a3d | Iustin Pop | 17773 success CLUSTER_VERIFY_DISKS |

1142 | c71a1a3d | Iustin Pop | 17775 success CLUSTER_REPAIR_DISK_SIZES |

1143 | c71a1a3d | Iustin Pop | 17776 error CLUSTER_RENAME(cluster.example.com) |

1144 | c71a1a3d | Iustin Pop | 17780 success CLUSTER_REDIST_CONF |

1145 | c71a1a3d | Iustin Pop | 17792 success INSTANCE_REBOOT(instance1.example.com) |

1146 | c71a1a3d | Iustin Pop | |

1147 | c71a1a3d | Iustin Pop | More detailed information about a job can be found via the ``info`` |

1148 | c71a1a3d | Iustin Pop | command:: |

1149 | c71a1a3d | Iustin Pop | |

1150 | c71a1a3d | Iustin Pop | node1# gnt-job info 17776 |

1151 | c71a1a3d | Iustin Pop | Job ID: 17776 |

1152 | c71a1a3d | Iustin Pop | Status: error |

1153 | c71a1a3d | Iustin Pop | Received: 2009-10-25 23:18:02.180569 |

1154 | c71a1a3d | Iustin Pop | Processing start: 2009-10-25 23:18:02.200335 (delta 0.019766s) |

1155 | c71a1a3d | Iustin Pop | Processing end: 2009-10-25 23:18:02.279743 (delta 0.079408s) |

1156 | c71a1a3d | Iustin Pop | Total processing time: 0.099174 seconds |

1157 | c71a1a3d | Iustin Pop | Opcodes: |

1158 | c71a1a3d | Iustin Pop | OP_CLUSTER_RENAME |

1159 | c71a1a3d | Iustin Pop | Status: error |

1160 | c71a1a3d | Iustin Pop | Processing start: 2009-10-25 23:18:02.200335 |

1161 | c71a1a3d | Iustin Pop | Processing end: 2009-10-25 23:18:02.252282 |

1162 | c71a1a3d | Iustin Pop | Input fields: |

1163 | c71a1a3d | Iustin Pop | name: cluster.example.com |

1164 | c71a1a3d | Iustin Pop | Result: |

1165 | c71a1a3d | Iustin Pop | OpPrereqError |

1166 | c71a1a3d | Iustin Pop | [Neither the name nor the IP address of the cluster has changed] |

1167 | c71a1a3d | Iustin Pop | Execution log: |

1168 | c71a1a3d | Iustin Pop | |

1169 | c71a1a3d | Iustin Pop | During the execution of a job, it's possible to follow the output of a |

1170 | c71a1a3d | Iustin Pop | job, similar to the log that one get from the ``gnt-`` commands, via the |

1171 | c71a1a3d | Iustin Pop | watch command:: |

1172 | c71a1a3d | Iustin Pop | |

1173 | c71a1a3d | Iustin Pop | node1# gnt-instance add --submit … instance1 |

1174 | c71a1a3d | Iustin Pop | JobID: 17818 |

1175 | c71a1a3d | Iustin Pop | node1# gnt-job watch 17818 |

1176 | c71a1a3d | Iustin Pop | Output from job 17818 follows |

1177 | c71a1a3d | Iustin Pop | ----------------------------- |

1178 | c71a1a3d | Iustin Pop | Mon Oct 26 00:22:48 2009 - INFO: Selected nodes for instance instance1 via iallocator dumb: node1, node2 |

1179 | c71a1a3d | Iustin Pop | Mon Oct 26 00:22:49 2009 * creating instance disks... |

1180 | c71a1a3d | Iustin Pop | Mon Oct 26 00:22:52 2009 adding instance instance1 to cluster config |

1181 | c71a1a3d | Iustin Pop | Mon Oct 26 00:22:52 2009 - INFO: Waiting for instance instance1 to sync disks. |

1182 | c71a1a3d | Iustin Pop | … |

1183 | c71a1a3d | Iustin Pop | Mon Oct 26 00:23:03 2009 creating os for instance xen-devi-18.fra.corp.google.com on node mpgntac4.fra.corp.google.com |

1184 | c71a1a3d | Iustin Pop | Mon Oct 26 00:23:03 2009 * running the instance OS create scripts... |

1185 | c71a1a3d | Iustin Pop | Mon Oct 26 00:23:13 2009 * starting instance... |

1186 | c71a1a3d | Iustin Pop | node1# |

1187 | c71a1a3d | Iustin Pop | |

1188 | c71a1a3d | Iustin Pop | This is useful if you need to follow a job's progress from multiple |

1189 | c71a1a3d | Iustin Pop | terminals. |

1190 | c71a1a3d | Iustin Pop | |

1191 | c71a1a3d | Iustin Pop | A job that has not yet started to run can be canceled:: |

1192 | c71a1a3d | Iustin Pop | |

1193 | c71a1a3d | Iustin Pop | node1# gnt-job cancel 17810 |

1194 | c71a1a3d | Iustin Pop | |

1195 | c71a1a3d | Iustin Pop | But not one that has already started execution:: |

1196 | c71a1a3d | Iustin Pop | |

1197 | c71a1a3d | Iustin Pop | node1# gnt-job cancel 17805 |

1198 | c71a1a3d | Iustin Pop | Job 17805 is no longer waiting in the queue |

1199 | c71a1a3d | Iustin Pop | |

1200 | c71a1a3d | Iustin Pop | There are two queues for jobs: the *current* and the *archive* |

1201 | c71a1a3d | Iustin Pop | queue. Jobs are initially submitted to the current queue, and they stay |

1202 | c71a1a3d | Iustin Pop | in that queue until they have finished execution (either successfully or |

1203 | c71a1a3d | Iustin Pop | not). At that point, they can be moved into the archive queue, and the |

1204 | c71a1a3d | Iustin Pop | ganeti-watcher script will do this automatically after 6 hours. The |

1205 | c71a1a3d | Iustin Pop | ganeti-cleaner script will remove the jobs from the archive directory |

1206 | c71a1a3d | Iustin Pop | after three weeks. |

1207 | c71a1a3d | Iustin Pop | |

1208 | c71a1a3d | Iustin Pop | Note that only jobs in the current queue can be viewed via the list and |

1209 | c71a1a3d | Iustin Pop | info commands; Ganeti itself doesn't examine the archive directory. If |

1210 | c71a1a3d | Iustin Pop | you need to see an older job, either move the file manually in the |

1211 | c71a1a3d | Iustin Pop | top-level queue directory, or look at its contents (it's a |

1212 | c71a1a3d | Iustin Pop | JSON-formatted file). |

1213 | c71a1a3d | Iustin Pop | |

1214 | c71a1a3d | Iustin Pop | Ganeti tools |

1215 | c71a1a3d | Iustin Pop | ------------ |

1216 | c71a1a3d | Iustin Pop | |

1217 | c71a1a3d | Iustin Pop | Beside the usual ``gnt-`` and ``ganeti-`` commands which are provided |

1218 | c71a1a3d | Iustin Pop | and installed in ``$prefix/sbin`` at install time, there are a couple of |

1219 | c71a1a3d | Iustin Pop | other tools installed which are used seldom but can be helpful in some |

1220 | c71a1a3d | Iustin Pop | cases. |

1221 | c71a1a3d | Iustin Pop | |

1222 | c71a1a3d | Iustin Pop | lvmstrap |

1223 | c71a1a3d | Iustin Pop | ++++++++ |

1224 | c71a1a3d | Iustin Pop | |

1225 | c71a1a3d | Iustin Pop | The ``lvmstrap`` tool, introduced in :ref:`configure-lvm-label` section, |

1226 | c71a1a3d | Iustin Pop | has two modes of operation: |

1227 | c71a1a3d | Iustin Pop | |

1228 | c71a1a3d | Iustin Pop | - ``diskinfo`` shows the discovered disks on the system and their status |

1229 | c71a1a3d | Iustin Pop | - ``create`` takes all not-in-use disks and creates a volume group out |

1230 | c71a1a3d | Iustin Pop | of them |

1231 | c71a1a3d | Iustin Pop | |

1232 | c71a1a3d | Iustin Pop | .. warning:: The ``create`` argument to this command causes data-loss! |

1233 | c71a1a3d | Iustin Pop | |

1234 | c71a1a3d | Iustin Pop | cfgupgrade |

1235 | c71a1a3d | Iustin Pop | ++++++++++ |

1236 | c71a1a3d | Iustin Pop | |

1237 | c71a1a3d | Iustin Pop | The ``cfgupgrade`` tools is used to upgrade between major (and minor) |

1238 | c71a1a3d | Iustin Pop | Ganeti versions. Point-releases are usually transparent for the admin. |

1239 | c71a1a3d | Iustin Pop | |

1240 | c71a1a3d | Iustin Pop | More information about the upgrade procedure is listed on the wiki at |

1241 | c71a1a3d | Iustin Pop | http://code.google.com/p/ganeti/wiki/UpgradeNotes. |

1242 | c71a1a3d | Iustin Pop | |

1243 | c71a1a3d | Iustin Pop | cfgshell |

1244 | c71a1a3d | Iustin Pop | ++++++++ |

1245 | c71a1a3d | Iustin Pop | |

1246 | c71a1a3d | Iustin Pop | .. note:: This command is not actively maintained; make sure you backup |

1247 | c71a1a3d | Iustin Pop | your configuration before using it |

1248 | c71a1a3d | Iustin Pop | |

1249 | c71a1a3d | Iustin Pop | This can be used as an alternative to direct editing of the |

1250 | c71a1a3d | Iustin Pop | main configuration file if Ganeti has a bug and prevents you, for |

1251 | c71a1a3d | Iustin Pop | example, from removing an instance or a node from the configuration |

1252 | c71a1a3d | Iustin Pop | file. |

1253 | c71a1a3d | Iustin Pop | |

1254 | c71a1a3d | Iustin Pop | .. _burnin-label: |

1255 | c71a1a3d | Iustin Pop | |

1256 | c71a1a3d | Iustin Pop | burnin |

1257 | c71a1a3d | Iustin Pop | ++++++ |

1258 | c71a1a3d | Iustin Pop | |

1259 | c71a1a3d | Iustin Pop | .. warning:: This command will erase existing instances if given as |

1260 | c71a1a3d | Iustin Pop | arguments! |

1261 | c71a1a3d | Iustin Pop | |

1262 | c71a1a3d | Iustin Pop | This tool is used to exercise either the hardware of machines or |

1263 | c71a1a3d | Iustin Pop | alternatively the Ganeti software. It is safe to run on an existing |

1264 | c71a1a3d | Iustin Pop | cluster **as long as you don't pass it existing instance names**. |

1265 | c71a1a3d | Iustin Pop | |

1266 | c71a1a3d | Iustin Pop | The command will, by default, execute a comprehensive set of operations |

1267 | c71a1a3d | Iustin Pop | against a list of instances, these being: |

1268 | c71a1a3d | Iustin Pop | |

1269 | c71a1a3d | Iustin Pop | - creation |

1270 | c71a1a3d | Iustin Pop | - disk replacement (for redundant instances) |

1271 | c71a1a3d | Iustin Pop | - failover and migration (for redundant instances) |

1272 | c71a1a3d | Iustin Pop | - move (for non-redundant instances) |

1273 | c71a1a3d | Iustin Pop | - disk growth |

1274 | c71a1a3d | Iustin Pop | - add disks, remove disk |

1275 | c71a1a3d | Iustin Pop | - add NICs, remove NICs |

1276 | c71a1a3d | Iustin Pop | - export and then import |

1277 | c71a1a3d | Iustin Pop | - rename |

1278 | c71a1a3d | Iustin Pop | - reboot |

1279 | c71a1a3d | Iustin Pop | - shutdown/startup |

1280 | c71a1a3d | Iustin Pop | - and finally removal of the test instances |

1281 | c71a1a3d | Iustin Pop | |

1282 | c71a1a3d | Iustin Pop | Executing all these operations will test that the hardware performs |

1283 | c71a1a3d | Iustin Pop | well: the creation, disk replace, disk add and disk growth will exercise |

1284 | c71a1a3d | Iustin Pop | the storage and network; the migrate command will test the memory of the |

1285 | c71a1a3d | Iustin Pop | systems. Depending on the passed options, it can also test that the |

1286 | c71a1a3d | Iustin Pop | instance OS definitions are executing properly the rename, import and |

1287 | c71a1a3d | Iustin Pop | export operations. |

1288 | c71a1a3d | Iustin Pop | |

1289 | c71a1a3d | Iustin Pop | Other Ganeti projects |

1290 | c71a1a3d | Iustin Pop | --------------------- |

1291 | c71a1a3d | Iustin Pop | |

1292 | c71a1a3d | Iustin Pop | There are two other Ganeti-related projects that can be useful in a |

1293 | c71a1a3d | Iustin Pop | Ganeti deployment. These can be downloaded from the project site |

1294 | c71a1a3d | Iustin Pop | (http://code.google.com/p/ganeti/) and the repositories are also on the |

1295 | c71a1a3d | Iustin Pop | project git site (http://git.ganeti.org). |

1296 | c71a1a3d | Iustin Pop | |

1297 | c71a1a3d | Iustin Pop | NBMA tools |

1298 | c71a1a3d | Iustin Pop | ++++++++++ |

1299 | c71a1a3d | Iustin Pop | |

1300 | c71a1a3d | Iustin Pop | The ``ganeti-nbma`` software is designed to allow instances to live on a |

1301 | c71a1a3d | Iustin Pop | separate, virtual network from the nodes, and in an environment where |

1302 | c71a1a3d | Iustin Pop | nodes are not guaranteed to be able to reach each other via multicasting |

1303 | c71a1a3d | Iustin Pop | or broadcasting. For more information see the README in the source |

1304 | c71a1a3d | Iustin Pop | archive. |

1305 | c71a1a3d | Iustin Pop | |

1306 | c71a1a3d | Iustin Pop | ganeti-htools |

1307 | c71a1a3d | Iustin Pop | +++++++++++++ |

1308 | c71a1a3d | Iustin Pop | |

1309 | c71a1a3d | Iustin Pop | The ``ganeti-htools`` software consists of a set of tools: |

1310 | c71a1a3d | Iustin Pop | |

1311 | c71a1a3d | Iustin Pop | - ``hail``: an advanced iallocator script compared to Ganeti's builtin |

1312 | c71a1a3d | Iustin Pop | one |

1313 | c71a1a3d | Iustin Pop | - ``hbal``: a tool for rebalancing the cluster, i.e. moving instances |

1314 | c71a1a3d | Iustin Pop | around in order to better use the resources on the nodes |

1315 | c71a1a3d | Iustin Pop | - ``hspace``: a tool for estimating the available capacity of a cluster, |

1316 | c71a1a3d | Iustin Pop | so that capacity planning can be done efficiently |

1317 | c71a1a3d | Iustin Pop | |

1318 | c71a1a3d | Iustin Pop | For more information and installation instructions, see the README file |

1319 | c71a1a3d | Iustin Pop | in the source archive. |

1320 | c71a1a3d | Iustin Pop | |

1321 | 558fd122 | Michael Hanselmann | .. vim: set textwidth=72 : |

1322 | c71a1a3d | Iustin Pop | .. Local Variables: |

1323 | c71a1a3d | Iustin Pop | .. mode: rst |

1324 | c71a1a3d | Iustin Pop | .. fill-column: 72 |

1325 | c71a1a3d | Iustin Pop | .. End: |