Statistics
| Branch: | Tag: | Revision:

root / UPGRADE @ 63a3049a

History | View | Annotate | Download (12.4 kB)

1
Upgrade notes
2
=============
3

    
4
.. highlight:: shell-example
5

    
6
This document details the steps needed to upgrade a cluster to newer versions
7
of Ganeti.
8

    
9
As a general rule the node daemons need to be restarted after each software
10
upgrade; if using the provided example init.d script, this means running the
11
following command on all nodes::
12

    
13
    $ /etc/init.d/ganeti restart
14

    
15
2.11 and above
16
--------------
17

    
18
Starting from 2.10 onwards, Ganeti has support for parallely installed versions
19
and automated upgrades. The default configuration for 2.11 and higher already is
20
to install as a parallel version without changing the running version. If both
21
versions, the installed one and the one to upgrade to, are 2.10 or higher, the
22
actual switch of the live version can be carried out by the following command
23
on the master node.::
24

    
25
   $ gnt-cluster upgrade --to 2.11
26

    
27
This will carry out the steps described below in the section on upgrades from
28
2.1 and above. Downgrades to the previous minor version can be done in the same
29
way, specifiying the smaller version on the ``--to`` argument.
30

    
31

    
32
2.11
33
----
34

    
35
When upgrading to 2.11, first apply the instructions of ``2.11 and
36
above``. 2.11 comes with the new feature of enhanced RPC security
37
through client certificates. This features needs to be enabled after the
38
upgrade by::
39

    
40
   $ gnt-cluster renew-crypto --new-node-certificates
41

    
42
Note that new node certificates are generated automatically without
43
warning when upgrading with ``gnt-cluster upgrade``.
44

    
45

    
46
2.1 and above
47
-------------
48

    
49
Starting with Ganeti 2.0, upgrades between revisions (e.g. 2.1.0 to 2.1.1)
50
should not need manual intervention. As a safety measure, minor releases (e.g.
51
2.1.3 to 2.2.0) require the ``cfgupgrade`` command for changing the
52
configuration version. Below you find the steps necessary to upgrade between
53
minor releases.
54

    
55
To run commands on all nodes, the `distributed shell (dsh)
56
<http://www.netfort.gr.jp/~dancer/software/dsh.html.en>`_ can be used, e.g.
57
``dsh -M -F 8 -f /var/lib/ganeti/ssconf_online_nodes gnt-cluster --version``.
58

    
59
#. Ensure no jobs are running (master node only)::
60

    
61
    $ gnt-job list
62

    
63
#. Pause the watcher for an hour (master node only)::
64

    
65
    $ gnt-cluster watcher pause 1h
66

    
67
#. Stop all daemons on all nodes::
68

    
69
    $ /etc/init.d/ganeti stop
70

    
71
#. Backup old configuration (master node only)::
72

    
73
    $ tar czf /var/lib/ganeti-$(date +\%FT\%T).tar.gz -C /var/lib ganeti
74

    
75
#. Install new Ganeti version on all nodes
76
#. Run cfgupgrade on the master node::
77

    
78
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose --dry-run
79
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose
80

    
81
   (``cfgupgrade`` supports a number of parameters, run it with
82
   ``--help`` for more information)
83

    
84
#. Upgrade the directory permissions on all nodes::
85

    
86
    $ /usr/lib/ganeti/ensure-dirs --full-run
87

    
88
#. Create the (missing) required users and make users part of the required
89
   groups on all nodes::
90

    
91
    $ /usr/lib/ganeti/tools/users-setup
92

    
93
   This will ask for confirmation. To execute directly, add the ``--yes-do-it``
94
   option.
95

    
96
#. Restart daemons on all nodes::
97

    
98
    $ /etc/init.d/ganeti restart
99

    
100
#. Re-distribute configuration (master node only)::
101

    
102
    $ gnt-cluster redist-conf
103

    
104
#. If you use file storage, check that the ``/etc/ganeti/file-storage-paths``
105
   is correct on all nodes. For security reasons it's not copied
106
   automatically, but it can be copied manually via::
107

    
108
   $ gnt-cluster copyfile /etc/ganeti/file-storage-paths
109

    
110
#. Restart daemons again on all nodes::
111

    
112
    $ /etc/init.d/ganeti restart
113

    
114
#. Enable the watcher again (master node only)::
115

    
116
    $ gnt-cluster watcher continue
117

    
118
#. Verify cluster (master node only)::
119

    
120
    $ gnt-cluster verify
121

    
122
Reverting an upgrade
123
~~~~~~~~~~~~~~~~~~~~
124

    
125
For going back between revisions (e.g. 2.1.1 to 2.1.0) no manual
126
intervention is required, as for upgrades.
127

    
128
Starting from version 2.8, ``cfgupgrade`` supports ``--downgrade``
129
option to bring the configuration back to the previous stable version.
130
This is useful if you upgrade Ganeti and after some time you run into
131
problems with the new version. You can downgrade the configuration
132
without losing the changes made since the upgrade. Any feature not
133
supported by the old version will be removed from the configuration, of
134
course, but you get a warning about it. If there is any new feature and
135
you haven't changed from its default value, you don't have to worry
136
about it, as it will get the same value whenever you'll upgrade again.
137

    
138
Automatic downgrades
139
....................
140

    
141
From version 2.11 onwards, downgrades can be done by using the
142
``gnt-cluster upgrade`` command.::
143

    
144
  gnt-cluster upgrade --to 2.10
145

    
146
Manual downgrades
147
.................
148

    
149
The procedure is similar to upgrading, but please notice that you have to
150
revert the configuration **before** installing the old version.
151

    
152
#. Ensure no jobs are running (master node only)::
153

    
154
    $ gnt-job list
155

    
156
#. Pause the watcher for an hour (master node only)::
157

    
158
    $ gnt-cluster watcher pause 1h
159

    
160
#. Stop all daemons on all nodes::
161

    
162
    $ /etc/init.d/ganeti stop
163

    
164
#. Backup old configuration (master node only)::
165

    
166
    $ tar czf /var/lib/ganeti-$(date +\%FT\%T).tar.gz -C /var/lib ganeti
167

    
168
#. Run cfgupgrade on the master node::
169

    
170
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose --downgrade --dry-run
171
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose --downgrade
172

    
173
   You may want to copy all the messages about features that have been
174
   removed during the downgrade, in case you want to restore them when
175
   upgrading again.
176

    
177
#. Install the old Ganeti version on all nodes
178

    
179
   NB: in Ganeti 2.8, the ``cmdlib.py`` file was split into a series of files
180
   contained in the ``cmdlib`` directory. If Ganeti is installed from sources
181
   and not from a package, while downgrading Ganeti to a pre-2.8
182
   version it is important to remember to remove the ``cmdlib`` directory
183
   from the directory containing the Ganeti python files (which usually is
184
   ``${PREFIX}/lib/python${VERSION}/dist-packages/ganeti``).
185
   A simpler upgrade/downgrade procedure will be made available in future
186
   versions of Ganeti.
187

    
188
#. Restart daemons on all nodes::
189

    
190
    $ /etc/init.d/ganeti restart
191

    
192
#. Re-distribute configuration (master node only)::
193

    
194
    $ gnt-cluster redist-conf
195

    
196
#. Restart daemons again on all nodes::
197

    
198
    $ /etc/init.d/ganeti restart
199

    
200
#. Enable the watcher again (master node only)::
201

    
202
    $ gnt-cluster watcher continue
203

    
204
#. Verify cluster (master node only)::
205

    
206
    $ gnt-cluster verify
207

    
208
Specific tasks for 2.11 to 2.10 downgrade
209
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
210

    
211
After running ``cfgupgrade``, the ``client.pem`` and
212
``ssconf_master_candidates_certs`` files need to be removed
213
from Ganeti's data directory on all nodes. While this step is
214
not necessary for 2.10 to run cleanly, leaving them will cause
215
problems when upgrading again after the downgrade.
216

    
217
2.0 releases
218
------------
219

    
220
2.0.3 to 2.0.4
221
~~~~~~~~~~~~~~
222

    
223
No changes needed except restarting the daemon; but rollback to 2.0.3 might
224
require configuration editing.
225

    
226
If you're using Xen-HVM instances, please double-check the network
227
configuration (``nic_type`` parameter) as the defaults might have changed:
228
2.0.4 adds any missing configuration items and depending on the version of the
229
software the cluster has been installed with, some new keys might have been
230
added.
231

    
232
2.0.1 to 2.0.2/2.0.3
233
~~~~~~~~~~~~~~~~~~~~
234

    
235
Between 2.0.1 and 2.0.2 there have been some changes in the handling of block
236
devices, which can cause some issues. 2.0.3 was then released which adds two
237
new options/commands to fix this issue.
238

    
239
If you use DRBD-type instances and see problems in instance start or
240
activate-disks with messages from DRBD about "lower device too small" or
241
similar, it is recoomended to:
242

    
243
#. Run ``gnt-instance activate-disks --ignore-size $instance`` for each
244
   of the affected instances
245
#. Then run ``gnt-cluster repair-disk-sizes`` which will check that
246
   instances have the correct disk sizes
247

    
248
1.2 to 2.0
249
----------
250

    
251
Prerequisites:
252

    
253
- Ganeti 1.2.7 is currently installed
254
- All instances have been migrated from DRBD 0.7 to DRBD 8.x (i.e. no
255
  ``remote_raid1`` disk template)
256
- Upgrade to Ganeti 2.0.0~rc2 or later (~rc1 and earlier don't have the needed
257
  upgrade tool)
258

    
259
In the below steps, replace :file:`/var/lib` with ``$libdir`` if Ganeti was not
260
installed with this prefix (e.g. :file:`/usr/local/var`). Same for
261
:file:`/usr/lib`.
262

    
263
Execution (all steps are required in the order given):
264

    
265
#. Make a backup of the current configuration, for safety::
266

    
267
    $ cp -a /var/lib/ganeti /var/lib/ganeti-1.2.backup
268

    
269
#. Stop all instances::
270

    
271
    $ gnt-instance stop --all
272

    
273
#. Make sure no DRBD device are in use, the following command should show no
274
   active minors::
275

    
276
    $ gnt-cluster command grep cs: /proc/drbd | grep -v cs:Unconf
277

    
278
#. Stop the node daemons and rapi daemon on all nodes (note: should be logged
279
   in not via the cluster name, but the master node name, as the command below
280
   will remove the cluster ip from the master node)::
281

    
282
    $ gnt-cluster command /etc/init.d/ganeti stop
283

    
284
#. Install the new software on all nodes, either from packaging (if available)
285
   or from sources; the master daemon will not start but give error messages
286
   about wrong configuration file, which is normal
287
#. Upgrade the configuration file::
288

    
289
    $ /usr/lib/ganeti/tools/cfgupgrade12 -v --dry-run
290
    $ /usr/lib/ganeti/tools/cfgupgrade12 -v
291

    
292
#. Make sure ``ganeti-noded`` is running on all nodes (and start it if
293
   not)
294
#. Start the master daemon::
295

    
296
    $ ganeti-masterd
297

    
298
#. Check that a simple node-list works::
299

    
300
    $ gnt-node list
301

    
302
#. Redistribute updated configuration to all nodes::
303

    
304
    $ gnt-cluster redist-conf
305
    $ gnt-cluster copyfile /var/lib/ganeti/known_hosts
306

    
307
#. Optional: if needed, install RAPI-specific certificates under
308
   :file:`/var/lib/ganeti/rapi.pem` and run::
309

    
310
    $ gnt-cluster copyfile /var/lib/ganeti/rapi.pem
311

    
312
#. Run a cluster verify, this should show no problems::
313

    
314
    $ gnt-cluster verify
315

    
316
#. Remove some obsolete files::
317

    
318
    $ gnt-cluster command rm /var/lib/ganeti/ssconf_node_pass
319
    $ gnt-cluster command rm /var/lib/ganeti/ssconf_hypervisor
320

    
321
#. Update the xen pvm (if this was a pvm cluster) setting for 1.2
322
   compatibility::
323

    
324
    $ gnt-cluster modify -H xen-pvm:root_path=/dev/sda
325

    
326
#. Depending on your setup, you might also want to reset the initrd parameter::
327

    
328
    $ gnt-cluster modify -H xen-pvm:initrd_path=/boot/initrd-2.6-xenU
329

    
330
#. Reset the instance autobalance setting to default::
331

    
332
    $ for i in $(gnt-instance list -o name --no-headers); do \
333
       gnt-instance modify -B auto_balance=default $i; \
334
      done
335

    
336
#. Optional: start the RAPI demon::
337

    
338
    $ ganeti-rapi
339

    
340
#. Restart instances::
341

    
342
    $ gnt-instance start --force-multiple --all
343

    
344
At this point, ``gnt-cluster verify`` should show no errors and the migration
345
is complete.
346

    
347
1.2 releases
348
------------
349

    
350
1.2.4 to any other higher 1.2 version
351
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
352

    
353
No changes needed. Rollback will usually require manual edit of the
354
configuration file.
355

    
356
1.2.3 to 1.2.4
357
~~~~~~~~~~~~~~
358

    
359
No changes needed. Note that going back from 1.2.4 to 1.2.3 will require manual
360
edit of the configuration file (since we added some HVM-related new
361
attributes).
362

    
363
1.2.2 to 1.2.3
364
~~~~~~~~~~~~~~
365

    
366
No changes needed. Note that the drbd7-to-8 upgrade tool does a disk format
367
change for the DRBD metadata, so in theory this might be **risky**. It is
368
advised to have (good) backups before doing the upgrade.
369

    
370
1.2.1 to 1.2.2
371
~~~~~~~~~~~~~~
372

    
373
No changes needed.
374

    
375
1.2.0 to 1.2.1
376
~~~~~~~~~~~~~~
377

    
378
No changes needed. Only some bugfixes and new additions that don't affect
379
existing clusters.
380

    
381
1.2.0 beta 3 to 1.2.0
382
~~~~~~~~~~~~~~~~~~~~~
383

    
384
No changes needed.
385

    
386
1.2.0 beta 2 to beta 3
387
~~~~~~~~~~~~~~~~~~~~~~
388

    
389
No changes needed. A new version of the debian-etch-instance OS (0.3) has been
390
released, but upgrading it is not required.
391

    
392
1.2.0 beta 1 to beta 2
393
~~~~~~~~~~~~~~~~~~~~~~
394

    
395
Beta 2 switched the config file format to JSON. Steps to upgrade:
396

    
397
#. Stop the daemons (``/etc/init.d/ganeti stop``) on all nodes
398
#. Disable the cron job (default is :file:`/etc/cron.d/ganeti`)
399
#. Install the new version
400
#. Make a backup copy of the config file
401
#. Upgrade the config file using the following command::
402

    
403
    $ /usr/share/ganeti/cfgupgrade --verbose /var/lib/ganeti/config.data
404

    
405
#. Start the daemons and run ``gnt-cluster info``, ``gnt-node list`` and
406
   ``gnt-instance list`` to check if the upgrade process finished successfully
407

    
408
The OS definition also need to be upgraded. There is a new version of the
409
debian-etch-instance OS (0.2) that goes along with beta 2.
410

    
411
.. vim: set textwidth=72 :
412
.. Local Variables:
413
.. mode: rst
414
.. fill-column: 72
415
.. End: