4 .. highlight:: shell-example
6 This document details the steps needed to upgrade a cluster to newer versions
9 As a general rule the node daemons need to be restarted after each software
10 upgrade; if using the provided example init.d script, this means running the
11 following command on all nodes::
13 $ /etc/init.d/ganeti restart
19 Starting with Ganeti 2.0, upgrades between revisions (e.g. 2.1.0 to 2.1.1)
20 should not need manual intervention. As a safety measure, minor releases (e.g.
21 2.1.3 to 2.2.0) require the ``cfgupgrade`` command for changing the
22 configuration version. Below you find the steps necessary to upgrade between
25 To run commands on all nodes, the `distributed shell (dsh)
26 <http://www.netfort.gr.jp/~dancer/software/dsh.html.en>`_ can be used, e.g.
27 ``dsh -M -F 8 -f /var/lib/ganeti/ssconf_online_nodes gnt-cluster --version``.
29 #. Ensure no jobs are running (master node only)::
33 #. Pause the watcher for an hour (master node only)::
35 $ gnt-cluster watcher pause 1h
37 #. Stop all daemons on all nodes::
39 $ /etc/init.d/ganeti stop
41 #. Backup old configuration (master node only)::
43 $ tar czf /var/lib/ganeti-$(date +\%FT\%T).tar.gz -C /var/lib ganeti
45 #. Install new Ganeti version on all nodes
46 #. Run cfgupgrade on the master node::
48 $ /usr/lib/ganeti/tools/cfgupgrade --verbose --dry-run
49 $ /usr/lib/ganeti/tools/cfgupgrade --verbose
51 (``cfgupgrade`` supports a number of parameters, run it with
52 ``--help`` for more information)
54 #. Upgrade the directory permissions on all nodes::
56 $ /usr/lib/ganeti/ensure-dirs --full-run
58 #. Create the (missing) required users and make users part of the required
61 $ /usr/lib/ganeti/tools/users-setup
63 This will ask for confirmation. To execute directly, add the ``--yes-do-it``
66 #. Restart daemons on all nodes::
68 $ /etc/init.d/ganeti restart
70 #. Re-distribute configuration (master node only)::
72 $ gnt-cluster redist-conf
74 #. If you use file storage, check that the ``/etc/ganeti/file-storage-paths``
75 is correct on all nodes. For security reasons it's not copied
76 automatically, but it can be copied manually via::
78 $ gnt-cluster copyfile /etc/ganeti/file-storage-paths
80 #. Restart daemons again on all nodes::
82 $ /etc/init.d/ganeti restart
84 #. Enable the watcher again (master node only)::
86 $ gnt-cluster watcher continue
88 #. Verify cluster (master node only)::
95 For going back between revisions (e.g. 2.1.1 to 2.1.0) no manual
96 intervention is required, as for upgrades.
98 Starting from version 2.8, ``cfgupgrade`` supports ``--downgrade``
99 option to bring the configuration back to the previous stable version.
100 This is useful if you upgrade Ganeti and after some time you run into
101 problems with the new version. You can downgrade the configuration
102 without losing the changes made since the upgrade. Any feature not
103 supported by the old version will be removed from the configuration, of
104 course, but you get a warning about it. If there is any new feature and
105 you haven't changed from its default value, you don't have to worry
106 about it, as it will get the same value whenever you'll upgrade again.
108 The procedure is similar to upgrading, but please notice that you have to
109 revert the configuration **before** installing the old version.
111 #. Ensure no jobs are running (master node only)::
115 #. Pause the watcher for an hour (master node only)::
117 $ gnt-cluster watcher pause 1h
119 #. Stop all daemons on all nodes::
121 $ /etc/init.d/ganeti stop
123 #. Backup old configuration (master node only)::
125 $ tar czf /var/lib/ganeti-$(date +\%FT\%T).tar.gz -C /var/lib ganeti
127 #. Run cfgupgrade on the master node::
129 $ /usr/lib/ganeti/tools/cfgupgrade --verbose --downgrade --dry-run
130 $ /usr/lib/ganeti/tools/cfgupgrade --verbose --downgrade
132 You may want to copy all the messages about features that have been
133 removed during the downgrade, in case you want to restore them when
136 #. Install the old Ganeti version on all nodes
138 NB: in Ganeti 2.8, the ``cmdlib.py`` file was split into a series of files
139 contained in the ``cmdlib`` directory. If Ganeti is installed from sources
140 and not from a package, while downgrading Ganeti to a pre-2.8
141 version it is important to remember to remove the ``cmdlib`` directory
142 from the directory containing the Ganeti python files (which usually is
143 ``${PREFIX}/lib/python${VERSION}/dist-packages/ganeti``).
144 A simpler upgrade/downgrade procedure will be made available in future
147 #. Restart daemons on all nodes::
149 $ /etc/init.d/ganeti restart
151 #. Re-distribute configuration (master node only)::
153 $ gnt-cluster redist-conf
155 #. Restart daemons again on all nodes::
157 $ /etc/init.d/ganeti restart
159 #. Enable the watcher again (master node only)::
161 $ gnt-cluster watcher continue
163 #. Verify cluster (master node only)::
174 No changes needed except restarting the daemon; but rollback to 2.0.3 might
175 require configuration editing.
177 If you're using Xen-HVM instances, please double-check the network
178 configuration (``nic_type`` parameter) as the defaults might have changed:
179 2.0.4 adds any missing configuration items and depending on the version of the
180 software the cluster has been installed with, some new keys might have been
186 Between 2.0.1 and 2.0.2 there have been some changes in the handling of block
187 devices, which can cause some issues. 2.0.3 was then released which adds two
188 new options/commands to fix this issue.
190 If you use DRBD-type instances and see problems in instance start or
191 activate-disks with messages from DRBD about "lower device too small" or
192 similar, it is recoomended to:
194 #. Run ``gnt-instance activate-disks --ignore-size $instance`` for each
195 of the affected instances
196 #. Then run ``gnt-cluster repair-disk-sizes`` which will check that
197 instances have the correct disk sizes
204 - Ganeti 1.2.7 is currently installed
205 - All instances have been migrated from DRBD 0.7 to DRBD 8.x (i.e. no
206 ``remote_raid1`` disk template)
207 - Upgrade to Ganeti 2.0.0~rc2 or later (~rc1 and earlier don't have the needed
210 In the below steps, replace :file:`/var/lib` with ``$libdir`` if Ganeti was not
211 installed with this prefix (e.g. :file:`/usr/local/var`). Same for
214 Execution (all steps are required in the order given):
216 #. Make a backup of the current configuration, for safety::
218 $ cp -a /var/lib/ganeti /var/lib/ganeti-1.2.backup
220 #. Stop all instances::
222 $ gnt-instance stop --all
224 #. Make sure no DRBD device are in use, the following command should show no
227 $ gnt-cluster command grep cs: /proc/drbd | grep -v cs:Unconf
229 #. Stop the node daemons and rapi daemon on all nodes (note: should be logged
230 in not via the cluster name, but the master node name, as the command below
231 will remove the cluster ip from the master node)::
233 $ gnt-cluster command /etc/init.d/ganeti stop
235 #. Install the new software on all nodes, either from packaging (if available)
236 or from sources; the master daemon will not start but give error messages
237 about wrong configuration file, which is normal
238 #. Upgrade the configuration file::
240 $ /usr/lib/ganeti/tools/cfgupgrade12 -v --dry-run
241 $ /usr/lib/ganeti/tools/cfgupgrade12 -v
243 #. Make sure ``ganeti-noded`` is running on all nodes (and start it if
245 #. Start the master daemon::
249 #. Check that a simple node-list works::
253 #. Redistribute updated configuration to all nodes::
255 $ gnt-cluster redist-conf
256 $ gnt-cluster copyfile /var/lib/ganeti/known_hosts
258 #. Optional: if needed, install RAPI-specific certificates under
259 :file:`/var/lib/ganeti/rapi.pem` and run::
261 $ gnt-cluster copyfile /var/lib/ganeti/rapi.pem
263 #. Run a cluster verify, this should show no problems::
267 #. Remove some obsolete files::
269 $ gnt-cluster command rm /var/lib/ganeti/ssconf_node_pass
270 $ gnt-cluster command rm /var/lib/ganeti/ssconf_hypervisor
272 #. Update the xen pvm (if this was a pvm cluster) setting for 1.2
275 $ gnt-cluster modify -H xen-pvm:root_path=/dev/sda
277 #. Depending on your setup, you might also want to reset the initrd parameter::
279 $ gnt-cluster modify -H xen-pvm:initrd_path=/boot/initrd-2.6-xenU
281 #. Reset the instance autobalance setting to default::
283 $ for i in $(gnt-instance list -o name --no-headers); do \
284 gnt-instance modify -B auto_balance=default $i; \
287 #. Optional: start the RAPI demon::
291 #. Restart instances::
293 $ gnt-instance start --force-multiple --all
295 At this point, ``gnt-cluster verify`` should show no errors and the migration
301 1.2.4 to any other higher 1.2 version
302 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
304 No changes needed. Rollback will usually require manual edit of the
310 No changes needed. Note that going back from 1.2.4 to 1.2.3 will require manual
311 edit of the configuration file (since we added some HVM-related new
317 No changes needed. Note that the drbd7-to-8 upgrade tool does a disk format
318 change for the DRBD metadata, so in theory this might be **risky**. It is
319 advised to have (good) backups before doing the upgrade.
329 No changes needed. Only some bugfixes and new additions that don't affect
332 1.2.0 beta 3 to 1.2.0
333 ~~~~~~~~~~~~~~~~~~~~~
337 1.2.0 beta 2 to beta 3
338 ~~~~~~~~~~~~~~~~~~~~~~
340 No changes needed. A new version of the debian-etch-instance OS (0.3) has been
341 released, but upgrading it is not required.
343 1.2.0 beta 1 to beta 2
344 ~~~~~~~~~~~~~~~~~~~~~~
346 Beta 2 switched the config file format to JSON. Steps to upgrade:
348 #. Stop the daemons (``/etc/init.d/ganeti stop``) on all nodes
349 #. Disable the cron job (default is :file:`/etc/cron.d/ganeti`)
350 #. Install the new version
351 #. Make a backup copy of the config file
352 #. Upgrade the config file using the following command::
354 $ /usr/share/ganeti/cfgupgrade --verbose /var/lib/ganeti/config.data
356 #. Start the daemons and run ``gnt-cluster info``, ``gnt-node list`` and
357 ``gnt-instance list`` to check if the upgrade process finished successfully
359 The OS definition also need to be upgraded. There is a new version of the
360 debian-etch-instance OS (0.2) that goes along with beta 2.
362 .. vim: set textwidth=72 :