4 .. highlight:: shell-example
6 This document details the steps needed to upgrade a cluster to newer versions
9 As a general rule the node daemons need to be restarted after each software
10 upgrade; if using the provided example init.d script, this means running the
11 following command on all nodes::
13 $ /etc/init.d/ganeti restart
19 Starting with Ganeti 2.0, upgrades between revisions (e.g. 2.1.0 to 2.1.1)
20 should not need manual intervention. As a safety measure, minor releases (e.g.
21 2.1.3 to 2.2.0) require the ``cfgupgrade`` command for changing the
22 configuration version. Below you find the steps necessary to upgrade between
25 To run commands on all nodes, the `distributed shell (dsh)
26 <http://www.netfort.gr.jp/~dancer/software/dsh.html.en>`_ can be used, e.g.
27 ``dsh -M -F 8 -f /var/lib/ganeti/ssconf_online_nodes gnt-cluster --version``.
29 #. Ensure no jobs are running (master node only)::
33 #. Pause the watcher for an hour (master node only)::
35 $ gnt-cluster watcher pause 1h
37 #. Stop all daemons on all nodes::
39 $ /etc/init.d/ganeti stop
41 #. Backup old configuration (master node only)::
43 $ tar czf /var/lib/ganeti-$(date +\%FT\%T).tar.gz -C /var/lib ganeti
45 #. Install new Ganeti version on all nodes
46 #. Run cfgupgrade on the master node::
48 $ /usr/lib/ganeti/tools/cfgupgrade --verbose --dry-run
49 $ /usr/lib/ganeti/tools/cfgupgrade --verbose
51 (``cfgupgrade`` supports a number of parameters, run it with
52 ``--help`` for more information)
54 #. Restart daemons on all nodes::
56 $ /etc/init.d/ganeti restart
58 #. Re-distribute configuration (master node only)::
60 $ gnt-cluster redist-conf
62 #. Restart daemons again on all nodes::
64 $ /etc/init.d/ganeti restart
66 #. Enable the watcher again (master node only)::
68 $ gnt-cluster watcher continue
70 #. Verify cluster (master node only)::
81 No changes needed except restarting the daemon; but rollback to 2.0.3 might
82 require configuration editing.
84 If you're using Xen-HVM instances, please double-check the network
85 configuration (``nic_type`` parameter) as the defaults might have changed:
86 2.0.4 adds any missing configuration items and depending on the version of the
87 software the cluster has been installed with, some new keys might have been
93 Between 2.0.1 and 2.0.2 there have been some changes in the handling of block
94 devices, which can cause some issues. 2.0.3 was then released which adds two
95 new options/commands to fix this issue.
97 If you use DRBD-type instances and see problems in instance start or
98 activate-disks with messages from DRBD about "lower device too small" or
99 similar, it is recoomended to:
101 #. Run ``gnt-instance activate-disks --ignore-size $instance`` for each
102 of the affected instances
103 #. Then run ``gnt-cluster repair-disk-sizes`` which will check that
104 instances have the correct disk sizes
111 - Ganeti 1.2.7 is currently installed
112 - All instances have been migrated from DRBD 0.7 to DRBD 8.x (i.e. no
113 ``remote_raid1`` disk template)
114 - Upgrade to Ganeti 2.0.0~rc2 or later (~rc1 and earlier don't have the needed
117 In the below steps, replace :file:`/var/lib` with ``$libdir`` if Ganeti was not
118 installed with this prefix (e.g. :file:`/usr/local/var`). Same for
121 Execution (all steps are required in the order given):
123 #. Make a backup of the current configuration, for safety::
125 $ cp -a /var/lib/ganeti /var/lib/ganeti-1.2.backup
127 #. Stop all instances::
129 $ gnt-instance stop --all
131 #. Make sure no DRBD device are in use, the following command should show no
134 $ gnt-cluster command grep cs: /proc/drbd | grep -v cs:Unconf
136 #. Stop the node daemons and rapi daemon on all nodes (note: should be logged
137 in not via the cluster name, but the master node name, as the command below
138 will remove the cluster ip from the master node)::
140 $ gnt-cluster command /etc/init.d/ganeti stop
142 #. Install the new software on all nodes, either from packaging (if available)
143 or from sources; the master daemon will not start but give error messages
144 about wrong configuration file, which is normal
145 #. Upgrade the configuration file::
147 $ /usr/lib/ganeti/tools/cfgupgrade12 -v --dry-run
148 $ /usr/lib/ganeti/tools/cfgupgrade12 -v
150 #. Make sure ``ganeti-noded`` is running on all nodes (and start it if
152 #. Start the master daemon::
156 #. Check that a simple node-list works::
160 #. Redistribute updated configuration to all nodes::
162 $ gnt-cluster redist-conf
163 $ gnt-cluster copyfile /var/lib/ganeti/known_hosts
165 #. Optional: if needed, install RAPI-specific certificates under
166 :file:`/var/lib/ganeti/rapi.pem` and run::
168 $ gnt-cluster copyfile /var/lib/ganeti/rapi.pem
170 #. Run a cluster verify, this should show no problems::
174 #. Remove some obsolete files::
176 $ gnt-cluster command rm /var/lib/ganeti/ssconf_node_pass
177 $ gnt-cluster command rm /var/lib/ganeti/ssconf_hypervisor
179 #. Update the xen pvm (if this was a pvm cluster) setting for 1.2
182 $ gnt-cluster modify -H xen-pvm:root_path=/dev/sda
184 #. Depending on your setup, you might also want to reset the initrd parameter::
186 $ gnt-cluster modify -H xen-pvm:initrd_path=/boot/initrd-2.6-xenU
188 #. Reset the instance autobalance setting to default::
190 $ for i in $(gnt-instance list -o name --no-headers); do \
191 gnt-instance modify -B auto_balance=default $i; \
194 #. Optional: start the RAPI demon::
198 #. Restart instances::
200 $ gnt-instance start --force-multiple --all
202 At this point, ``gnt-cluster verify`` should show no errors and the migration
208 1.2.4 to any other higher 1.2 version
209 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
211 No changes needed. Rollback will usually require manual edit of the
217 No changes needed. Note that going back from 1.2.4 to 1.2.3 will require manual
218 edit of the configuration file (since we added some HVM-related new
224 No changes needed. Note that the drbd7-to-8 upgrade tool does a disk format
225 change for the DRBD metadata, so in theory this might be **risky**. It is
226 advised to have (good) backups before doing the upgrade.
236 No changes needed. Only some bugfixes and new additions that don't affect
239 1.2.0 beta 3 to 1.2.0
240 ~~~~~~~~~~~~~~~~~~~~~
244 1.2.0 beta 2 to beta 3
245 ~~~~~~~~~~~~~~~~~~~~~~
247 No changes needed. A new version of the debian-etch-instance OS (0.3) has been
248 released, but upgrading it is not required.
250 1.2.0 beta 1 to beta 2
251 ~~~~~~~~~~~~~~~~~~~~~~
253 Beta 2 switched the config file format to JSON. Steps to upgrade:
255 #. Stop the daemons (``/etc/init.d/ganeti stop``) on all nodes
256 #. Disable the cron job (default is :file:`/etc/cron.d/ganeti`)
257 #. Install the new version
258 #. Make a backup copy of the config file
259 #. Upgrade the config file using the following command::
261 $ /usr/share/ganeti/cfgupgrade --verbose /var/lib/ganeti/config.data
263 #. Start the daemons and run ``gnt-cluster info``, ``gnt-node list`` and
264 ``gnt-instance list`` to check if the upgrade process finished successfully
266 The OS definition also need to be upgraded. There is a new version of the
267 debian-etch-instance OS (0.2) that goes along with beta 2.
269 .. vim: set textwidth=72 :