Statistics
| Branch: | Tag: | Revision:

root / UPGRADE @ ef947a42

History | View | Annotate | Download (7.4 kB)

1 35dd762d Michael Hanselmann
Upgrade notes
2 35dd762d Michael Hanselmann
=============
3 35dd762d Michael Hanselmann
4 e1ff0de1 Iustin Pop
.. highlight:: shell-example
5 35dd762d Michael Hanselmann
6 35dd762d Michael Hanselmann
This document details the steps needed to upgrade a cluster to newer versions
7 35dd762d Michael Hanselmann
of Ganeti.
8 35dd762d Michael Hanselmann
9 35dd762d Michael Hanselmann
As a general rule the node daemons need to be restarted after each software
10 35dd762d Michael Hanselmann
upgrade; if using the provided example init.d script, this means running the
11 35dd762d Michael Hanselmann
following command on all nodes::
12 35dd762d Michael Hanselmann
13 e1ff0de1 Iustin Pop
    $ /etc/init.d/ganeti restart
14 35dd762d Michael Hanselmann
15 35dd762d Michael Hanselmann
16 35dd762d Michael Hanselmann
2.1 and above
17 35dd762d Michael Hanselmann
-------------
18 35dd762d Michael Hanselmann
19 35dd762d Michael Hanselmann
Starting with Ganeti 2.0, upgrades between revisions (e.g. 2.1.0 to 2.1.1)
20 35dd762d Michael Hanselmann
should not need manual intervention. As a safety measure, minor releases (e.g.
21 35dd762d Michael Hanselmann
2.1.3 to 2.2.0) require the ``cfgupgrade`` command for changing the
22 35dd762d Michael Hanselmann
configuration version. Below you find the steps necessary to upgrade between
23 35dd762d Michael Hanselmann
minor releases.
24 35dd762d Michael Hanselmann
25 35dd762d Michael Hanselmann
To run commands on all nodes, the `distributed shell (dsh)
26 35dd762d Michael Hanselmann
<http://www.netfort.gr.jp/~dancer/software/dsh.html.en>`_ can be used, e.g.
27 35dd762d Michael Hanselmann
``dsh -M -F 8 -f /var/lib/ganeti/ssconf_online_nodes gnt-cluster --version``.
28 35dd762d Michael Hanselmann
29 35dd762d Michael Hanselmann
#. Ensure no jobs are running (master node only)::
30 35dd762d Michael Hanselmann
31 e1ff0de1 Iustin Pop
    $ gnt-job list
32 35dd762d Michael Hanselmann
33 35dd762d Michael Hanselmann
#. Stop all daemons on all nodes::
34 35dd762d Michael Hanselmann
35 e1ff0de1 Iustin Pop
    $ /etc/init.d/ganeti stop
36 35dd762d Michael Hanselmann
37 35dd762d Michael Hanselmann
#. Backup old configuration (master node only)::
38 35dd762d Michael Hanselmann
39 e1ff0de1 Iustin Pop
    $ tar czf /var/lib/ganeti-$(date +\%FT\%T).tar.gz -C /var/lib ganeti
40 35dd762d Michael Hanselmann
41 35dd762d Michael Hanselmann
#. Install new Ganeti version on all nodes
42 35dd762d Michael Hanselmann
#. Run cfgupgrade on the master node::
43 35dd762d Michael Hanselmann
44 e1ff0de1 Iustin Pop
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose --dry-run
45 e1ff0de1 Iustin Pop
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose
46 35dd762d Michael Hanselmann
47 35dd762d Michael Hanselmann
   (``cfgupgrade`` supports a number of parameters, run it with
48 35dd762d Michael Hanselmann
   ``--help`` for more information)
49 35dd762d Michael Hanselmann
50 35dd762d Michael Hanselmann
#. Restart daemons on all nodes::
51 35dd762d Michael Hanselmann
52 e1ff0de1 Iustin Pop
    $ /etc/init.d/ganeti restart
53 35dd762d Michael Hanselmann
54 35dd762d Michael Hanselmann
#. Re-distribute configuration (master node only)::
55 35dd762d Michael Hanselmann
56 e1ff0de1 Iustin Pop
    $ gnt-cluster redist-conf
57 35dd762d Michael Hanselmann
58 35dd762d Michael Hanselmann
#. Restart daemons again on all nodes::
59 35dd762d Michael Hanselmann
60 e1ff0de1 Iustin Pop
    $ /etc/init.d/ganeti restart
61 35dd762d Michael Hanselmann
62 35dd762d Michael Hanselmann
#. Verify cluster (master node only)::
63 35dd762d Michael Hanselmann
64 e1ff0de1 Iustin Pop
    $ gnt-cluster verify
65 35dd762d Michael Hanselmann
66 35dd762d Michael Hanselmann
67 35dd762d Michael Hanselmann
2.0 releases
68 35dd762d Michael Hanselmann
------------
69 35dd762d Michael Hanselmann
70 35dd762d Michael Hanselmann
2.0.3 to 2.0.4
71 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
72 35dd762d Michael Hanselmann
73 35dd762d Michael Hanselmann
No changes needed except restarting the daemon; but rollback to 2.0.3 might
74 35dd762d Michael Hanselmann
require configuration editing.
75 35dd762d Michael Hanselmann
76 35dd762d Michael Hanselmann
If you're using Xen-HVM instances, please double-check the network
77 35dd762d Michael Hanselmann
configuration (``nic_type`` parameter) as the defaults might have changed:
78 35dd762d Michael Hanselmann
2.0.4 adds any missing configuration items and depending on the version of the
79 35dd762d Michael Hanselmann
software the cluster has been installed with, some new keys might have been
80 35dd762d Michael Hanselmann
added.
81 35dd762d Michael Hanselmann
82 35dd762d Michael Hanselmann
2.0.1 to 2.0.2/2.0.3
83 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~
84 35dd762d Michael Hanselmann
85 35dd762d Michael Hanselmann
Between 2.0.1 and 2.0.2 there have been some changes in the handling of block
86 35dd762d Michael Hanselmann
devices, which can cause some issues. 2.0.3 was then released which adds two
87 35dd762d Michael Hanselmann
new options/commands to fix this issue.
88 35dd762d Michael Hanselmann
89 35dd762d Michael Hanselmann
If you use DRBD-type instances and see problems in instance start or
90 35dd762d Michael Hanselmann
activate-disks with messages from DRBD about "lower device too small" or
91 35dd762d Michael Hanselmann
similar, it is recoomended to:
92 35dd762d Michael Hanselmann
93 35dd762d Michael Hanselmann
#. Run ``gnt-instance activate-disks --ignore-size $instance`` for each
94 35dd762d Michael Hanselmann
   of the affected instances
95 35dd762d Michael Hanselmann
#. Then run ``gnt-cluster repair-disk-sizes`` which will check that
96 35dd762d Michael Hanselmann
   instances have the correct disk sizes
97 35dd762d Michael Hanselmann
98 35dd762d Michael Hanselmann
1.2 to 2.0
99 35dd762d Michael Hanselmann
----------
100 35dd762d Michael Hanselmann
101 35dd762d Michael Hanselmann
Prerequisites:
102 35dd762d Michael Hanselmann
103 35dd762d Michael Hanselmann
- Ganeti 1.2.7 is currently installed
104 35dd762d Michael Hanselmann
- All instances have been migrated from DRBD 0.7 to DRBD 8.x (i.e. no
105 35dd762d Michael Hanselmann
  ``remote_raid1`` disk template)
106 35dd762d Michael Hanselmann
- Upgrade to Ganeti 2.0.0~rc2 or later (~rc1 and earlier don't have the needed
107 35dd762d Michael Hanselmann
  upgrade tool)
108 35dd762d Michael Hanselmann
109 35dd762d Michael Hanselmann
In the below steps, replace :file:`/var/lib` with ``$libdir`` if Ganeti was not
110 35dd762d Michael Hanselmann
installed with this prefix (e.g. :file:`/usr/local/var`). Same for
111 35dd762d Michael Hanselmann
:file:`/usr/lib`.
112 35dd762d Michael Hanselmann
113 35dd762d Michael Hanselmann
Execution (all steps are required in the order given):
114 35dd762d Michael Hanselmann
115 35dd762d Michael Hanselmann
#. Make a backup of the current configuration, for safety::
116 35dd762d Michael Hanselmann
117 e1ff0de1 Iustin Pop
    $ cp -a /var/lib/ganeti /var/lib/ganeti-1.2.backup
118 35dd762d Michael Hanselmann
119 35dd762d Michael Hanselmann
#. Stop all instances::
120 35dd762d Michael Hanselmann
121 e1ff0de1 Iustin Pop
    $ gnt-instance stop --all
122 35dd762d Michael Hanselmann
123 35dd762d Michael Hanselmann
#. Make sure no DRBD device are in use, the following command should show no
124 35dd762d Michael Hanselmann
   active minors::
125 35dd762d Michael Hanselmann
126 e1ff0de1 Iustin Pop
    $ gnt-cluster command grep cs: /proc/drbd | grep -v cs:Unconf
127 35dd762d Michael Hanselmann
128 35dd762d Michael Hanselmann
#. Stop the node daemons and rapi daemon on all nodes (note: should be logged
129 35dd762d Michael Hanselmann
   in not via the cluster name, but the master node name, as the command below
130 35dd762d Michael Hanselmann
   will remove the cluster ip from the master node)::
131 35dd762d Michael Hanselmann
132 e1ff0de1 Iustin Pop
    $ gnt-cluster command /etc/init.d/ganeti stop
133 35dd762d Michael Hanselmann
134 35dd762d Michael Hanselmann
#. Install the new software on all nodes, either from packaging (if available)
135 35dd762d Michael Hanselmann
   or from sources; the master daemon will not start but give error messages
136 35dd762d Michael Hanselmann
   about wrong configuration file, which is normal
137 35dd762d Michael Hanselmann
#. Upgrade the configuration file::
138 35dd762d Michael Hanselmann
139 e1ff0de1 Iustin Pop
    $ /usr/lib/ganeti/tools/cfgupgrade12 -v --dry-run
140 e1ff0de1 Iustin Pop
    $ /usr/lib/ganeti/tools/cfgupgrade12 -v
141 35dd762d Michael Hanselmann
142 35dd762d Michael Hanselmann
#. Make sure ``ganeti-noded`` is running on all nodes (and start it if
143 35dd762d Michael Hanselmann
   not)
144 35dd762d Michael Hanselmann
#. Start the master daemon::
145 35dd762d Michael Hanselmann
146 e1ff0de1 Iustin Pop
    $ ganeti-masterd
147 35dd762d Michael Hanselmann
148 35dd762d Michael Hanselmann
#. Check that a simple node-list works::
149 35dd762d Michael Hanselmann
150 e1ff0de1 Iustin Pop
    $ gnt-node list
151 35dd762d Michael Hanselmann
152 35dd762d Michael Hanselmann
#. Redistribute updated configuration to all nodes::
153 35dd762d Michael Hanselmann
154 e1ff0de1 Iustin Pop
    $ gnt-cluster redist-conf
155 e1ff0de1 Iustin Pop
    $ gnt-cluster copyfile /var/lib/ganeti/known_hosts
156 35dd762d Michael Hanselmann
157 35dd762d Michael Hanselmann
#. Optional: if needed, install RAPI-specific certificates under
158 35dd762d Michael Hanselmann
   :file:`/var/lib/ganeti/rapi.pem` and run::
159 35dd762d Michael Hanselmann
160 e1ff0de1 Iustin Pop
    $ gnt-cluster copyfile /var/lib/ganeti/rapi.pem
161 35dd762d Michael Hanselmann
162 35dd762d Michael Hanselmann
#. Run a cluster verify, this should show no problems::
163 35dd762d Michael Hanselmann
164 e1ff0de1 Iustin Pop
    $ gnt-cluster verify
165 35dd762d Michael Hanselmann
166 35dd762d Michael Hanselmann
#. Remove some obsolete files::
167 35dd762d Michael Hanselmann
168 e1ff0de1 Iustin Pop
    $ gnt-cluster command rm /var/lib/ganeti/ssconf_node_pass
169 e1ff0de1 Iustin Pop
    $ gnt-cluster command rm /var/lib/ganeti/ssconf_hypervisor
170 35dd762d Michael Hanselmann
171 35dd762d Michael Hanselmann
#. Update the xen pvm (if this was a pvm cluster) setting for 1.2
172 35dd762d Michael Hanselmann
   compatibility::
173 35dd762d Michael Hanselmann
174 e1ff0de1 Iustin Pop
    $ gnt-cluster modify -H xen-pvm:root_path=/dev/sda
175 35dd762d Michael Hanselmann
176 35dd762d Michael Hanselmann
#. Depending on your setup, you might also want to reset the initrd parameter::
177 35dd762d Michael Hanselmann
178 e1ff0de1 Iustin Pop
    $ gnt-cluster modify -H xen-pvm:initrd_path=/boot/initrd-2.6-xenU
179 35dd762d Michael Hanselmann
180 35dd762d Michael Hanselmann
#. Reset the instance autobalance setting to default::
181 35dd762d Michael Hanselmann
182 e1ff0de1 Iustin Pop
    $ for i in $(gnt-instance list -o name --no-headers); do \
183 e1ff0de1 Iustin Pop
       gnt-instance modify -B auto_balance=default $i; \
184 e1ff0de1 Iustin Pop
      done
185 35dd762d Michael Hanselmann
186 35dd762d Michael Hanselmann
#. Optional: start the RAPI demon::
187 35dd762d Michael Hanselmann
188 e1ff0de1 Iustin Pop
    $ ganeti-rapi
189 35dd762d Michael Hanselmann
190 35dd762d Michael Hanselmann
#. Restart instances::
191 35dd762d Michael Hanselmann
192 e1ff0de1 Iustin Pop
    $ gnt-instance start --force-multiple --all
193 35dd762d Michael Hanselmann
194 35dd762d Michael Hanselmann
At this point, ``gnt-cluster verify`` should show no errors and the migration
195 35dd762d Michael Hanselmann
is complete.
196 35dd762d Michael Hanselmann
197 35dd762d Michael Hanselmann
1.2 releases
198 35dd762d Michael Hanselmann
------------
199 35dd762d Michael Hanselmann
200 35dd762d Michael Hanselmann
1.2.4 to any other higher 1.2 version
201 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
202 35dd762d Michael Hanselmann
203 35dd762d Michael Hanselmann
No changes needed. Rollback will usually require manual edit of the
204 35dd762d Michael Hanselmann
configuration file.
205 35dd762d Michael Hanselmann
206 35dd762d Michael Hanselmann
1.2.3 to 1.2.4
207 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
208 35dd762d Michael Hanselmann
209 35dd762d Michael Hanselmann
No changes needed. Note that going back from 1.2.4 to 1.2.3 will require manual
210 35dd762d Michael Hanselmann
edit of the configuration file (since we added some HVM-related new
211 35dd762d Michael Hanselmann
attributes).
212 35dd762d Michael Hanselmann
213 35dd762d Michael Hanselmann
1.2.2 to 1.2.3
214 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
215 35dd762d Michael Hanselmann
216 35dd762d Michael Hanselmann
No changes needed. Note that the drbd7-to-8 upgrade tool does a disk format
217 35dd762d Michael Hanselmann
change for the DRBD metadata, so in theory this might be **risky**. It is
218 35dd762d Michael Hanselmann
advised to have (good) backups before doing the upgrade.
219 35dd762d Michael Hanselmann
220 35dd762d Michael Hanselmann
1.2.1 to 1.2.2
221 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
222 35dd762d Michael Hanselmann
223 35dd762d Michael Hanselmann
No changes needed.
224 35dd762d Michael Hanselmann
225 35dd762d Michael Hanselmann
1.2.0 to 1.2.1
226 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
227 35dd762d Michael Hanselmann
228 35dd762d Michael Hanselmann
No changes needed. Only some bugfixes and new additions that don't affect
229 35dd762d Michael Hanselmann
existing clusters.
230 35dd762d Michael Hanselmann
231 35dd762d Michael Hanselmann
1.2.0 beta 3 to 1.2.0
232 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~~
233 35dd762d Michael Hanselmann
234 35dd762d Michael Hanselmann
No changes needed.
235 35dd762d Michael Hanselmann
236 35dd762d Michael Hanselmann
1.2.0 beta 2 to beta 3
237 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~~~
238 35dd762d Michael Hanselmann
239 35dd762d Michael Hanselmann
No changes needed. A new version of the debian-etch-instance OS (0.3) has been
240 35dd762d Michael Hanselmann
released, but upgrading it is not required.
241 35dd762d Michael Hanselmann
242 35dd762d Michael Hanselmann
1.2.0 beta 1 to beta 2
243 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~~~
244 35dd762d Michael Hanselmann
245 35dd762d Michael Hanselmann
Beta 2 switched the config file format to JSON. Steps to upgrade:
246 35dd762d Michael Hanselmann
247 35dd762d Michael Hanselmann
#. Stop the daemons (``/etc/init.d/ganeti stop``) on all nodes
248 35dd762d Michael Hanselmann
#. Disable the cron job (default is :file:`/etc/cron.d/ganeti`)
249 35dd762d Michael Hanselmann
#. Install the new version
250 35dd762d Michael Hanselmann
#. Make a backup copy of the config file
251 35dd762d Michael Hanselmann
#. Upgrade the config file using the following command::
252 35dd762d Michael Hanselmann
253 e1ff0de1 Iustin Pop
    $ /usr/share/ganeti/cfgupgrade --verbose /var/lib/ganeti/config.data
254 35dd762d Michael Hanselmann
255 35dd762d Michael Hanselmann
#. Start the daemons and run ``gnt-cluster info``, ``gnt-node list`` and
256 35dd762d Michael Hanselmann
   ``gnt-instance list`` to check if the upgrade process finished successfully
257 35dd762d Michael Hanselmann
258 35dd762d Michael Hanselmann
The OS definition also need to be upgraded. There is a new version of the
259 35dd762d Michael Hanselmann
debian-etch-instance OS (0.2) that goes along with beta 2.
260 9ff4f2c0 Michael Hanselmann
261 9ff4f2c0 Michael Hanselmann
.. vim: set textwidth=72 :
262 9ff4f2c0 Michael Hanselmann
.. Local Variables:
263 9ff4f2c0 Michael Hanselmann
.. mode: rst
264 9ff4f2c0 Michael Hanselmann
.. fill-column: 72
265 9ff4f2c0 Michael Hanselmann
.. End: