Statistics
| Branch: | Tag: | Revision:

root / UPGRADE @ db1ad1d5

History | View | Annotate | Download (9.7 kB)

1 35dd762d Michael Hanselmann
Upgrade notes
2 35dd762d Michael Hanselmann
=============
3 35dd762d Michael Hanselmann
4 e1ff0de1 Iustin Pop
.. highlight:: shell-example
5 35dd762d Michael Hanselmann
6 35dd762d Michael Hanselmann
This document details the steps needed to upgrade a cluster to newer versions
7 35dd762d Michael Hanselmann
of Ganeti.
8 35dd762d Michael Hanselmann
9 35dd762d Michael Hanselmann
As a general rule the node daemons need to be restarted after each software
10 35dd762d Michael Hanselmann
upgrade; if using the provided example init.d script, this means running the
11 35dd762d Michael Hanselmann
following command on all nodes::
12 35dd762d Michael Hanselmann
13 e1ff0de1 Iustin Pop
    $ /etc/init.d/ganeti restart
14 35dd762d Michael Hanselmann
15 35dd762d Michael Hanselmann
16 35dd762d Michael Hanselmann
2.1 and above
17 35dd762d Michael Hanselmann
-------------
18 35dd762d Michael Hanselmann
19 35dd762d Michael Hanselmann
Starting with Ganeti 2.0, upgrades between revisions (e.g. 2.1.0 to 2.1.1)
20 35dd762d Michael Hanselmann
should not need manual intervention. As a safety measure, minor releases (e.g.
21 35dd762d Michael Hanselmann
2.1.3 to 2.2.0) require the ``cfgupgrade`` command for changing the
22 35dd762d Michael Hanselmann
configuration version. Below you find the steps necessary to upgrade between
23 35dd762d Michael Hanselmann
minor releases.
24 35dd762d Michael Hanselmann
25 35dd762d Michael Hanselmann
To run commands on all nodes, the `distributed shell (dsh)
26 35dd762d Michael Hanselmann
<http://www.netfort.gr.jp/~dancer/software/dsh.html.en>`_ can be used, e.g.
27 35dd762d Michael Hanselmann
``dsh -M -F 8 -f /var/lib/ganeti/ssconf_online_nodes gnt-cluster --version``.
28 35dd762d Michael Hanselmann
29 35dd762d Michael Hanselmann
#. Ensure no jobs are running (master node only)::
30 35dd762d Michael Hanselmann
31 e1ff0de1 Iustin Pop
    $ gnt-job list
32 35dd762d Michael Hanselmann
33 6c28c480 Bernardo Dal Seno
#. Pause the watcher for an hour (master node only)::
34 6c28c480 Bernardo Dal Seno
35 6c28c480 Bernardo Dal Seno
    $ gnt-cluster watcher pause 1h
36 6c28c480 Bernardo Dal Seno
37 35dd762d Michael Hanselmann
#. Stop all daemons on all nodes::
38 35dd762d Michael Hanselmann
39 e1ff0de1 Iustin Pop
    $ /etc/init.d/ganeti stop
40 35dd762d Michael Hanselmann
41 35dd762d Michael Hanselmann
#. Backup old configuration (master node only)::
42 35dd762d Michael Hanselmann
43 e1ff0de1 Iustin Pop
    $ tar czf /var/lib/ganeti-$(date +\%FT\%T).tar.gz -C /var/lib ganeti
44 35dd762d Michael Hanselmann
45 35dd762d Michael Hanselmann
#. Install new Ganeti version on all nodes
46 35dd762d Michael Hanselmann
#. Run cfgupgrade on the master node::
47 35dd762d Michael Hanselmann
48 e1ff0de1 Iustin Pop
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose --dry-run
49 e1ff0de1 Iustin Pop
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose
50 35dd762d Michael Hanselmann
51 35dd762d Michael Hanselmann
   (``cfgupgrade`` supports a number of parameters, run it with
52 35dd762d Michael Hanselmann
   ``--help`` for more information)
53 35dd762d Michael Hanselmann
54 fe05a931 Michele Tartara
#. Upgrade the directory permissions on all nodes::
55 fe05a931 Michele Tartara
56 fe05a931 Michele Tartara
    $ /usr/lib/ganeti/ensure-dirs --full-run
57 fe05a931 Michele Tartara
58 35dd762d Michael Hanselmann
#. Restart daemons on all nodes::
59 35dd762d Michael Hanselmann
60 e1ff0de1 Iustin Pop
    $ /etc/init.d/ganeti restart
61 35dd762d Michael Hanselmann
62 35dd762d Michael Hanselmann
#. Re-distribute configuration (master node only)::
63 91fb0d18 Bernardo Dal Seno
64 91fb0d18 Bernardo Dal Seno
    $ gnt-cluster redist-conf
65 91fb0d18 Bernardo Dal Seno
66 91fb0d18 Bernardo Dal Seno
#. Restart daemons again on all nodes::
67 91fb0d18 Bernardo Dal Seno
68 91fb0d18 Bernardo Dal Seno
    $ /etc/init.d/ganeti restart
69 91fb0d18 Bernardo Dal Seno
70 91fb0d18 Bernardo Dal Seno
#. Enable the watcher again (master node only)::
71 91fb0d18 Bernardo Dal Seno
72 91fb0d18 Bernardo Dal Seno
    $ gnt-cluster watcher continue
73 91fb0d18 Bernardo Dal Seno
74 91fb0d18 Bernardo Dal Seno
#. Verify cluster (master node only)::
75 91fb0d18 Bernardo Dal Seno
76 91fb0d18 Bernardo Dal Seno
    $ gnt-cluster verify
77 91fb0d18 Bernardo Dal Seno
78 91fb0d18 Bernardo Dal Seno
Reverting an upgrade
79 91fb0d18 Bernardo Dal Seno
~~~~~~~~~~~~~~~~~~~~
80 91fb0d18 Bernardo Dal Seno
81 91fb0d18 Bernardo Dal Seno
For going back between revisions (e.g. 2.1.1 to 2.1.0) no manual
82 91fb0d18 Bernardo Dal Seno
intervention is required, as for upgrades.
83 91fb0d18 Bernardo Dal Seno
84 91fb0d18 Bernardo Dal Seno
Starting from version 2.8, ``cfgupgrade`` supports ``--downgrade``
85 91fb0d18 Bernardo Dal Seno
option to bring the configuration back to the previous stable version.
86 91fb0d18 Bernardo Dal Seno
This is useful if you upgrade Ganeti and after some time you run into
87 91fb0d18 Bernardo Dal Seno
problems with the new version. You can downgrade the configuration
88 91fb0d18 Bernardo Dal Seno
without losing the changes made since the upgrade. Any feature not
89 91fb0d18 Bernardo Dal Seno
supported by the old version will be removed from the configuration, of
90 91fb0d18 Bernardo Dal Seno
course, but you get a warning about it. If there is any new feature and
91 91fb0d18 Bernardo Dal Seno
you haven't changed from its default value, you don't have to worry
92 91fb0d18 Bernardo Dal Seno
about it, as it will get the same value whenever you'll upgrade again.
93 91fb0d18 Bernardo Dal Seno
94 91fb0d18 Bernardo Dal Seno
The procedure is similar to upgrading, but please notice that you have to
95 91fb0d18 Bernardo Dal Seno
revert the configuration **before** installing the old version.
96 91fb0d18 Bernardo Dal Seno
97 91fb0d18 Bernardo Dal Seno
#. Ensure no jobs are running (master node only)::
98 91fb0d18 Bernardo Dal Seno
99 91fb0d18 Bernardo Dal Seno
    $ gnt-job list
100 91fb0d18 Bernardo Dal Seno
101 91fb0d18 Bernardo Dal Seno
#. Pause the watcher for an hour (master node only)::
102 91fb0d18 Bernardo Dal Seno
103 91fb0d18 Bernardo Dal Seno
    $ gnt-cluster watcher pause 1h
104 91fb0d18 Bernardo Dal Seno
105 91fb0d18 Bernardo Dal Seno
#. Stop all daemons on all nodes::
106 91fb0d18 Bernardo Dal Seno
107 91fb0d18 Bernardo Dal Seno
    $ /etc/init.d/ganeti stop
108 91fb0d18 Bernardo Dal Seno
109 91fb0d18 Bernardo Dal Seno
#. Backup old configuration (master node only)::
110 91fb0d18 Bernardo Dal Seno
111 91fb0d18 Bernardo Dal Seno
    $ tar czf /var/lib/ganeti-$(date +\%FT\%T).tar.gz -C /var/lib ganeti
112 91fb0d18 Bernardo Dal Seno
113 91fb0d18 Bernardo Dal Seno
#. Run cfgupgrade on the master node::
114 91fb0d18 Bernardo Dal Seno
115 91fb0d18 Bernardo Dal Seno
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose --downgrade --dry-run
116 91fb0d18 Bernardo Dal Seno
    $ /usr/lib/ganeti/tools/cfgupgrade --verbose --downgrade
117 91fb0d18 Bernardo Dal Seno
118 91fb0d18 Bernardo Dal Seno
   You may want to copy all the messages about features that have been
119 91fb0d18 Bernardo Dal Seno
   removed during the downgrade, in case you want to restore them when
120 91fb0d18 Bernardo Dal Seno
   upgrading again.
121 91fb0d18 Bernardo Dal Seno
122 91fb0d18 Bernardo Dal Seno
#. Install the old Ganeti version on all nodes
123 91fb0d18 Bernardo Dal Seno
#. Restart daemons on all nodes::
124 91fb0d18 Bernardo Dal Seno
125 91fb0d18 Bernardo Dal Seno
    $ /etc/init.d/ganeti restart
126 91fb0d18 Bernardo Dal Seno
127 91fb0d18 Bernardo Dal Seno
#. Re-distribute configuration (master node only)::
128 35dd762d Michael Hanselmann
129 e1ff0de1 Iustin Pop
    $ gnt-cluster redist-conf
130 35dd762d Michael Hanselmann
131 35dd762d Michael Hanselmann
#. Restart daemons again on all nodes::
132 35dd762d Michael Hanselmann
133 e1ff0de1 Iustin Pop
    $ /etc/init.d/ganeti restart
134 35dd762d Michael Hanselmann
135 6c28c480 Bernardo Dal Seno
#. Enable the watcher again (master node only)::
136 6c28c480 Bernardo Dal Seno
137 6c28c480 Bernardo Dal Seno
    $ gnt-cluster watcher continue
138 6c28c480 Bernardo Dal Seno
139 35dd762d Michael Hanselmann
#. Verify cluster (master node only)::
140 35dd762d Michael Hanselmann
141 e1ff0de1 Iustin Pop
    $ gnt-cluster verify
142 35dd762d Michael Hanselmann
143 35dd762d Michael Hanselmann
144 35dd762d Michael Hanselmann
2.0 releases
145 35dd762d Michael Hanselmann
------------
146 35dd762d Michael Hanselmann
147 35dd762d Michael Hanselmann
2.0.3 to 2.0.4
148 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
149 35dd762d Michael Hanselmann
150 35dd762d Michael Hanselmann
No changes needed except restarting the daemon; but rollback to 2.0.3 might
151 35dd762d Michael Hanselmann
require configuration editing.
152 35dd762d Michael Hanselmann
153 35dd762d Michael Hanselmann
If you're using Xen-HVM instances, please double-check the network
154 35dd762d Michael Hanselmann
configuration (``nic_type`` parameter) as the defaults might have changed:
155 35dd762d Michael Hanselmann
2.0.4 adds any missing configuration items and depending on the version of the
156 35dd762d Michael Hanselmann
software the cluster has been installed with, some new keys might have been
157 35dd762d Michael Hanselmann
added.
158 35dd762d Michael Hanselmann
159 35dd762d Michael Hanselmann
2.0.1 to 2.0.2/2.0.3
160 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~
161 35dd762d Michael Hanselmann
162 35dd762d Michael Hanselmann
Between 2.0.1 and 2.0.2 there have been some changes in the handling of block
163 35dd762d Michael Hanselmann
devices, which can cause some issues. 2.0.3 was then released which adds two
164 35dd762d Michael Hanselmann
new options/commands to fix this issue.
165 35dd762d Michael Hanselmann
166 35dd762d Michael Hanselmann
If you use DRBD-type instances and see problems in instance start or
167 35dd762d Michael Hanselmann
activate-disks with messages from DRBD about "lower device too small" or
168 35dd762d Michael Hanselmann
similar, it is recoomended to:
169 35dd762d Michael Hanselmann
170 35dd762d Michael Hanselmann
#. Run ``gnt-instance activate-disks --ignore-size $instance`` for each
171 35dd762d Michael Hanselmann
   of the affected instances
172 35dd762d Michael Hanselmann
#. Then run ``gnt-cluster repair-disk-sizes`` which will check that
173 35dd762d Michael Hanselmann
   instances have the correct disk sizes
174 35dd762d Michael Hanselmann
175 35dd762d Michael Hanselmann
1.2 to 2.0
176 35dd762d Michael Hanselmann
----------
177 35dd762d Michael Hanselmann
178 35dd762d Michael Hanselmann
Prerequisites:
179 35dd762d Michael Hanselmann
180 35dd762d Michael Hanselmann
- Ganeti 1.2.7 is currently installed
181 35dd762d Michael Hanselmann
- All instances have been migrated from DRBD 0.7 to DRBD 8.x (i.e. no
182 35dd762d Michael Hanselmann
  ``remote_raid1`` disk template)
183 35dd762d Michael Hanselmann
- Upgrade to Ganeti 2.0.0~rc2 or later (~rc1 and earlier don't have the needed
184 35dd762d Michael Hanselmann
  upgrade tool)
185 35dd762d Michael Hanselmann
186 35dd762d Michael Hanselmann
In the below steps, replace :file:`/var/lib` with ``$libdir`` if Ganeti was not
187 35dd762d Michael Hanselmann
installed with this prefix (e.g. :file:`/usr/local/var`). Same for
188 35dd762d Michael Hanselmann
:file:`/usr/lib`.
189 35dd762d Michael Hanselmann
190 35dd762d Michael Hanselmann
Execution (all steps are required in the order given):
191 35dd762d Michael Hanselmann
192 35dd762d Michael Hanselmann
#. Make a backup of the current configuration, for safety::
193 35dd762d Michael Hanselmann
194 e1ff0de1 Iustin Pop
    $ cp -a /var/lib/ganeti /var/lib/ganeti-1.2.backup
195 35dd762d Michael Hanselmann
196 35dd762d Michael Hanselmann
#. Stop all instances::
197 35dd762d Michael Hanselmann
198 e1ff0de1 Iustin Pop
    $ gnt-instance stop --all
199 35dd762d Michael Hanselmann
200 35dd762d Michael Hanselmann
#. Make sure no DRBD device are in use, the following command should show no
201 35dd762d Michael Hanselmann
   active minors::
202 35dd762d Michael Hanselmann
203 e1ff0de1 Iustin Pop
    $ gnt-cluster command grep cs: /proc/drbd | grep -v cs:Unconf
204 35dd762d Michael Hanselmann
205 35dd762d Michael Hanselmann
#. Stop the node daemons and rapi daemon on all nodes (note: should be logged
206 35dd762d Michael Hanselmann
   in not via the cluster name, but the master node name, as the command below
207 35dd762d Michael Hanselmann
   will remove the cluster ip from the master node)::
208 35dd762d Michael Hanselmann
209 e1ff0de1 Iustin Pop
    $ gnt-cluster command /etc/init.d/ganeti stop
210 35dd762d Michael Hanselmann
211 35dd762d Michael Hanselmann
#. Install the new software on all nodes, either from packaging (if available)
212 35dd762d Michael Hanselmann
   or from sources; the master daemon will not start but give error messages
213 35dd762d Michael Hanselmann
   about wrong configuration file, which is normal
214 35dd762d Michael Hanselmann
#. Upgrade the configuration file::
215 35dd762d Michael Hanselmann
216 e1ff0de1 Iustin Pop
    $ /usr/lib/ganeti/tools/cfgupgrade12 -v --dry-run
217 e1ff0de1 Iustin Pop
    $ /usr/lib/ganeti/tools/cfgupgrade12 -v
218 35dd762d Michael Hanselmann
219 35dd762d Michael Hanselmann
#. Make sure ``ganeti-noded`` is running on all nodes (and start it if
220 35dd762d Michael Hanselmann
   not)
221 35dd762d Michael Hanselmann
#. Start the master daemon::
222 35dd762d Michael Hanselmann
223 e1ff0de1 Iustin Pop
    $ ganeti-masterd
224 35dd762d Michael Hanselmann
225 35dd762d Michael Hanselmann
#. Check that a simple node-list works::
226 35dd762d Michael Hanselmann
227 e1ff0de1 Iustin Pop
    $ gnt-node list
228 35dd762d Michael Hanselmann
229 35dd762d Michael Hanselmann
#. Redistribute updated configuration to all nodes::
230 35dd762d Michael Hanselmann
231 e1ff0de1 Iustin Pop
    $ gnt-cluster redist-conf
232 e1ff0de1 Iustin Pop
    $ gnt-cluster copyfile /var/lib/ganeti/known_hosts
233 35dd762d Michael Hanselmann
234 35dd762d Michael Hanselmann
#. Optional: if needed, install RAPI-specific certificates under
235 35dd762d Michael Hanselmann
   :file:`/var/lib/ganeti/rapi.pem` and run::
236 35dd762d Michael Hanselmann
237 e1ff0de1 Iustin Pop
    $ gnt-cluster copyfile /var/lib/ganeti/rapi.pem
238 35dd762d Michael Hanselmann
239 35dd762d Michael Hanselmann
#. Run a cluster verify, this should show no problems::
240 35dd762d Michael Hanselmann
241 e1ff0de1 Iustin Pop
    $ gnt-cluster verify
242 35dd762d Michael Hanselmann
243 35dd762d Michael Hanselmann
#. Remove some obsolete files::
244 35dd762d Michael Hanselmann
245 e1ff0de1 Iustin Pop
    $ gnt-cluster command rm /var/lib/ganeti/ssconf_node_pass
246 e1ff0de1 Iustin Pop
    $ gnt-cluster command rm /var/lib/ganeti/ssconf_hypervisor
247 35dd762d Michael Hanselmann
248 35dd762d Michael Hanselmann
#. Update the xen pvm (if this was a pvm cluster) setting for 1.2
249 35dd762d Michael Hanselmann
   compatibility::
250 35dd762d Michael Hanselmann
251 e1ff0de1 Iustin Pop
    $ gnt-cluster modify -H xen-pvm:root_path=/dev/sda
252 35dd762d Michael Hanselmann
253 35dd762d Michael Hanselmann
#. Depending on your setup, you might also want to reset the initrd parameter::
254 35dd762d Michael Hanselmann
255 e1ff0de1 Iustin Pop
    $ gnt-cluster modify -H xen-pvm:initrd_path=/boot/initrd-2.6-xenU
256 35dd762d Michael Hanselmann
257 35dd762d Michael Hanselmann
#. Reset the instance autobalance setting to default::
258 35dd762d Michael Hanselmann
259 e1ff0de1 Iustin Pop
    $ for i in $(gnt-instance list -o name --no-headers); do \
260 e1ff0de1 Iustin Pop
       gnt-instance modify -B auto_balance=default $i; \
261 e1ff0de1 Iustin Pop
      done
262 35dd762d Michael Hanselmann
263 35dd762d Michael Hanselmann
#. Optional: start the RAPI demon::
264 35dd762d Michael Hanselmann
265 e1ff0de1 Iustin Pop
    $ ganeti-rapi
266 35dd762d Michael Hanselmann
267 35dd762d Michael Hanselmann
#. Restart instances::
268 35dd762d Michael Hanselmann
269 e1ff0de1 Iustin Pop
    $ gnt-instance start --force-multiple --all
270 35dd762d Michael Hanselmann
271 35dd762d Michael Hanselmann
At this point, ``gnt-cluster verify`` should show no errors and the migration
272 35dd762d Michael Hanselmann
is complete.
273 35dd762d Michael Hanselmann
274 35dd762d Michael Hanselmann
1.2 releases
275 35dd762d Michael Hanselmann
------------
276 35dd762d Michael Hanselmann
277 35dd762d Michael Hanselmann
1.2.4 to any other higher 1.2 version
278 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
279 35dd762d Michael Hanselmann
280 35dd762d Michael Hanselmann
No changes needed. Rollback will usually require manual edit of the
281 35dd762d Michael Hanselmann
configuration file.
282 35dd762d Michael Hanselmann
283 35dd762d Michael Hanselmann
1.2.3 to 1.2.4
284 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
285 35dd762d Michael Hanselmann
286 35dd762d Michael Hanselmann
No changes needed. Note that going back from 1.2.4 to 1.2.3 will require manual
287 35dd762d Michael Hanselmann
edit of the configuration file (since we added some HVM-related new
288 35dd762d Michael Hanselmann
attributes).
289 35dd762d Michael Hanselmann
290 35dd762d Michael Hanselmann
1.2.2 to 1.2.3
291 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
292 35dd762d Michael Hanselmann
293 35dd762d Michael Hanselmann
No changes needed. Note that the drbd7-to-8 upgrade tool does a disk format
294 35dd762d Michael Hanselmann
change for the DRBD metadata, so in theory this might be **risky**. It is
295 35dd762d Michael Hanselmann
advised to have (good) backups before doing the upgrade.
296 35dd762d Michael Hanselmann
297 35dd762d Michael Hanselmann
1.2.1 to 1.2.2
298 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
299 35dd762d Michael Hanselmann
300 35dd762d Michael Hanselmann
No changes needed.
301 35dd762d Michael Hanselmann
302 35dd762d Michael Hanselmann
1.2.0 to 1.2.1
303 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~
304 35dd762d Michael Hanselmann
305 35dd762d Michael Hanselmann
No changes needed. Only some bugfixes and new additions that don't affect
306 35dd762d Michael Hanselmann
existing clusters.
307 35dd762d Michael Hanselmann
308 35dd762d Michael Hanselmann
1.2.0 beta 3 to 1.2.0
309 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~~
310 35dd762d Michael Hanselmann
311 35dd762d Michael Hanselmann
No changes needed.
312 35dd762d Michael Hanselmann
313 35dd762d Michael Hanselmann
1.2.0 beta 2 to beta 3
314 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~~~
315 35dd762d Michael Hanselmann
316 35dd762d Michael Hanselmann
No changes needed. A new version of the debian-etch-instance OS (0.3) has been
317 35dd762d Michael Hanselmann
released, but upgrading it is not required.
318 35dd762d Michael Hanselmann
319 35dd762d Michael Hanselmann
1.2.0 beta 1 to beta 2
320 35dd762d Michael Hanselmann
~~~~~~~~~~~~~~~~~~~~~~
321 35dd762d Michael Hanselmann
322 35dd762d Michael Hanselmann
Beta 2 switched the config file format to JSON. Steps to upgrade:
323 35dd762d Michael Hanselmann
324 35dd762d Michael Hanselmann
#. Stop the daemons (``/etc/init.d/ganeti stop``) on all nodes
325 35dd762d Michael Hanselmann
#. Disable the cron job (default is :file:`/etc/cron.d/ganeti`)
326 35dd762d Michael Hanselmann
#. Install the new version
327 35dd762d Michael Hanselmann
#. Make a backup copy of the config file
328 35dd762d Michael Hanselmann
#. Upgrade the config file using the following command::
329 35dd762d Michael Hanselmann
330 e1ff0de1 Iustin Pop
    $ /usr/share/ganeti/cfgupgrade --verbose /var/lib/ganeti/config.data
331 35dd762d Michael Hanselmann
332 35dd762d Michael Hanselmann
#. Start the daemons and run ``gnt-cluster info``, ``gnt-node list`` and
333 35dd762d Michael Hanselmann
   ``gnt-instance list`` to check if the upgrade process finished successfully
334 35dd762d Michael Hanselmann
335 35dd762d Michael Hanselmann
The OS definition also need to be upgraded. There is a new version of the
336 35dd762d Michael Hanselmann
debian-etch-instance OS (0.2) that goes along with beta 2.
337 9ff4f2c0 Michael Hanselmann
338 9ff4f2c0 Michael Hanselmann
.. vim: set textwidth=72 :
339 9ff4f2c0 Michael Hanselmann
.. Local Variables:
340 9ff4f2c0 Michael Hanselmann
.. mode: rst
341 9ff4f2c0 Michael Hanselmann
.. fill-column: 72
342 9ff4f2c0 Michael Hanselmann
.. End: