root / doc / hooks.rst @ 625ac113
History | View | Annotate | Download (16 kB)
1 |
Ganeti customisation using hooks |
---|---|
2 |
================================ |
3 |
|
4 |
Documents ganeti version 2.0 |
5 |
|
6 |
.. contents:: |
7 |
|
8 |
Introduction |
9 |
------------ |
10 |
|
11 |
|
12 |
In order to allow customisation of operations, ganeti runs scripts |
13 |
under ``/etc/ganeti/hooks`` based on certain rules. |
14 |
|
15 |
|
16 |
This is similar to the ``/etc/network/`` structure present in Debian |
17 |
for network interface handling. |
18 |
|
19 |
Organisation |
20 |
------------ |
21 |
|
22 |
For every operation, two sets of scripts are run: |
23 |
|
24 |
- pre phase (for authorization/checking) |
25 |
- post phase (for logging) |
26 |
|
27 |
Also, for each operation, the scripts are run on one or more nodes, |
28 |
depending on the operation type. |
29 |
|
30 |
Note that, even though we call them scripts, we are actually talking |
31 |
about any executable. |
32 |
|
33 |
*pre* scripts |
34 |
~~~~~~~~~~~~~ |
35 |
|
36 |
The *pre* scripts have a definite target: to check that the operation |
37 |
is allowed given the site-specific constraints. You could have, for |
38 |
example, a rule that says every new instance is required to exists in |
39 |
a database; to implement this, you could write a script that checks |
40 |
the new instance parameters against your database. |
41 |
|
42 |
The objective of these scripts should be their return code (zero or |
43 |
non-zero for success and failure). However, if they modify the |
44 |
environment in any way, they should be idempotent, as failed |
45 |
executions could be restarted and thus the script(s) run again with |
46 |
exactly the same parameters. |
47 |
|
48 |
Note that if a node is unreachable at the time a hooks is run, this |
49 |
will not be interpreted as a deny for the execution. In other words, |
50 |
only an actual error returned from a script will cause abort, and not |
51 |
an unreachable node. |
52 |
|
53 |
Therefore, if you want to guarantee that a hook script is run and |
54 |
denies an action, it's best to put it on the master node. |
55 |
|
56 |
*post* scripts |
57 |
~~~~~~~~~~~~~~ |
58 |
|
59 |
These scripts should do whatever you need as a reaction to the |
60 |
completion of an operation. Their return code is not checked (but |
61 |
logged), and they should not depend on the fact that the *pre* scripts |
62 |
have been run. |
63 |
|
64 |
Naming |
65 |
~~~~~~ |
66 |
|
67 |
The allowed names for the scripts consist of (similar to *run-parts* ) |
68 |
upper and lower case, digits, underscores and hyphens. In other words, |
69 |
the regexp ``^[a-zA-Z0-9_-]+$``. Also, non-executable scripts will be |
70 |
ignored. |
71 |
|
72 |
|
73 |
Order of execution |
74 |
~~~~~~~~~~~~~~~~~~ |
75 |
|
76 |
On a single node, the scripts in a directory are run in lexicographic |
77 |
order (more exactly, the python string comparison order). It is |
78 |
advisable to implement the usual *NN-name* convention where *NN* is a |
79 |
two digit number. |
80 |
|
81 |
For an operation whose hooks are run on multiple nodes, there is no |
82 |
specific ordering of nodes with regard to hooks execution; you should |
83 |
assume that the scripts are run in parallel on the target nodes |
84 |
(keeping on each node the above specified ordering). If you need any |
85 |
kind of inter-node synchronisation, you have to implement it yourself |
86 |
in the scripts. |
87 |
|
88 |
Execution environment |
89 |
~~~~~~~~~~~~~~~~~~~~~ |
90 |
|
91 |
The scripts will be run as follows: |
92 |
|
93 |
- no command line arguments |
94 |
|
95 |
- no controlling *tty* |
96 |
|
97 |
- stdin is actually */dev/null* |
98 |
|
99 |
- stdout and stderr are directed to files |
100 |
|
101 |
- PATH is reset to ``/sbin:/bin:/usr/sbin:/usr/bin`` |
102 |
|
103 |
- the environment is cleared, and only ganeti-specific variables will |
104 |
be left |
105 |
|
106 |
|
107 |
All information about the cluster is passed using environment |
108 |
variables. Different operations will have sligthly different |
109 |
environments, but most of the variables are common. |
110 |
|
111 |
Operation list |
112 |
-------------- |
113 |
|
114 |
Node operations |
115 |
~~~~~~~~~~~~~~~ |
116 |
|
117 |
OP_ADD_NODE |
118 |
+++++++++++ |
119 |
|
120 |
Adds a node to the cluster. |
121 |
|
122 |
:directory: node-add |
123 |
:env. vars: NODE_NAME, NODE_PIP, NODE_SIP |
124 |
:pre-execution: all existing nodes |
125 |
:post-execution: all nodes plus the new node |
126 |
|
127 |
|
128 |
OP_REMOVE_NODE |
129 |
++++++++++++++ |
130 |
|
131 |
Removes a node from the cluster. On the removed node the hooks are |
132 |
called during the execution of the operation and not after its |
133 |
completion. |
134 |
|
135 |
:directory: node-remove |
136 |
:env. vars: NODE_NAME |
137 |
:pre-execution: all existing nodes except the removed node |
138 |
:post-execution: all existing nodes |
139 |
|
140 |
OP_NODE_SET_PARAMS |
141 |
++++++++++++++++++ |
142 |
|
143 |
Changes a node's parameters. |
144 |
|
145 |
:directory: node-modify |
146 |
:env. vars: MASTER_CANDIDATE, OFFLINE, DRAINED |
147 |
:pre-execution: master node, the target node |
148 |
:post-execution: master node, the target node |
149 |
|
150 |
OP_NODE_EVACUATE |
151 |
++++++++++++++++ |
152 |
|
153 |
Relocate secondary instances from a node. |
154 |
|
155 |
:directory: node-evacuate |
156 |
:env. vars: NEW_SECONDARY, NODE_NAME |
157 |
:pre-execution: master node, target node |
158 |
:post-execution: master node, target node |
159 |
|
160 |
OP_NODE_MIGRATE |
161 |
++++++++++++++++ |
162 |
|
163 |
Relocate secondary instances from a node. |
164 |
|
165 |
:directory: node-migrate |
166 |
:env. vars: NODE_NAME |
167 |
:pre-execution: master node |
168 |
:post-execution: master node |
169 |
|
170 |
|
171 |
Instance operations |
172 |
~~~~~~~~~~~~~~~~~~~ |
173 |
|
174 |
All instance operations take at least the following variables: |
175 |
INSTANCE_NAME, INSTANCE_PRIMARY, INSTANCE_SECONDARIES, |
176 |
INSTANCE_OS_TYPE, INSTANCE_DISK_TEMPLATE, INSTANCE_MEMORY, |
177 |
INSTANCE_DISK_SIZES, INSTANCE_VCPUS, INSTANCE_NIC_COUNT, |
178 |
INSTANCE_NICn_IP, INSTANCE_NICn_BRIDGE, INSTANCE_NICn_MAC, |
179 |
INSTANCE_DISK_COUNT, INSTANCE_DISKn_SIZE, INSTANCE_DISKn_MODE. |
180 |
|
181 |
The INSTANCE_NICn_* and INSTANCE_DISKn_* variables represent the |
182 |
properties of the *n* -th NIC and disk, and are zero-indexed. |
183 |
|
184 |
|
185 |
OP_INSTANCE_ADD |
186 |
+++++++++++++++ |
187 |
|
188 |
Creates a new instance. |
189 |
|
190 |
:directory: instance-add |
191 |
:env. vars: ADD_MODE, SRC_NODE, SRC_PATH, SRC_IMAGES |
192 |
:pre-execution: master node, primary and secondary nodes |
193 |
:post-execution: master node, primary and secondary nodes |
194 |
|
195 |
OP_INSTANCE_REINSTALL |
196 |
+++++++++++++++++++++ |
197 |
|
198 |
Reinstalls an instance. |
199 |
|
200 |
:directory: instance-reinstall |
201 |
:env. vars: only the standard instance vars |
202 |
:pre-execution: master node, primary and secondary nodes |
203 |
:post-execution: master node, primary and secondary nodes |
204 |
|
205 |
OP_BACKUP_EXPORT |
206 |
++++++++++++++++ |
207 |
|
208 |
Exports the instance. |
209 |
|
210 |
:directory: instance-export |
211 |
:env. vars: EXPORT_NODE, EXPORT_DO_SHUTDOWN |
212 |
:pre-execution: master node, primary and secondary nodes |
213 |
:post-execution: master node, primary and secondary nodes |
214 |
|
215 |
OP_INSTANCE_START |
216 |
+++++++++++++++++ |
217 |
|
218 |
Starts an instance. |
219 |
|
220 |
:directory: instance-start |
221 |
:env. vars: FORCE |
222 |
:pre-execution: master node, primary and secondary nodes |
223 |
:post-execution: master node, primary and secondary nodes |
224 |
|
225 |
OP_INSTANCE_SHUTDOWN |
226 |
++++++++++++++++++++ |
227 |
|
228 |
Stops an instance. |
229 |
|
230 |
:directory: instance-stop |
231 |
:env. vars: only the standard instance vars |
232 |
:pre-execution: master node, primary and secondary nodes |
233 |
:post-execution: master node, primary and secondary nodes |
234 |
|
235 |
OP_INSTANCE_REBOOT |
236 |
++++++++++++++++++ |
237 |
|
238 |
Reboots an instance. |
239 |
|
240 |
:directory: instance-reboot |
241 |
:env. vars: IGNORE_SECONDARIES, REBOOT_TYPE |
242 |
:pre-execution: master node, primary and secondary nodes |
243 |
:post-execution: master node, primary and secondary nodes |
244 |
|
245 |
OP_INSTANCE_MODIFY |
246 |
++++++++++++++++++ |
247 |
|
248 |
Modifies the instance parameters. |
249 |
|
250 |
:directory: instance-modify |
251 |
:env. vars: only the standard instance vars |
252 |
:pre-execution: master node, primary and secondary nodes |
253 |
:post-execution: master node, primary and secondary nodes |
254 |
|
255 |
OP_INSTANCE_FAILOVER |
256 |
++++++++++++++++++++ |
257 |
|
258 |
Failovers an instance. In the post phase INSTANCE_PRIMARY and |
259 |
INSTANCE_SECONDARIES refer to the nodes that were repectively primary |
260 |
and secondary before failover. |
261 |
|
262 |
:directory: instance-failover |
263 |
:env. vars: IGNORE_CONSISTENCY, OLD_SECONDARY, OLD_PRIMARY, NEW_SECONDARY, NEW_PRIMARY |
264 |
:pre-execution: master node, secondary node |
265 |
:post-execution: master node, primary and secondary nodes |
266 |
|
267 |
OP_INSTANCE_MIGRATE |
268 |
++++++++++++++++++++ |
269 |
|
270 |
Migrates an instance. In the post phase INSTANCE_PRIMARY and |
271 |
INSTANCE_SECONDARIES refer to the nodes that were repectively primary |
272 |
and secondary before migration. |
273 |
|
274 |
:directory: instance-migrate |
275 |
:env. vars: MIGRATE_LIVE, MIGRATE_CLEANUP, OLD_SECONDARY, OLD_PRIMARY, NEW_SECONDARY, NEW_PRIMARY |
276 |
:pre-execution: master node, secondary node |
277 |
:post-execution: master node, primary and secondary nodes |
278 |
|
279 |
|
280 |
OP_INSTANCE_REMOVE |
281 |
++++++++++++++++++ |
282 |
|
283 |
Remove an instance. |
284 |
|
285 |
:directory: instance-remove |
286 |
:env. vars: only the standard instance vars |
287 |
:pre-execution: master node |
288 |
:post-execution: master node, primary and secondary nodes |
289 |
|
290 |
OP_INSTANCE_REPLACE_DISKS |
291 |
+++++++++++++++++++++++++ |
292 |
|
293 |
Replace an instance's disks. |
294 |
|
295 |
:directory: mirror-replace |
296 |
:env. vars: MODE, NEW_SECONDARY, OLD_SECONDARY |
297 |
:pre-execution: master node, primary and secondary nodes |
298 |
:post-execution: master node, primary and secondary nodes |
299 |
|
300 |
OP_INSTANCE_GROW_DISK |
301 |
+++++++++++++++++++++ |
302 |
|
303 |
Grows the disk of an instance. |
304 |
|
305 |
:directory: disk-grow |
306 |
:env. vars: DISK, AMOUNT |
307 |
:pre-execution: master node, primary and secondary nodes |
308 |
:post-execution: master node, primary and secondary nodes |
309 |
|
310 |
OP_INSTANCE_RENAME |
311 |
++++++++++++++++++ |
312 |
|
313 |
Renames an instance. |
314 |
|
315 |
:directory: instance-rename |
316 |
:env. vars: INSTANCE_NEW_NAME |
317 |
:pre-execution: master node, primary and secondary nodes |
318 |
:post-execution: master node, primary and secondary nodes |
319 |
|
320 |
OP_INSTANCE_MOVE |
321 |
++++++++++++++++ |
322 |
|
323 |
Move an instance by data-copying. |
324 |
|
325 |
:directory: instance-move |
326 |
:env. vars: TARGET_NODE |
327 |
:pre-execution: master node, primary and target nodes |
328 |
:post-execution: master node, primary and target nodes |
329 |
|
330 |
OP_INSTANCE_RECREATE_DISKS |
331 |
++++++++++++++++++++++++++ |
332 |
|
333 |
Recreate an instance's missing disks. |
334 |
|
335 |
:directory: instance-recreate-disks |
336 |
:env. vars: only the standard instance vars |
337 |
:pre-execution: master node, primary and secondary nodes |
338 |
:post-execution: master node, primary and secondary nodes |
339 |
|
340 |
OP_INSTANCE_REPLACE_DISKS |
341 |
+++++++++++++++++++++++++ |
342 |
|
343 |
Replace the disks of an instance. |
344 |
|
345 |
:directory: mirrors-replace |
346 |
:env. vars: MODE, NEW_SECONDARY, OLD_SECONDARY |
347 |
:pre-execution: master node, primary and new secondary nodes |
348 |
:post-execution: master node, primary and new secondary nodes |
349 |
|
350 |
|
351 |
Cluster operations |
352 |
~~~~~~~~~~~~~~~~~~ |
353 |
|
354 |
OP_POST_INIT_CLUSTER |
355 |
++++++++++++++++++++ |
356 |
|
357 |
This hook is called via a special "empty" LU right after cluster |
358 |
initialization. |
359 |
|
360 |
:directory: cluster-init |
361 |
:env. vars: none |
362 |
:pre-execution: none |
363 |
:post-execution: master node |
364 |
|
365 |
OP_DESTROY_CLUSTER |
366 |
++++++++++++++++++ |
367 |
|
368 |
The post phase of this hook is called during the execution of destroy |
369 |
operation and not after its completion. |
370 |
|
371 |
:directory: cluster-destroy |
372 |
:env. vars: none |
373 |
:pre-execution: none |
374 |
:post-execution: master node |
375 |
|
376 |
OP_CLUSTER_VERIFY |
377 |
+++++++++++++++++ |
378 |
|
379 |
Verifies the cluster status. This is a special LU with regard to |
380 |
hooks, as the result of the opcode will be combined with the result of |
381 |
post-execution hooks, in order to allow administrators to enhance the |
382 |
cluster verification procedure. |
383 |
|
384 |
:directory: cluster-verify |
385 |
:env. vars: CLUSTER, MASTER, CLUSTER_TAGS, NODE_TAGS_<name> |
386 |
:pre-execution: none |
387 |
:post-execution: all nodes |
388 |
|
389 |
OP_CLUSTER_RENAME |
390 |
+++++++++++++++++ |
391 |
|
392 |
Renames the cluster. |
393 |
|
394 |
:directory: cluster-rename |
395 |
:env. vars: NEW_NAME |
396 |
:pre-execution: master-node |
397 |
:post-execution: master-node |
398 |
|
399 |
OP_CLUSTER_SET_PARAMS |
400 |
+++++++++++++++++++++ |
401 |
|
402 |
Modifies the cluster parameters. |
403 |
|
404 |
:directory: cluster-modify |
405 |
:env. vars: NEW_VG_NAME |
406 |
:pre-execution: master node |
407 |
:post-execution: master node |
408 |
|
409 |
|
410 |
Obsolete operations |
411 |
~~~~~~~~~~~~~~~~~~~ |
412 |
|
413 |
The following operations are no longer present or don't execute hooks |
414 |
anymore in Ganeti 2.0: |
415 |
|
416 |
- OP_INIT_CLUSTER |
417 |
- OP_MASTER_FAILOVER |
418 |
- OP_INSTANCE_ADD_MDDRBD |
419 |
- OP_INSTANCE_REMOVE_MDDRBD |
420 |
|
421 |
|
422 |
Environment variables |
423 |
--------------------- |
424 |
|
425 |
Note that all variables listed here are actually prefixed with |
426 |
*GANETI_* in order to provide a clear namespace. |
427 |
|
428 |
Common variables |
429 |
~~~~~~~~~~~~~~~~ |
430 |
|
431 |
This is the list of environment variables supported by all operations: |
432 |
|
433 |
HOOKS_VERSION |
434 |
Documents the hooks interface version. In case this doesnt match |
435 |
what the script expects, it should not run. The documents conforms |
436 |
to the version 2. |
437 |
|
438 |
HOOKS_PHASE |
439 |
One of *PRE* or *POST* denoting which phase are we in. |
440 |
|
441 |
CLUSTER |
442 |
The cluster name. |
443 |
|
444 |
MASTER |
445 |
The master node. |
446 |
|
447 |
OP_CODE |
448 |
One of the *OP_* values from the list of operations. |
449 |
|
450 |
OBJECT_TYPE |
451 |
One of ``INSTANCE``, ``NODE``, ``CLUSTER``. |
452 |
|
453 |
DATA_DIR |
454 |
The path to the Ganeti configuration directory (to read, for |
455 |
example, the *ssconf* files). |
456 |
|
457 |
|
458 |
Specialised variables |
459 |
~~~~~~~~~~~~~~~~~~~~~ |
460 |
|
461 |
This is the list of variables which are specific to one or more |
462 |
operations. |
463 |
|
464 |
INSTANCE_NAME |
465 |
The name of the instance which is the target of the operation. |
466 |
|
467 |
INSTANCE_DISK_TEMPLATE |
468 |
The disk type for the instance. |
469 |
|
470 |
INSTANCE_DISK_COUNT |
471 |
The number of disks for the instance. |
472 |
|
473 |
INSTANCE_DISKn_SIZE |
474 |
The size of disk *n* for the instance. |
475 |
|
476 |
INSTANCE_DISKn_MODE |
477 |
Either *rw* for a read-write disk or *ro* for a read-only one. |
478 |
|
479 |
INSTANCE_NIC_COUNT |
480 |
The number of NICs for the instance. |
481 |
|
482 |
INSTANCE_NICn_BRIDGE |
483 |
The bridge to which the *n* -th NIC of the instance is attached. |
484 |
|
485 |
INSTANCE_NICn_IP |
486 |
The IP (if any) of the *n* -th NIC of the instance. |
487 |
|
488 |
INSTANCE_NICn_MAC |
489 |
The MAC address of the *n* -th NIC of the instance. |
490 |
|
491 |
INSTANCE_OS_TYPE |
492 |
The name of the instance OS. |
493 |
|
494 |
INSTANCE_PRIMARY |
495 |
The name of the node which is the primary for the instance. Note that |
496 |
for migrations/failovers, you shouldn't rely on this variable since |
497 |
the nodes change during the exectution, but on the |
498 |
OLD_PRIMARY/NEW_PRIMARY values. |
499 |
|
500 |
INSTANCE_SECONDARIES |
501 |
Space-separated list of secondary nodes for the instance. Note that |
502 |
for migrations/failovers, you shouldn't rely on this variable since |
503 |
the nodes change during the exectution, but on the |
504 |
OLD_SECONDARY/NEW_SECONDARY values. |
505 |
|
506 |
INSTANCE_MEMORY |
507 |
The memory size (in MiBs) of the instance. |
508 |
|
509 |
INSTANCE_VCPUS |
510 |
The number of virtual CPUs for the instance. |
511 |
|
512 |
INSTANCE_STATUS |
513 |
The run status of the instance. |
514 |
|
515 |
NODE_NAME |
516 |
The target node of this operation (not the node on which the hook |
517 |
runs). |
518 |
|
519 |
NODE_PIP |
520 |
The primary IP of the target node (the one over which inter-node |
521 |
communication is done). |
522 |
|
523 |
NODE_SIP |
524 |
The secondary IP of the target node (the one over which drbd |
525 |
replication is done). This can be equal to the primary ip, in case |
526 |
the cluster is not dual-homed. |
527 |
|
528 |
FORCE |
529 |
This is provided by some operations when the user gave this flag. |
530 |
|
531 |
IGNORE_CONSISTENCY |
532 |
The user has specified this flag. It is used when failing over |
533 |
instances in case the primary node is down. |
534 |
|
535 |
ADD_MODE |
536 |
The mode of the instance create: either *create* for create from |
537 |
scratch or *import* for restoring from an exported image. |
538 |
|
539 |
SRC_NODE, SRC_PATH, SRC_IMAGE |
540 |
In case the instance has been added by import, these variables are |
541 |
defined and point to the source node, source path (the directory |
542 |
containing the image and the config file) and the source disk image |
543 |
file. |
544 |
|
545 |
NEW_SECONDARY |
546 |
The name of the node on which the new mirror component is being |
547 |
added (for replace disk). This can be the name of the current |
548 |
secondary, if the new mirror is on the same secondary. For |
549 |
migrations/failovers, this is the old primary node. |
550 |
|
551 |
OLD_SECONDARY |
552 |
The name of the old secondary in the replace-disks command. Note that |
553 |
this can be equal to the new secondary if the secondary node hasn't |
554 |
actually changed. For migrations/failovers, this is the new primary |
555 |
node. |
556 |
|
557 |
OLD_PRIMARY, NEW_PRIMARY |
558 |
For migrations/failovers, the old and respectively new primary |
559 |
nodes. These two mirror the NEW_SECONDARY/OLD_SECONDARY variables |
560 |
|
561 |
EXPORT_NODE |
562 |
The node on which the exported image of the instance was done. |
563 |
|
564 |
EXPORT_DO_SHUTDOWN |
565 |
This variable tells if the instance has been shutdown or not while |
566 |
doing the export. In the "was shutdown" case, it's likely that the |
567 |
filesystem is consistent, whereas in the "did not shutdown" case, |
568 |
the filesystem would need a check (journal replay or full fsck) in |
569 |
order to guarantee consistency. |
570 |
|
571 |
CLUSTER_TAGS |
572 |
The list of cluster tags, space separated. |
573 |
|
574 |
NODE_TAGS_<name> |
575 |
The list of tags for node *<name>*, space separated. |
576 |
|
577 |
Examples |
578 |
-------- |
579 |
|
580 |
The startup of an instance will pass this environment to the hook |
581 |
script:: |
582 |
|
583 |
GANETI_CLUSTER=cluster1.example.com |
584 |
GANETI_DATA_DIR=/var/lib/ganeti |
585 |
GANETI_FORCE=False |
586 |
GANETI_HOOKS_PATH=instance-start |
587 |
GANETI_HOOKS_PHASE=post |
588 |
GANETI_HOOKS_VERSION=2 |
589 |
GANETI_INSTANCE_DISK0_MODE=rw |
590 |
GANETI_INSTANCE_DISK0_SIZE=128 |
591 |
GANETI_INSTANCE_DISK_COUNT=1 |
592 |
GANETI_INSTANCE_DISK_TEMPLATE=drbd |
593 |
GANETI_INSTANCE_MEMORY=128 |
594 |
GANETI_INSTANCE_NAME=instance2.example.com |
595 |
GANETI_INSTANCE_NIC0_BRIDGE=xen-br0 |
596 |
GANETI_INSTANCE_NIC0_IP= |
597 |
GANETI_INSTANCE_NIC0_MAC=aa:00:00:a5:91:58 |
598 |
GANETI_INSTANCE_NIC_COUNT=1 |
599 |
GANETI_INSTANCE_OS_TYPE=debootstrap |
600 |
GANETI_INSTANCE_PRIMARY=node3.example.com |
601 |
GANETI_INSTANCE_SECONDARIES=node5.example.com |
602 |
GANETI_INSTANCE_STATUS=down |
603 |
GANETI_INSTANCE_VCPUS=1 |
604 |
GANETI_MASTER=node1.example.com |
605 |
GANETI_OBJECT_TYPE=INSTANCE |
606 |
GANETI_OP_CODE=OP_INSTANCE_STARTUP |
607 |
GANETI_OP_TARGET=instance2.example.com |
608 |
|
609 |
.. vim: set textwidth=72 : |
610 |
.. Local Variables: |
611 |
.. mode: rst |
612 |
.. fill-column: 72 |
613 |
.. End: |