|
1 |
============================================================
|
|
2 |
Detection of user-initiated shutdown from inside an instance
|
|
3 |
============================================================
|
|
4 |
|
|
5 |
.. contents:: :depth: 2
|
|
6 |
|
|
7 |
This is a design document detailing the implementation of a way for Ganeti to
|
|
8 |
detect whether a machine marked as up but not running was shutdown gracefully
|
|
9 |
by the user from inside the machine itself.
|
|
10 |
|
|
11 |
Current state and shortcomings
|
|
12 |
==============================
|
|
13 |
|
|
14 |
Ganeti keeps track of the desired status of instances in order to be able to
|
|
15 |
take proper actions (e.g.: reboot) on the ones that happen to crash.
|
|
16 |
Currently, the only way to properly shut down a machine is through Ganeti's own
|
|
17 |
commands, that will mark an instance as ``ADMIN_down``.
|
|
18 |
If a user shuts down an instance from inside, through the proper command of the
|
|
19 |
operating system it is running, the instance will be shutdown gracefully, but
|
|
20 |
Ganeti is not aware of that: the desired status of the instance will still be
|
|
21 |
marked as ``running``, so when the watcher realises that the instance is down,
|
|
22 |
it will restart it. This behaviour is usually not what the user expects.
|
|
23 |
|
|
24 |
Proposed changes
|
|
25 |
================
|
|
26 |
|
|
27 |
We propose to modify Ganeti in such a way that it will detect when an instance
|
|
28 |
was shutdown because of an explicit user request. When such a situation is
|
|
29 |
detected, instead of presenting an error as it happens now, either the state
|
|
30 |
of the instance will be set to ADMIN_down, or the instance will be
|
|
31 |
automatically rebooted, depending on a instance-specific configuration value.
|
|
32 |
The default behavior in case no such parameter is found will be to follow
|
|
33 |
the apparent will of the user, and setting to ADMIN_down an instance that
|
|
34 |
was shut down correctly from inside.
|
|
35 |
|
|
36 |
This design document applies to the Xen backend of Ganeti, because it uses
|
|
37 |
features specific of such hypervisor. Initial analysis suggests that a similar
|
|
38 |
approach might be used for KVM as well, so this design document will be later
|
|
39 |
extended to add more details about it.
|
|
40 |
|
|
41 |
Implementation
|
|
42 |
==============
|
|
43 |
|
|
44 |
Xen knows why a domain is being shut down (a crash or an explicit shutdown
|
|
45 |
or poweroff request), but such information is not usually readily available
|
|
46 |
externally, because all such cases lead to the virtual machine being destroyed
|
|
47 |
immediately after the event is detected.
|
|
48 |
|
|
49 |
Still, Xen allows the instance configuration file to define what action to be
|
|
50 |
taken in all those cases through the ``on_poweroff``, ``on_shutdown`` and
|
|
51 |
``on_crash`` variables. By setting them to ``preserve``, Xen will avoid
|
|
52 |
destroying the domains automatically.
|
|
53 |
|
|
54 |
When the domain is not destroyed, it can be viewed by using ``xm list`` (or ``xl
|
|
55 |
list`` in newer Xen versions), and the ``State`` field of the output will
|
|
56 |
provide useful information.
|
|
57 |
|
|
58 |
If the state is ``----c-`` it means the instance has crashed.
|
|
59 |
|
|
60 |
If the state is ``---s--`` it means the instance was properly shutdown.
|
|
61 |
|
|
62 |
If the instance was properly shutdown and it is still marked as ``running`` by
|
|
63 |
Ganeti, it means that it was shutdown from inside by the user, and the ganeti
|
|
64 |
status of the instance needs to be changed to ``ADMIN_down``.
|
|
65 |
|
|
66 |
This will be done at regular intervals by the group watcher, just before
|
|
67 |
deciding which instances to reboot.
|
|
68 |
|
|
69 |
On top of that, at the same times, the watcher will also need to issue ``xm
|
|
70 |
destroy`` commands for all the domains that are in crashed or shutdown state,
|
|
71 |
since this will not be done automatically by Xen anymore because of the
|
|
72 |
``preserve`` setting in their config files.
|
|
73 |
|
|
74 |
This behavior will be limited to the domains shut down from inside, because it
|
|
75 |
will actually keep the resources of the domain busy until the watcher will do
|
|
76 |
the cleaning job (that, with the default setting, is up to every 5 minutes).
|
|
77 |
Still, this is considered acceptable, because it is not frequent for a domain
|
|
78 |
to be shut down this way. The cleanup function will be also run
|
|
79 |
automatically just before performing any job that requires resources to be
|
|
80 |
available (such as when creating a new instance), in order to ensure that the
|
|
81 |
new resource allocation happens starting from a clean state. Functionalities
|
|
82 |
that only query the state of instances will not run the cleanup function.
|
|
83 |
|
|
84 |
The cleanup operation includes both node-specific operations (the actual
|
|
85 |
destruction of the stopped domains) and configuration changes, to be performed
|
|
86 |
on the master node (marking as offline an instance that was shut down
|
|
87 |
internally). The watcher, on the master node, will fetch the list of instances
|
|
88 |
that have been shutdown from inside (recognizable by their ``oper_state``
|
|
89 |
as described below). It will then submit a series of ``InstanceShutdown`` jobs
|
|
90 |
that will mark such instances as ``ADMIN_down`` and clean them up (after
|
|
91 |
the functionality of ``InstanceShutdown`` will have been extended as specified
|
|
92 |
in the rest of this design document).
|
|
93 |
|
|
94 |
LUs performing operations other than an explicit cleanup will have to be
|
|
95 |
modified to perform the cleanup as well, either by submitting a job to perform
|
|
96 |
the cleanup (to be completed before actually performing the task at hand) or by
|
|
97 |
explicitly performing the cleanup themselves through the RPC calls.
|
|
98 |
|
|
99 |
Other required changes
|
|
100 |
++++++++++++++++++++++
|
|
101 |
|
|
102 |
The implementation of this design document will require some commands to be
|
|
103 |
changed in order to cope with the new shutdown procedure.
|
|
104 |
|
|
105 |
With the default shutdown action in Xen set to ``preserve``, the Ganeti
|
|
106 |
command for shutting down instances would leave them in a shutdown but
|
|
107 |
preserved state. Therefore, it will have to be changed in such a way to
|
|
108 |
immediately perform the cleanup of the instance after verifying its correct
|
|
109 |
shutdown. Also, it will correctly deal with instances that have been shutdown
|
|
110 |
from inside but are still active according to Ganeti, by detecting this
|
|
111 |
situation, destroying the instance and carrying out the rest of the Ganeti
|
|
112 |
shutdown procedure as usual.
|
|
113 |
|
|
114 |
The ``gnt-instance list`` command will need to be able to handle the situation
|
|
115 |
where an instance was shutdown internally but not yet cleaned up.
|
|
116 |
The ``admin_state`` field will maintain the current meaning unchanged. The
|
|
117 |
``oper_state`` field will get a new possible state, ``S``, meaning that the
|
|
118 |
instance was shutdown internally.
|
|
119 |
|
|
120 |
The ``gnt-instance info`` command ``State`` field, in such case, will show a
|
|
121 |
message stating that the instance was supposed to be run but was shut down
|
|
122 |
internally.
|
|
123 |
|
|
124 |
.. vim: set textwidth=72 :
|
|
125 |
.. Local Variables:
|
|
126 |
.. mode: rst
|
|
127 |
.. fill-column: 72
|
|
128 |
.. End:
|