root / doc / design-internal-shutdown.rst @ 27eb699d
History | View | Annotate | Download (6.3 kB)
1 |
============================================================ |
---|---|
2 |
Detection of user-initiated shutdown from inside an instance |
3 |
============================================================ |
4 |
|
5 |
.. contents:: :depth: 2 |
6 |
|
7 |
This is a design document detailing the implementation of a way for Ganeti to |
8 |
detect whether a machine marked as up but not running was shutdown gracefully |
9 |
by the user from inside the machine itself. |
10 |
|
11 |
Current state and shortcomings |
12 |
============================== |
13 |
|
14 |
Ganeti keeps track of the desired status of instances in order to be able to |
15 |
take proper actions (e.g.: reboot) on the ones that happen to crash. |
16 |
Currently, the only way to properly shut down a machine is through Ganeti's own |
17 |
commands, that will mark an instance as ``ADMIN_down``. |
18 |
If a user shuts down an instance from inside, through the proper command of the |
19 |
operating system it is running, the instance will be shutdown gracefully, but |
20 |
Ganeti is not aware of that: the desired status of the instance will still be |
21 |
marked as ``running``, so when the watcher realises that the instance is down, |
22 |
it will restart it. This behaviour is usually not what the user expects. |
23 |
|
24 |
Proposed changes |
25 |
================ |
26 |
|
27 |
We propose to modify Ganeti in such a way that it will detect when an instance |
28 |
was shutdown because of an explicit user request. When such a situation is |
29 |
detected, instead of presenting an error as it happens now, either the state |
30 |
of the instance will be set to ADMIN_down, or the instance will be |
31 |
automatically rebooted, depending on a instance-specific configuration value. |
32 |
The default behavior in case no such parameter is found will be to follow |
33 |
the apparent will of the user, and setting to ADMIN_down an instance that |
34 |
was shut down correctly from inside. |
35 |
|
36 |
This design document applies to the Xen backend of Ganeti, because it uses |
37 |
features specific of such hypervisor. Initial analysis suggests that a similar |
38 |
approach might be used for KVM as well, so this design document will be later |
39 |
extended to add more details about it. |
40 |
|
41 |
Implementation |
42 |
============== |
43 |
|
44 |
Xen knows why a domain is being shut down (a crash or an explicit shutdown |
45 |
or poweroff request), but such information is not usually readily available |
46 |
externally, because all such cases lead to the virtual machine being destroyed |
47 |
immediately after the event is detected. |
48 |
|
49 |
Still, Xen allows the instance configuration file to define what action to be |
50 |
taken in all those cases through the ``on_poweroff``, ``on_shutdown`` and |
51 |
``on_crash`` variables. By setting them to ``preserve``, Xen will avoid |
52 |
destroying the domains automatically. |
53 |
|
54 |
When the domain is not destroyed, it can be viewed by using ``xm list`` (or ``xl |
55 |
list`` in newer Xen versions), and the ``State`` field of the output will |
56 |
provide useful information. |
57 |
|
58 |
If the state is ``----c-`` it means the instance has crashed. |
59 |
|
60 |
If the state is ``---s--`` it means the instance was properly shutdown. |
61 |
|
62 |
If the instance was properly shutdown and it is still marked as ``running`` by |
63 |
Ganeti, it means that it was shutdown from inside by the user, and the ganeti |
64 |
status of the instance needs to be changed to ``ADMIN_down``. |
65 |
|
66 |
This will be done at regular intervals by the group watcher, just before |
67 |
deciding which instances to reboot. |
68 |
|
69 |
On top of that, at the same times, the watcher will also need to issue ``xm |
70 |
destroy`` commands for all the domains that are in crashed or shutdown state, |
71 |
since this will not be done automatically by Xen anymore because of the |
72 |
``preserve`` setting in their config files. |
73 |
|
74 |
This behavior will be limited to the domains shut down from inside, because it |
75 |
will actually keep the resources of the domain busy until the watcher will do |
76 |
the cleaning job (that, with the default setting, is up to every 5 minutes). |
77 |
Still, this is considered acceptable, because it is not frequent for a domain |
78 |
to be shut down this way. The cleanup function will be also run |
79 |
automatically just before performing any job that requires resources to be |
80 |
available (such as when creating a new instance), in order to ensure that the |
81 |
new resource allocation happens starting from a clean state. Functionalities |
82 |
that only query the state of instances will not run the cleanup function. |
83 |
|
84 |
The cleanup operation includes both node-specific operations (the actual |
85 |
destruction of the stopped domains) and configuration changes, to be performed |
86 |
on the master node (marking as offline an instance that was shut down |
87 |
internally). The watcher, on the master node, will fetch the list of instances |
88 |
that have been shutdown from inside (recognizable by their ``oper_state`` |
89 |
as described below). It will then submit a series of ``InstanceShutdown`` jobs |
90 |
that will mark such instances as ``ADMIN_down`` and clean them up (after |
91 |
the functionality of ``InstanceShutdown`` will have been extended as specified |
92 |
in the rest of this design document). |
93 |
|
94 |
LUs performing operations other than an explicit cleanup will have to be |
95 |
modified to perform the cleanup as well, either by submitting a job to perform |
96 |
the cleanup (to be completed before actually performing the task at hand) or by |
97 |
explicitly performing the cleanup themselves through the RPC calls. |
98 |
|
99 |
Other required changes |
100 |
++++++++++++++++++++++ |
101 |
|
102 |
The implementation of this design document will require some commands to be |
103 |
changed in order to cope with the new shutdown procedure. |
104 |
|
105 |
With the default shutdown action in Xen set to ``preserve``, the Ganeti |
106 |
command for shutting down instances would leave them in a shutdown but |
107 |
preserved state. Therefore, it will have to be changed in such a way to |
108 |
immediately perform the cleanup of the instance after verifying its correct |
109 |
shutdown. Also, it will correctly deal with instances that have been shutdown |
110 |
from inside but are still active according to Ganeti, by detecting this |
111 |
situation, destroying the instance and carrying out the rest of the Ganeti |
112 |
shutdown procedure as usual. |
113 |
|
114 |
The ``gnt-instance list`` command will need to be able to handle the situation |
115 |
where an instance was shutdown internally but not yet cleaned up. |
116 |
The ``admin_state`` field will maintain the current meaning unchanged. The |
117 |
``oper_state`` field will get a new possible state, ``S``, meaning that the |
118 |
instance was shutdown internally. |
119 |
|
120 |
The ``gnt-instance info`` command ``State`` field, in such case, will show a |
121 |
message stating that the instance was supposed to be run but was shut down |
122 |
internally. |
123 |
|
124 |
.. vim: set textwidth=72 : |
125 |
.. Local Variables: |
126 |
.. mode: rst |
127 |
.. fill-column: 72 |
128 |
.. End: |