root / doc / design-internal-shutdown.rst @ c8fca479
History | View | Annotate | Download (6.3 kB)
1 | ca93ea0a | Michele Tartara | ============================================================ |
---|---|---|---|
2 | ca93ea0a | Michele Tartara | Detection of user-initiated shutdown from inside an instance |
3 | ca93ea0a | Michele Tartara | ============================================================ |
4 | ca93ea0a | Michele Tartara | |
5 | ca93ea0a | Michele Tartara | .. contents:: :depth: 2 |
6 | ca93ea0a | Michele Tartara | |
7 | ca93ea0a | Michele Tartara | This is a design document detailing the implementation of a way for Ganeti to |
8 | ca93ea0a | Michele Tartara | detect whether a machine marked as up but not running was shutdown gracefully |
9 | ca93ea0a | Michele Tartara | by the user from inside the machine itself. |
10 | ca93ea0a | Michele Tartara | |
11 | ca93ea0a | Michele Tartara | Current state and shortcomings |
12 | ca93ea0a | Michele Tartara | ============================== |
13 | ca93ea0a | Michele Tartara | |
14 | ca93ea0a | Michele Tartara | Ganeti keeps track of the desired status of instances in order to be able to |
15 | ca93ea0a | Michele Tartara | take proper actions (e.g.: reboot) on the ones that happen to crash. |
16 | ca93ea0a | Michele Tartara | Currently, the only way to properly shut down a machine is through Ganeti's own |
17 | ca93ea0a | Michele Tartara | commands, that will mark an instance as ``ADMIN_down``. |
18 | ca93ea0a | Michele Tartara | If a user shuts down an instance from inside, through the proper command of the |
19 | ca93ea0a | Michele Tartara | operating system it is running, the instance will be shutdown gracefully, but |
20 | ca93ea0a | Michele Tartara | Ganeti is not aware of that: the desired status of the instance will still be |
21 | ca93ea0a | Michele Tartara | marked as ``running``, so when the watcher realises that the instance is down, |
22 | ca93ea0a | Michele Tartara | it will restart it. This behaviour is usually not what the user expects. |
23 | ca93ea0a | Michele Tartara | |
24 | ca93ea0a | Michele Tartara | Proposed changes |
25 | ca93ea0a | Michele Tartara | ================ |
26 | ca93ea0a | Michele Tartara | |
27 | ca93ea0a | Michele Tartara | We propose to modify Ganeti in such a way that it will detect when an instance |
28 | ca93ea0a | Michele Tartara | was shutdown because of an explicit user request. When such a situation is |
29 | ca93ea0a | Michele Tartara | detected, instead of presenting an error as it happens now, either the state |
30 | ca93ea0a | Michele Tartara | of the instance will be set to ADMIN_down, or the instance will be |
31 | ca93ea0a | Michele Tartara | automatically rebooted, depending on a instance-specific configuration value. |
32 | ca93ea0a | Michele Tartara | The default behavior in case no such parameter is found will be to follow |
33 | ca93ea0a | Michele Tartara | the apparent will of the user, and setting to ADMIN_down an instance that |
34 | ca93ea0a | Michele Tartara | was shut down correctly from inside. |
35 | ca93ea0a | Michele Tartara | |
36 | ca93ea0a | Michele Tartara | This design document applies to the Xen backend of Ganeti, because it uses |
37 | ca93ea0a | Michele Tartara | features specific of such hypervisor. Initial analysis suggests that a similar |
38 | ca93ea0a | Michele Tartara | approach might be used for KVM as well, so this design document will be later |
39 | ca93ea0a | Michele Tartara | extended to add more details about it. |
40 | ca93ea0a | Michele Tartara | |
41 | ca93ea0a | Michele Tartara | Implementation |
42 | ca93ea0a | Michele Tartara | ============== |
43 | ca93ea0a | Michele Tartara | |
44 | ca93ea0a | Michele Tartara | Xen knows why a domain is being shut down (a crash or an explicit shutdown |
45 | ca93ea0a | Michele Tartara | or poweroff request), but such information is not usually readily available |
46 | ca93ea0a | Michele Tartara | externally, because all such cases lead to the virtual machine being destroyed |
47 | ca93ea0a | Michele Tartara | immediately after the event is detected. |
48 | ca93ea0a | Michele Tartara | |
49 | ca93ea0a | Michele Tartara | Still, Xen allows the instance configuration file to define what action to be |
50 | ca93ea0a | Michele Tartara | taken in all those cases through the ``on_poweroff``, ``on_shutdown`` and |
51 | ca93ea0a | Michele Tartara | ``on_crash`` variables. By setting them to ``preserve``, Xen will avoid |
52 | ca93ea0a | Michele Tartara | destroying the domains automatically. |
53 | ca93ea0a | Michele Tartara | |
54 | ca93ea0a | Michele Tartara | When the domain is not destroyed, it can be viewed by using ``xm list`` (or ``xl |
55 | ca93ea0a | Michele Tartara | list`` in newer Xen versions), and the ``State`` field of the output will |
56 | ca93ea0a | Michele Tartara | provide useful information. |
57 | ca93ea0a | Michele Tartara | |
58 | ca93ea0a | Michele Tartara | If the state is ``----c-`` it means the instance has crashed. |
59 | ca93ea0a | Michele Tartara | |
60 | ca93ea0a | Michele Tartara | If the state is ``---s--`` it means the instance was properly shutdown. |
61 | ca93ea0a | Michele Tartara | |
62 | ca93ea0a | Michele Tartara | If the instance was properly shutdown and it is still marked as ``running`` by |
63 | ca93ea0a | Michele Tartara | Ganeti, it means that it was shutdown from inside by the user, and the ganeti |
64 | ca93ea0a | Michele Tartara | status of the instance needs to be changed to ``ADMIN_down``. |
65 | ca93ea0a | Michele Tartara | |
66 | ca93ea0a | Michele Tartara | This will be done at regular intervals by the group watcher, just before |
67 | ca93ea0a | Michele Tartara | deciding which instances to reboot. |
68 | ca93ea0a | Michele Tartara | |
69 | ca93ea0a | Michele Tartara | On top of that, at the same times, the watcher will also need to issue ``xm |
70 | ca93ea0a | Michele Tartara | destroy`` commands for all the domains that are in crashed or shutdown state, |
71 | ca93ea0a | Michele Tartara | since this will not be done automatically by Xen anymore because of the |
72 | ca93ea0a | Michele Tartara | ``preserve`` setting in their config files. |
73 | ca93ea0a | Michele Tartara | |
74 | ca93ea0a | Michele Tartara | This behavior will be limited to the domains shut down from inside, because it |
75 | ca93ea0a | Michele Tartara | will actually keep the resources of the domain busy until the watcher will do |
76 | ca93ea0a | Michele Tartara | the cleaning job (that, with the default setting, is up to every 5 minutes). |
77 | ca93ea0a | Michele Tartara | Still, this is considered acceptable, because it is not frequent for a domain |
78 | ca93ea0a | Michele Tartara | to be shut down this way. The cleanup function will be also run |
79 | ca93ea0a | Michele Tartara | automatically just before performing any job that requires resources to be |
80 | ca93ea0a | Michele Tartara | available (such as when creating a new instance), in order to ensure that the |
81 | ca93ea0a | Michele Tartara | new resource allocation happens starting from a clean state. Functionalities |
82 | ca93ea0a | Michele Tartara | that only query the state of instances will not run the cleanup function. |
83 | ca93ea0a | Michele Tartara | |
84 | ca93ea0a | Michele Tartara | The cleanup operation includes both node-specific operations (the actual |
85 | ca93ea0a | Michele Tartara | destruction of the stopped domains) and configuration changes, to be performed |
86 | ca93ea0a | Michele Tartara | on the master node (marking as offline an instance that was shut down |
87 | ca93ea0a | Michele Tartara | internally). The watcher, on the master node, will fetch the list of instances |
88 | ca93ea0a | Michele Tartara | that have been shutdown from inside (recognizable by their ``oper_state`` |
89 | ca93ea0a | Michele Tartara | as described below). It will then submit a series of ``InstanceShutdown`` jobs |
90 | ca93ea0a | Michele Tartara | that will mark such instances as ``ADMIN_down`` and clean them up (after |
91 | ca93ea0a | Michele Tartara | the functionality of ``InstanceShutdown`` will have been extended as specified |
92 | ca93ea0a | Michele Tartara | in the rest of this design document). |
93 | ca93ea0a | Michele Tartara | |
94 | ca93ea0a | Michele Tartara | LUs performing operations other than an explicit cleanup will have to be |
95 | ca93ea0a | Michele Tartara | modified to perform the cleanup as well, either by submitting a job to perform |
96 | ca93ea0a | Michele Tartara | the cleanup (to be completed before actually performing the task at hand) or by |
97 | ca93ea0a | Michele Tartara | explicitly performing the cleanup themselves through the RPC calls. |
98 | ca93ea0a | Michele Tartara | |
99 | ca93ea0a | Michele Tartara | Other required changes |
100 | ca93ea0a | Michele Tartara | ++++++++++++++++++++++ |
101 | ca93ea0a | Michele Tartara | |
102 | ca93ea0a | Michele Tartara | The implementation of this design document will require some commands to be |
103 | ca93ea0a | Michele Tartara | changed in order to cope with the new shutdown procedure. |
104 | ca93ea0a | Michele Tartara | |
105 | ca93ea0a | Michele Tartara | With the default shutdown action in Xen set to ``preserve``, the Ganeti |
106 | ca93ea0a | Michele Tartara | command for shutting down instances would leave them in a shutdown but |
107 | ca93ea0a | Michele Tartara | preserved state. Therefore, it will have to be changed in such a way to |
108 | ca93ea0a | Michele Tartara | immediately perform the cleanup of the instance after verifying its correct |
109 | ca93ea0a | Michele Tartara | shutdown. Also, it will correctly deal with instances that have been shutdown |
110 | ca93ea0a | Michele Tartara | from inside but are still active according to Ganeti, by detecting this |
111 | ca93ea0a | Michele Tartara | situation, destroying the instance and carrying out the rest of the Ganeti |
112 | ca93ea0a | Michele Tartara | shutdown procedure as usual. |
113 | ca93ea0a | Michele Tartara | |
114 | ca93ea0a | Michele Tartara | The ``gnt-instance list`` command will need to be able to handle the situation |
115 | ca93ea0a | Michele Tartara | where an instance was shutdown internally but not yet cleaned up. |
116 | ca93ea0a | Michele Tartara | The ``admin_state`` field will maintain the current meaning unchanged. The |
117 | ca93ea0a | Michele Tartara | ``oper_state`` field will get a new possible state, ``S``, meaning that the |
118 | ca93ea0a | Michele Tartara | instance was shutdown internally. |
119 | ca93ea0a | Michele Tartara | |
120 | ca93ea0a | Michele Tartara | The ``gnt-instance info`` command ``State`` field, in such case, will show a |
121 | ca93ea0a | Michele Tartara | message stating that the instance was supposed to be run but was shut down |
122 | ca93ea0a | Michele Tartara | internally. |
123 | ca93ea0a | Michele Tartara | |
124 | ca93ea0a | Michele Tartara | .. vim: set textwidth=72 : |
125 | ca93ea0a | Michele Tartara | .. Local Variables: |
126 | ca93ea0a | Michele Tartara | .. mode: rst |
127 | ca93ea0a | Michele Tartara | .. fill-column: 72 |
128 | ca93ea0a | Michele Tartara | .. End: |