root / doc / design-internal-shutdown.rst @ 0565f862
History | View | Annotate | Download (6.2 kB)
1 | ca93ea0a | Michele Tartara | ============================================================ |
---|---|---|---|
2 | ca93ea0a | Michele Tartara | Detection of user-initiated shutdown from inside an instance |
3 | ca93ea0a | Michele Tartara | ============================================================ |
4 | ca93ea0a | Michele Tartara | |
5 | ca93ea0a | Michele Tartara | .. contents:: :depth: 2 |
6 | ca93ea0a | Michele Tartara | |
7 | ca93ea0a | Michele Tartara | This is a design document detailing the implementation of a way for Ganeti to |
8 | 96eeb742 | Jose A. Lopes | detect whether an instance marked as up but not running was shutdown gracefully |
9 | 96eeb742 | Jose A. Lopes | by the user from inside the instance itself. |
10 | ca93ea0a | Michele Tartara | |
11 | ca93ea0a | Michele Tartara | Current state and shortcomings |
12 | ca93ea0a | Michele Tartara | ============================== |
13 | ca93ea0a | Michele Tartara | |
14 | ca93ea0a | Michele Tartara | Ganeti keeps track of the desired status of instances in order to be able to |
15 | 96eeb742 | Jose A. Lopes | take proper action (e.g.: reboot) on the instances that happen to crash. |
16 | 96eeb742 | Jose A. Lopes | Currently, the only way to properly shut down an instance is through Ganeti's |
17 | 96eeb742 | Jose A. Lopes | own commands, which can be used to mark an instance as ``ADMIN_down``. |
18 | 96eeb742 | Jose A. Lopes | |
19 | ca93ea0a | Michele Tartara | If a user shuts down an instance from inside, through the proper command of the |
20 | ca93ea0a | Michele Tartara | operating system it is running, the instance will be shutdown gracefully, but |
21 | ca93ea0a | Michele Tartara | Ganeti is not aware of that: the desired status of the instance will still be |
22 | ca93ea0a | Michele Tartara | marked as ``running``, so when the watcher realises that the instance is down, |
23 | ca93ea0a | Michele Tartara | it will restart it. This behaviour is usually not what the user expects. |
24 | ca93ea0a | Michele Tartara | |
25 | ca93ea0a | Michele Tartara | Proposed changes |
26 | ca93ea0a | Michele Tartara | ================ |
27 | ca93ea0a | Michele Tartara | |
28 | ca93ea0a | Michele Tartara | We propose to modify Ganeti in such a way that it will detect when an instance |
29 | 96eeb742 | Jose A. Lopes | was shutdown as a result of an explicit request from the user. When such a |
30 | 96eeb742 | Jose A. Lopes | situation is detected, instead of presenting an error as it happens now, either |
31 | 96eeb742 | Jose A. Lopes | the state of the instance will be set to ``ADMIN_down``, or the instance will be |
32 | 96eeb742 | Jose A. Lopes | automatically rebooted, depending on an instance-specific configuration value. |
33 | 96eeb742 | Jose A. Lopes | The default behavior in case no such parameter is found will be to follow the |
34 | 96eeb742 | Jose A. Lopes | apparent will of the user, and setting to ``ADMIN_down`` an instance that was |
35 | 96eeb742 | Jose A. Lopes | shut down correctly from inside. |
36 | 96eeb742 | Jose A. Lopes | |
37 | 96eeb742 | Jose A. Lopes | The rest of this design document details the implementation of instance shutdown |
38 | 96eeb742 | Jose A. Lopes | detection for Xen. The KVM implementation is detailed in :doc:`design-kvmd`. |
39 | ca93ea0a | Michele Tartara | |
40 | ca93ea0a | Michele Tartara | Implementation |
41 | ca93ea0a | Michele Tartara | ============== |
42 | ca93ea0a | Michele Tartara | |
43 | ca93ea0a | Michele Tartara | Xen knows why a domain is being shut down (a crash or an explicit shutdown |
44 | ca93ea0a | Michele Tartara | or poweroff request), but such information is not usually readily available |
45 | ca93ea0a | Michele Tartara | externally, because all such cases lead to the virtual machine being destroyed |
46 | ca93ea0a | Michele Tartara | immediately after the event is detected. |
47 | ca93ea0a | Michele Tartara | |
48 | ca93ea0a | Michele Tartara | Still, Xen allows the instance configuration file to define what action to be |
49 | ca93ea0a | Michele Tartara | taken in all those cases through the ``on_poweroff``, ``on_shutdown`` and |
50 | ca93ea0a | Michele Tartara | ``on_crash`` variables. By setting them to ``preserve``, Xen will avoid |
51 | ca93ea0a | Michele Tartara | destroying the domains automatically. |
52 | ca93ea0a | Michele Tartara | |
53 | ca93ea0a | Michele Tartara | When the domain is not destroyed, it can be viewed by using ``xm list`` (or ``xl |
54 | ca93ea0a | Michele Tartara | list`` in newer Xen versions), and the ``State`` field of the output will |
55 | ca93ea0a | Michele Tartara | provide useful information. |
56 | ca93ea0a | Michele Tartara | |
57 | ca93ea0a | Michele Tartara | If the state is ``----c-`` it means the instance has crashed. |
58 | ca93ea0a | Michele Tartara | |
59 | ca93ea0a | Michele Tartara | If the state is ``---s--`` it means the instance was properly shutdown. |
60 | ca93ea0a | Michele Tartara | |
61 | ca93ea0a | Michele Tartara | If the instance was properly shutdown and it is still marked as ``running`` by |
62 | 96eeb742 | Jose A. Lopes | Ganeti, it means that it was shutdown from inside by the user, and the Ganeti |
63 | ca93ea0a | Michele Tartara | status of the instance needs to be changed to ``ADMIN_down``. |
64 | ca93ea0a | Michele Tartara | |
65 | ca93ea0a | Michele Tartara | This will be done at regular intervals by the group watcher, just before |
66 | ca93ea0a | Michele Tartara | deciding which instances to reboot. |
67 | ca93ea0a | Michele Tartara | |
68 | 96eeb742 | Jose A. Lopes | On top of that, at the same time, the watcher will also need to issue ``xm |
69 | 96eeb742 | Jose A. Lopes | destroy`` commands for all the domains that are in a crashed or shutdown state, |
70 | ca93ea0a | Michele Tartara | since this will not be done automatically by Xen anymore because of the |
71 | ca93ea0a | Michele Tartara | ``preserve`` setting in their config files. |
72 | ca93ea0a | Michele Tartara | |
73 | ca93ea0a | Michele Tartara | This behavior will be limited to the domains shut down from inside, because it |
74 | ca93ea0a | Michele Tartara | will actually keep the resources of the domain busy until the watcher will do |
75 | ca93ea0a | Michele Tartara | the cleaning job (that, with the default setting, is up to every 5 minutes). |
76 | 96eeb742 | Jose A. Lopes | Still, this is considered acceptable, because it is not frequent for a domain to |
77 | 96eeb742 | Jose A. Lopes | be shut down this way. The cleanup function will be also run automatically just |
78 | 96eeb742 | Jose A. Lopes | before performing any job that requires resources to be available (such as when |
79 | 96eeb742 | Jose A. Lopes | creating a new instance), in order to ensure that the new resource allocation |
80 | 96eeb742 | Jose A. Lopes | happens starting from a clean state. Functionalities that only query the state |
81 | 96eeb742 | Jose A. Lopes | of instances will not run the cleanup function. |
82 | ca93ea0a | Michele Tartara | |
83 | ca93ea0a | Michele Tartara | The cleanup operation includes both node-specific operations (the actual |
84 | ca93ea0a | Michele Tartara | destruction of the stopped domains) and configuration changes, to be performed |
85 | ca93ea0a | Michele Tartara | on the master node (marking as offline an instance that was shut down |
86 | ca93ea0a | Michele Tartara | internally). The watcher, on the master node, will fetch the list of instances |
87 | ca93ea0a | Michele Tartara | that have been shutdown from inside (recognizable by their ``oper_state`` |
88 | ca93ea0a | Michele Tartara | as described below). It will then submit a series of ``InstanceShutdown`` jobs |
89 | ca93ea0a | Michele Tartara | that will mark such instances as ``ADMIN_down`` and clean them up (after |
90 | ca93ea0a | Michele Tartara | the functionality of ``InstanceShutdown`` will have been extended as specified |
91 | ca93ea0a | Michele Tartara | in the rest of this design document). |
92 | ca93ea0a | Michele Tartara | |
93 | ca93ea0a | Michele Tartara | LUs performing operations other than an explicit cleanup will have to be |
94 | ca93ea0a | Michele Tartara | modified to perform the cleanup as well, either by submitting a job to perform |
95 | ca93ea0a | Michele Tartara | the cleanup (to be completed before actually performing the task at hand) or by |
96 | ca93ea0a | Michele Tartara | explicitly performing the cleanup themselves through the RPC calls. |
97 | ca93ea0a | Michele Tartara | |
98 | ca93ea0a | Michele Tartara | Other required changes |
99 | ca93ea0a | Michele Tartara | ++++++++++++++++++++++ |
100 | ca93ea0a | Michele Tartara | |
101 | ca93ea0a | Michele Tartara | The implementation of this design document will require some commands to be |
102 | ca93ea0a | Michele Tartara | changed in order to cope with the new shutdown procedure. |
103 | ca93ea0a | Michele Tartara | |
104 | ca93ea0a | Michele Tartara | With the default shutdown action in Xen set to ``preserve``, the Ganeti |
105 | ca93ea0a | Michele Tartara | command for shutting down instances would leave them in a shutdown but |
106 | ca93ea0a | Michele Tartara | preserved state. Therefore, it will have to be changed in such a way to |
107 | ca93ea0a | Michele Tartara | immediately perform the cleanup of the instance after verifying its correct |
108 | ca93ea0a | Michele Tartara | shutdown. Also, it will correctly deal with instances that have been shutdown |
109 | ca93ea0a | Michele Tartara | from inside but are still active according to Ganeti, by detecting this |
110 | ca93ea0a | Michele Tartara | situation, destroying the instance and carrying out the rest of the Ganeti |
111 | ca93ea0a | Michele Tartara | shutdown procedure as usual. |
112 | ca93ea0a | Michele Tartara | |
113 | ca93ea0a | Michele Tartara | The ``gnt-instance list`` command will need to be able to handle the situation |
114 | 96eeb742 | Jose A. Lopes | where an instance was shutdown internally but not yet cleaned up. The |
115 | 96eeb742 | Jose A. Lopes | ``admin_state`` field will maintain the current meaning unchanged. The |
116 | ca93ea0a | Michele Tartara | ``oper_state`` field will get a new possible state, ``S``, meaning that the |
117 | ca93ea0a | Michele Tartara | instance was shutdown internally. |
118 | ca93ea0a | Michele Tartara | |
119 | ca93ea0a | Michele Tartara | The ``gnt-instance info`` command ``State`` field, in such case, will show a |
120 | ca93ea0a | Michele Tartara | message stating that the instance was supposed to be run but was shut down |
121 | ca93ea0a | Michele Tartara | internally. |
122 | ca93ea0a | Michele Tartara | |
123 | ca93ea0a | Michele Tartara | .. vim: set textwidth=72 : |
124 | ca93ea0a | Michele Tartara | .. Local Variables: |
125 | ca93ea0a | Michele Tartara | .. mode: rst |
126 | ca93ea0a | Michele Tartara | .. fill-column: 72 |
127 | ca93ea0a | Michele Tartara | .. End: |