root / doc / design-optables.rst @ 0565f862
History | View | Annotate | Download (7.2 kB)
1 | e408eb8a | Klaus Aehlig | ========================================== |
---|---|---|---|
2 | e408eb8a | Klaus Aehlig | Filtering of jobs for the Ganeti job queue |
3 | e408eb8a | Klaus Aehlig | ========================================== |
4 | e408eb8a | Klaus Aehlig | |
5 | e408eb8a | Klaus Aehlig | .. contents:: :depth: 4 |
6 | e408eb8a | Klaus Aehlig | |
7 | e408eb8a | Klaus Aehlig | This is a design document detailing the semantics of the fine-grained control |
8 | e408eb8a | Klaus Aehlig | of jobs in Ganeti. For the implementation there will be a separate |
9 | e408eb8a | Klaus Aehlig | design document that also describes the vision for the Ganeti daemon |
10 | e408eb8a | Klaus Aehlig | structure. |
11 | e408eb8a | Klaus Aehlig | |
12 | e408eb8a | Klaus Aehlig | |
13 | e408eb8a | Klaus Aehlig | Current state and shortcomings |
14 | e408eb8a | Klaus Aehlig | ============================== |
15 | e408eb8a | Klaus Aehlig | |
16 | e408eb8a | Klaus Aehlig | Control of the Ganeti job queue is quite limited. There is a single |
17 | e408eb8a | Klaus Aehlig | status bit, the "drained flag". If set, no new jobs are accepted to |
18 | e408eb8a | Klaus Aehlig | the queue. This is too coarse for some use cases. |
19 | e408eb8a | Klaus Aehlig | |
20 | e408eb8a | Klaus Aehlig | - The queue might be required to be drained for several reasons, |
21 | e408eb8a | Klaus Aehlig | initiated by different persons or automatic programs. Each one |
22 | e408eb8a | Klaus Aehlig | should be able to indicate that his reasons for draining are over |
23 | e408eb8a | Klaus Aehlig | without affecting the others. |
24 | e408eb8a | Klaus Aehlig | |
25 | e408eb8a | Klaus Aehlig | - There is no support for partial drains. For example, one might want |
26 | e408eb8a | Klaus Aehlig | to allow all jobs belonging to a manual (or externally coordinated) |
27 | e408eb8a | Klaus Aehlig | maintenance, while disallowing all other jobs. |
28 | e408eb8a | Klaus Aehlig | |
29 | e408eb8a | Klaus Aehlig | - There is no support for blocking jobs by their op-codes, e.g., |
30 | e408eb8a | Klaus Aehlig | disallowing all jobs that bring new instances to a cluster. This might |
31 | e408eb8a | Klaus Aehlig | be part of a maintenance preparation. |
32 | e408eb8a | Klaus Aehlig | |
33 | e408eb8a | Klaus Aehlig | - There is no support for a soft version of draining, where all |
34 | e408eb8a | Klaus Aehlig | jobs currently in the queue are finished, while new jobs entering |
35 | e408eb8a | Klaus Aehlig | the queue are delayed until the drain is over. |
36 | e408eb8a | Klaus Aehlig | |
37 | e408eb8a | Klaus Aehlig | |
38 | e408eb8a | Klaus Aehlig | Proposed changes |
39 | e408eb8a | Klaus Aehlig | ================ |
40 | e408eb8a | Klaus Aehlig | |
41 | e408eb8a | Klaus Aehlig | We propose to add filters on the job queue. These will be part of the |
42 | e408eb8a | Klaus Aehlig | configuration and as such are persisted with it. Conceptionally, the |
43 | e408eb8a | Klaus Aehlig | filters are always processed when a job enters the queue and while it |
44 | e408eb8a | Klaus Aehlig | is still in the queue. Of course, in the implementation, reevaluation |
45 | e408eb8a | Klaus Aehlig | is only carried out, if something could make the result change, e.g., |
46 | e408eb8a | Klaus Aehlig | a new job is entered to the queue, or the filter rules are changed. |
47 | e408eb8a | Klaus Aehlig | There is no distinction between filter processing when a job is about |
48 | e408eb8a | Klaus Aehlig | to enter the queue and while it is in the queue, as this can be |
49 | e408eb8a | Klaus Aehlig | expressed by the filter rules themselves (see predicates below). |
50 | e408eb8a | Klaus Aehlig | |
51 | e408eb8a | Klaus Aehlig | Format of a Filter rule |
52 | e408eb8a | Klaus Aehlig | ----------------------- |
53 | e408eb8a | Klaus Aehlig | |
54 | e408eb8a | Klaus Aehlig | Filter rules are given by the following data. |
55 | e408eb8a | Klaus Aehlig | |
56 | e408eb8a | Klaus Aehlig | - A UUID. This ensures that there can be different filter rules |
57 | e408eb8a | Klaus Aehlig | that otherwise have all parameters equal. In this way, multiple |
58 | e408eb8a | Klaus Aehlig | drains for different reasons are possible. The UUID is used to |
59 | e408eb8a | Klaus Aehlig | address the filter rule, in particular for deletion. |
60 | e408eb8a | Klaus Aehlig | |
61 | e408eb8a | Klaus Aehlig | If no UUID is provided at rule addition, Ganeti will create one. |
62 | e408eb8a | Klaus Aehlig | |
63 | e408eb8a | Klaus Aehlig | - The watermark. This is the highest job id ever used, as valid in |
64 | e408eb8a | Klaus Aehlig | the moment when the filter was added. This data will be added |
65 | e408eb8a | Klaus Aehlig | automatically upon addition of the filter. |
66 | e408eb8a | Klaus Aehlig | |
67 | e408eb8a | Klaus Aehlig | - A priority. This is a non-negative integer. Filters are processed |
68 | e408eb8a | Klaus Aehlig | in order of increasing priority until a rule applies. While there |
69 | e408eb8a | Klaus Aehlig | is a well-defined order in which rules of the same priority are |
70 | e408eb8a | Klaus Aehlig | evaluated (increasing watermark, then the uuid, are taken as tie |
71 | e408eb8a | Klaus Aehlig | breakers), it is not recommended to have rules of the same priority |
72 | e408eb8a | Klaus Aehlig | that overlap and have different actions associated. |
73 | e408eb8a | Klaus Aehlig | |
74 | e408eb8a | Klaus Aehlig | - A list of predicates. The rule fires, if all of them hold true |
75 | e408eb8a | Klaus Aehlig | for the job. |
76 | e408eb8a | Klaus Aehlig | |
77 | e408eb8a | Klaus Aehlig | - An action. For the time being, one of the following, but more |
78 | e408eb8a | Klaus Aehlig | actions might be added in the future (in particular, future |
79 | e408eb8a | Klaus Aehlig | implementations might add an action making filtering continue with |
80 | e408eb8a | Klaus Aehlig | a different filter chain). |
81 | e408eb8a | Klaus Aehlig | |
82 | e408eb8a | Klaus Aehlig | - ACCEPT. The job will be accepted; no further filter rules |
83 | e408eb8a | Klaus Aehlig | are applied. |
84 | e408eb8a | Klaus Aehlig | - PAUSE. The job will be accepted to the queue and remain there; |
85 | e408eb8a | Klaus Aehlig | however, it is not executed. If an opcode is currently running, |
86 | e408eb8a | Klaus Aehlig | it continues, but the next opcode will not be started. For a paused |
87 | e408eb8a | Klaus Aehlig | job all locks it might have acquired will be released as soon as |
88 | e408eb8a | Klaus Aehlig | possible, at the latest when the currently running opcode has |
89 | e408eb8a | Klaus Aehlig | finished. The job queue will take care of this. |
90 | e408eb8a | Klaus Aehlig | - REJECT. The job is rejected. If it is already in the queue, |
91 | e408eb8a | Klaus Aehlig | it will be marked as cancelled. |
92 | e408eb8a | Klaus Aehlig | - CONTINUE. The filtering continues processing with the next |
93 | e408eb8a | Klaus Aehlig | rule. Such a rule will never have any direct or indirect effect, |
94 | e408eb8a | Klaus Aehlig | but it can serve as documentation for a "normally present, but |
95 | e408eb8a | Klaus Aehlig | currently disabled" rule. |
96 | e408eb8a | Klaus Aehlig | |
97 | e408eb8a | Klaus Aehlig | - A reason trail, in the same format as reason trails for opcodes. |
98 | e408eb8a | Klaus Aehlig | This allows to find out, which maintenance (or other reason) caused |
99 | e408eb8a | Klaus Aehlig | the addition of this filter rule. |
100 | e408eb8a | Klaus Aehlig | |
101 | e408eb8a | Klaus Aehlig | Predicates available for the filter rules |
102 | e408eb8a | Klaus Aehlig | ----------------------------------------- |
103 | e408eb8a | Klaus Aehlig | |
104 | e408eb8a | Klaus Aehlig | A predicate is a list, with the first element being the name of the |
105 | e408eb8a | Klaus Aehlig | predicate and the rest being parameters suitable for that predicate. |
106 | e408eb8a | Klaus Aehlig | In most cases, the name of the predicate will be a field of a job, |
107 | e408eb8a | Klaus Aehlig | and there will be a single parameter, which is a boolean expression |
108 | e408eb8a | Klaus Aehlig | (``filter``) in the sense |
109 | e408eb8a | Klaus Aehlig | of the Ganeti query language. However, no assumption should be made |
110 | e408eb8a | Klaus Aehlig | that all predicates are of this shape. More predicates may be added |
111 | e408eb8a | Klaus Aehlig | in the future. |
112 | e408eb8a | Klaus Aehlig | |
113 | e408eb8a | Klaus Aehlig | - ``jobid``. Only parameter is a boolean expression. For this expression, |
114 | e408eb8a | Klaus Aehlig | there is only one field available, ``id``, which represents the id the job to be |
115 | e408eb8a | Klaus Aehlig | filtered. In all value positions, the string ``watermark`` will be |
116 | e408eb8a | Klaus Aehlig | replaced by the value of the watermark. |
117 | e408eb8a | Klaus Aehlig | |
118 | e408eb8a | Klaus Aehlig | - ``opcode``. Only parameter is boolean expresion. For this expression, ``OP_ID`` |
119 | e408eb8a | Klaus Aehlig | and all other fields present in the opcode are available. This predicate |
120 | e408eb8a | Klaus Aehlig | will hold true, if the expression is true for at least one opcode in |
121 | e408eb8a | Klaus Aehlig | the job. |
122 | e408eb8a | Klaus Aehlig | |
123 | e408eb8a | Klaus Aehlig | - ``reason``. Only parameter is a boolean expression. For this expression, the three |
124 | e408eb8a | Klaus Aehlig | fields ``source``, ``reason``, ``timestamp`` of reason trail entries |
125 | e408eb8a | Klaus Aehlig | are available. This predicate is true, if one of the entries of one |
126 | e408eb8a | Klaus Aehlig | of the opcodes in this job satisfies the expression. |
127 | e408eb8a | Klaus Aehlig | |
128 | e408eb8a | Klaus Aehlig | |
129 | e408eb8a | Klaus Aehlig | Examples |
130 | e408eb8a | Klaus Aehlig | ======== |
131 | e408eb8a | Klaus Aehlig | |
132 | e408eb8a | Klaus Aehlig | Draining the queue. |
133 | e408eb8a | Klaus Aehlig | :: |
134 | e408eb8a | Klaus Aehlig | |
135 | e408eb8a | Klaus Aehlig | {'priority': 0, |
136 | e408eb8a | Klaus Aehlig | 'predicates': [['jobid', ['>', 'id', 'watermark']]], |
137 | e408eb8a | Klaus Aehlig | 'action': 'REJECT'} |
138 | e408eb8a | Klaus Aehlig | |
139 | e408eb8a | Klaus Aehlig | Soft draining could be achieved by replacing ``REJECT`` by ``PAUSE`` in the |
140 | e408eb8a | Klaus Aehlig | above example. |
141 | e408eb8a | Klaus Aehlig | |
142 | e408eb8a | Klaus Aehlig | Pausing all new jobs not belonging to a specific maintenance. |
143 | e408eb8a | Klaus Aehlig | :: |
144 | e408eb8a | Klaus Aehlig | |
145 | e408eb8a | Klaus Aehlig | {'priority': 1, |
146 | e408eb8a | Klaus Aehlig | 'predicates': [['jobid', ['>', 'id', 'watermark']], |
147 | e408eb8a | Klaus Aehlig | ['reason', ['!', ['=~', 'reason', 'maintenance pink bunny']]]], |
148 | e408eb8a | Klaus Aehlig | 'action': 'PAUSE'} |
149 | e408eb8a | Klaus Aehlig | |
150 | e408eb8a | Klaus Aehlig | Canceling all queued instance creations and disallowing new such jobs. |
151 | e408eb8a | Klaus Aehlig | :: |
152 | e408eb8a | Klaus Aehlig | |
153 | e408eb8a | Klaus Aehlig | {'priority': 1, |
154 | e408eb8a | Klaus Aehlig | 'predicates': [['opcode', ['=', 'OP_ID', 'OP_INSTANCE_CREATE']]], |
155 | e408eb8a | Klaus Aehlig | 'action': 'REJECT'} |
156 | e408eb8a | Klaus Aehlig | |
157 | e408eb8a | Klaus Aehlig | |
158 | e408eb8a | Klaus Aehlig | |
159 | e408eb8a | Klaus Aehlig | Interface |
160 | e408eb8a | Klaus Aehlig | ========= |
161 | e408eb8a | Klaus Aehlig | |
162 | e408eb8a | Klaus Aehlig | Since queue control is intended to be used by external maintenance-handling |
163 | e408eb8a | Klaus Aehlig | tools as well, the primary interface for manipulating queue filters is the |
164 | e408eb8a | Klaus Aehlig | :doc:`rapi`. For convenience, a command-line interface will be added as well. |
165 | e408eb8a | Klaus Aehlig | |
166 | e408eb8a | Klaus Aehlig | The following resources will be added. |
167 | e408eb8a | Klaus Aehlig | |
168 | e408eb8a | Klaus Aehlig | - /2/filters/ |
169 | e408eb8a | Klaus Aehlig | |
170 | e408eb8a | Klaus Aehlig | - GET returns the list of all currently set filters |
171 | e408eb8a | Klaus Aehlig | |
172 | e408eb8a | Klaus Aehlig | - POST adds a new filter |
173 | e408eb8a | Klaus Aehlig | |
174 | e408eb8a | Klaus Aehlig | - /2/filters/[uuid] |
175 | e408eb8a | Klaus Aehlig | |
176 | e408eb8a | Klaus Aehlig | - GET returns the description of the specified filter |
177 | e408eb8a | Klaus Aehlig | |
178 | e408eb8a | Klaus Aehlig | - DELETE removes the specified filter |
179 | e408eb8a | Klaus Aehlig | |
180 | e408eb8a | Klaus Aehlig | - PUT replaces the specified filter rule, or creates it, |
181 | e408eb8a | Klaus Aehlig | if it doesn't exist already. |
182 | e408eb8a | Klaus Aehlig | |
183 | e408eb8a | Klaus Aehlig | Security considerations |
184 | e408eb8a | Klaus Aehlig | ======================= |
185 | e408eb8a | Klaus Aehlig | |
186 | e408eb8a | Klaus Aehlig | Filtering of jobs is not a security feature. It merely serves the purpose |
187 | e408eb8a | Klaus Aehlig | of coordinating efforts and avoiding accidental conflicting |
188 | e408eb8a | Klaus Aehlig | jobs. Everybody with appropriate credentials can modify the filter |
189 | e408eb8a | Klaus Aehlig | rules, not just the originator of a rule. To avoid accidental |
190 | e408eb8a | Klaus Aehlig | lock-out, requests modifying the queue are executed directly and not |
191 | e408eb8a | Klaus Aehlig | going through the queue themselves. |