Statistics
| Branch: | Tag: | Revision:

root / doc / design-optables.rst @ 56c934da

History | View | Annotate | Download (7.2 kB)

1 e408eb8a Klaus Aehlig
==========================================
2 e408eb8a Klaus Aehlig
Filtering of jobs for the Ganeti job queue
3 e408eb8a Klaus Aehlig
==========================================
4 e408eb8a Klaus Aehlig
5 e408eb8a Klaus Aehlig
.. contents:: :depth: 4
6 e408eb8a Klaus Aehlig
7 e408eb8a Klaus Aehlig
This is a design document detailing the semantics of the fine-grained control
8 e408eb8a Klaus Aehlig
of jobs in Ganeti. For the implementation there will be a separate
9 e408eb8a Klaus Aehlig
design document that also describes the vision for the Ganeti daemon
10 e408eb8a Klaus Aehlig
structure.
11 e408eb8a Klaus Aehlig
12 e408eb8a Klaus Aehlig
13 e408eb8a Klaus Aehlig
Current state and shortcomings
14 e408eb8a Klaus Aehlig
==============================
15 e408eb8a Klaus Aehlig
16 e408eb8a Klaus Aehlig
Control of the Ganeti job queue is quite limited. There is a single
17 e408eb8a Klaus Aehlig
status bit, the "drained flag". If set, no new jobs are accepted to
18 e408eb8a Klaus Aehlig
the queue. This is too coarse for some use cases.
19 e408eb8a Klaus Aehlig
20 e408eb8a Klaus Aehlig
- The queue might be required to be drained for several reasons,
21 e408eb8a Klaus Aehlig
  initiated by different persons or automatic programs. Each one
22 e408eb8a Klaus Aehlig
  should be able to indicate that his reasons for draining are over
23 e408eb8a Klaus Aehlig
  without affecting the others.
24 e408eb8a Klaus Aehlig
25 e408eb8a Klaus Aehlig
- There is no support for partial drains. For example, one might want
26 e408eb8a Klaus Aehlig
  to allow all jobs belonging to a manual (or externally coordinated)
27 e408eb8a Klaus Aehlig
  maintenance, while disallowing all other jobs.
28 e408eb8a Klaus Aehlig
29 e408eb8a Klaus Aehlig
- There is no support for blocking jobs by their op-codes, e.g.,
30 e408eb8a Klaus Aehlig
  disallowing all jobs that bring new instances to a cluster. This might
31 e408eb8a Klaus Aehlig
  be part of a maintenance preparation.
32 e408eb8a Klaus Aehlig
33 e408eb8a Klaus Aehlig
- There is no support for a soft version of draining, where all
34 e408eb8a Klaus Aehlig
  jobs currently in the queue are finished, while new jobs entering
35 e408eb8a Klaus Aehlig
  the queue are delayed until the drain is over.
36 e408eb8a Klaus Aehlig
37 e408eb8a Klaus Aehlig
38 e408eb8a Klaus Aehlig
Proposed changes
39 e408eb8a Klaus Aehlig
================
40 e408eb8a Klaus Aehlig
41 e408eb8a Klaus Aehlig
We propose to add filters on the job queue. These will be part of the
42 e408eb8a Klaus Aehlig
configuration and as such are persisted with it. Conceptionally, the
43 e408eb8a Klaus Aehlig
filters are always processed when a job enters the queue and while it
44 e408eb8a Klaus Aehlig
is still in the queue. Of course, in the implementation, reevaluation
45 e408eb8a Klaus Aehlig
is only carried out, if something could make the result change, e.g.,
46 e408eb8a Klaus Aehlig
a new job is entered to the queue, or the filter rules are changed.
47 e408eb8a Klaus Aehlig
There is no distinction between filter processing when a job is about
48 e408eb8a Klaus Aehlig
to enter the queue and while it is in the queue, as this can be
49 e408eb8a Klaus Aehlig
expressed by the filter rules themselves (see predicates below).
50 e408eb8a Klaus Aehlig
51 e408eb8a Klaus Aehlig
Format of a Filter rule
52 e408eb8a Klaus Aehlig
-----------------------
53 e408eb8a Klaus Aehlig
54 e408eb8a Klaus Aehlig
Filter rules are given by the following data.
55 e408eb8a Klaus Aehlig
56 e408eb8a Klaus Aehlig
- A UUID. This ensures that there can be different filter rules
57 e408eb8a Klaus Aehlig
  that otherwise have all parameters equal. In this way, multiple
58 e408eb8a Klaus Aehlig
  drains for different reasons are possible. The UUID is used to
59 e408eb8a Klaus Aehlig
  address the filter rule, in particular for deletion.
60 e408eb8a Klaus Aehlig
61 e408eb8a Klaus Aehlig
  If no UUID is provided at rule addition, Ganeti will create one.
62 e408eb8a Klaus Aehlig
63 e408eb8a Klaus Aehlig
- The watermark. This is the highest job id ever used, as valid in
64 e408eb8a Klaus Aehlig
  the moment when the filter was added. This data will be added
65 e408eb8a Klaus Aehlig
  automatically upon addition of the filter.
66 e408eb8a Klaus Aehlig
67 e408eb8a Klaus Aehlig
- A priority. This is a non-negative integer. Filters are processed
68 e408eb8a Klaus Aehlig
  in order of increasing priority until a rule applies. While there
69 e408eb8a Klaus Aehlig
  is a well-defined order in which rules of the same priority are
70 e408eb8a Klaus Aehlig
  evaluated (increasing watermark, then the uuid, are taken as tie
71 e408eb8a Klaus Aehlig
  breakers), it is not recommended to have rules of the same priority
72 e408eb8a Klaus Aehlig
  that overlap and have different actions associated.
73 e408eb8a Klaus Aehlig
74 e408eb8a Klaus Aehlig
- A list of predicates. The rule fires, if all of them hold true
75 e408eb8a Klaus Aehlig
  for the job.
76 e408eb8a Klaus Aehlig
77 e408eb8a Klaus Aehlig
- An action. For the time being, one of the following, but more
78 e408eb8a Klaus Aehlig
  actions might be added in the future (in particular, future
79 e408eb8a Klaus Aehlig
  implementations might add an action making filtering continue with
80 e408eb8a Klaus Aehlig
  a different filter chain).
81 e408eb8a Klaus Aehlig
82 e408eb8a Klaus Aehlig
  - ACCEPT. The job will be accepted; no further filter rules
83 e408eb8a Klaus Aehlig
    are applied.
84 e408eb8a Klaus Aehlig
  - PAUSE. The job will be accepted to the queue and remain there;
85 e408eb8a Klaus Aehlig
    however, it is not executed. If an opcode is currently running,
86 e408eb8a Klaus Aehlig
    it continues, but the next opcode will not be started. For a paused
87 e408eb8a Klaus Aehlig
    job all locks it might have acquired will be released as soon as
88 e408eb8a Klaus Aehlig
    possible, at the latest when the currently running opcode has
89 e408eb8a Klaus Aehlig
    finished. The job queue will take care of this.
90 e408eb8a Klaus Aehlig
  - REJECT. The job is rejected. If it is already in the queue,
91 e408eb8a Klaus Aehlig
    it will be marked as cancelled.
92 e408eb8a Klaus Aehlig
  - CONTINUE. The filtering continues processing with the next
93 e408eb8a Klaus Aehlig
    rule. Such a rule will never have any direct or indirect effect,
94 e408eb8a Klaus Aehlig
    but it can serve as documentation for a "normally present, but
95 e408eb8a Klaus Aehlig
    currently disabled" rule.
96 e408eb8a Klaus Aehlig
97 e408eb8a Klaus Aehlig
- A reason trail, in the same format as reason trails for opcodes. 
98 e408eb8a Klaus Aehlig
  This allows to find out, which maintenance (or other reason) caused
99 e408eb8a Klaus Aehlig
  the addition of this filter rule.
100 e408eb8a Klaus Aehlig
101 e408eb8a Klaus Aehlig
Predicates available for the filter rules
102 e408eb8a Klaus Aehlig
-----------------------------------------
103 e408eb8a Klaus Aehlig
104 e408eb8a Klaus Aehlig
A predicate is a list, with the first element being the name of the
105 e408eb8a Klaus Aehlig
predicate and the rest being parameters suitable for that predicate.
106 e408eb8a Klaus Aehlig
In most cases, the name of the predicate will be a field of a job,
107 e408eb8a Klaus Aehlig
and there will be a single parameter, which is a boolean expression
108 e408eb8a Klaus Aehlig
(``filter``) in the sense
109 e408eb8a Klaus Aehlig
of the Ganeti query language. However, no assumption should be made
110 e408eb8a Klaus Aehlig
that all predicates are of this shape. More predicates may be added
111 e408eb8a Klaus Aehlig
in the future.
112 e408eb8a Klaus Aehlig
113 e408eb8a Klaus Aehlig
- ``jobid``. Only parameter is a boolean expression. For this expression,
114 e408eb8a Klaus Aehlig
  there is only one field available, ``id``, which represents the id the job to be
115 e408eb8a Klaus Aehlig
  filtered. In all value positions, the string ``watermark`` will be
116 e408eb8a Klaus Aehlig
  replaced by the value of the watermark.
117 e408eb8a Klaus Aehlig
118 e408eb8a Klaus Aehlig
- ``opcode``. Only parameter is boolean expresion. For this expression, ``OP_ID``
119 e408eb8a Klaus Aehlig
  and all other fields present in the opcode are available. This predicate
120 e408eb8a Klaus Aehlig
  will hold true, if the expression is true for at least one opcode in
121 e408eb8a Klaus Aehlig
  the job.
122 e408eb8a Klaus Aehlig
123 e408eb8a Klaus Aehlig
- ``reason``. Only parameter is a boolean expression. For this expression, the three
124 e408eb8a Klaus Aehlig
  fields ``source``, ``reason``, ``timestamp`` of reason trail entries
125 e408eb8a Klaus Aehlig
  are available. This predicate is true, if one of the entries of one
126 e408eb8a Klaus Aehlig
  of the opcodes in this job satisfies the expression.
127 e408eb8a Klaus Aehlig
128 e408eb8a Klaus Aehlig
129 e408eb8a Klaus Aehlig
Examples
130 e408eb8a Klaus Aehlig
========
131 e408eb8a Klaus Aehlig
132 e408eb8a Klaus Aehlig
Draining the queue.
133 e408eb8a Klaus Aehlig
::
134 e408eb8a Klaus Aehlig
135 e408eb8a Klaus Aehlig
   {'priority': 0,
136 e408eb8a Klaus Aehlig
    'predicates': [['jobid', ['>', 'id', 'watermark']]],
137 e408eb8a Klaus Aehlig
    'action': 'REJECT'}
138 e408eb8a Klaus Aehlig
139 e408eb8a Klaus Aehlig
Soft draining could be achieved by replacing ``REJECT`` by ``PAUSE`` in the
140 e408eb8a Klaus Aehlig
above example.
141 e408eb8a Klaus Aehlig
142 e408eb8a Klaus Aehlig
Pausing all new jobs not belonging to a specific maintenance.
143 e408eb8a Klaus Aehlig
::
144 e408eb8a Klaus Aehlig
145 e408eb8a Klaus Aehlig
   {'priority': 1,
146 e408eb8a Klaus Aehlig
    'predicates': [['jobid', ['>', 'id', 'watermark']],
147 e408eb8a Klaus Aehlig
                   ['reason', ['!', ['=~', 'reason', 'maintenance pink bunny']]]],
148 e408eb8a Klaus Aehlig
    'action': 'PAUSE'}
149 e408eb8a Klaus Aehlig
150 e408eb8a Klaus Aehlig
Canceling all queued instance creations and disallowing new such jobs.
151 e408eb8a Klaus Aehlig
::
152 e408eb8a Klaus Aehlig
153 e408eb8a Klaus Aehlig
  {'priority': 1,
154 e408eb8a Klaus Aehlig
   'predicates': [['opcode', ['=', 'OP_ID', 'OP_INSTANCE_CREATE']]],
155 e408eb8a Klaus Aehlig
   'action': 'REJECT'}
156 e408eb8a Klaus Aehlig
157 e408eb8a Klaus Aehlig
158 e408eb8a Klaus Aehlig
159 e408eb8a Klaus Aehlig
Interface
160 e408eb8a Klaus Aehlig
=========
161 e408eb8a Klaus Aehlig
162 e408eb8a Klaus Aehlig
Since queue control is intended to be used by external maintenance-handling
163 e408eb8a Klaus Aehlig
tools as well, the primary interface for manipulating queue filters is the
164 e408eb8a Klaus Aehlig
:doc:`rapi`. For convenience, a command-line interface will be added as well.
165 e408eb8a Klaus Aehlig
166 e408eb8a Klaus Aehlig
The following resources will be added.
167 e408eb8a Klaus Aehlig
168 e408eb8a Klaus Aehlig
- /2/filters/
169 e408eb8a Klaus Aehlig
170 e408eb8a Klaus Aehlig
  - GET returns the list of all currently set filters
171 e408eb8a Klaus Aehlig
172 e408eb8a Klaus Aehlig
  - POST adds a new filter
173 e408eb8a Klaus Aehlig
174 e408eb8a Klaus Aehlig
- /2/filters/[uuid]
175 e408eb8a Klaus Aehlig
176 e408eb8a Klaus Aehlig
  - GET returns the description of the specified filter
177 e408eb8a Klaus Aehlig
178 e408eb8a Klaus Aehlig
  - DELETE removes the specified filter
179 e408eb8a Klaus Aehlig
180 e408eb8a Klaus Aehlig
  - PUT replaces the specified filter rule, or creates it,
181 e408eb8a Klaus Aehlig
    if it doesn't exist already.
182 e408eb8a Klaus Aehlig
183 e408eb8a Klaus Aehlig
Security considerations
184 e408eb8a Klaus Aehlig
=======================
185 e408eb8a Klaus Aehlig
186 e408eb8a Klaus Aehlig
Filtering of jobs is not a security feature. It merely serves the purpose
187 e408eb8a Klaus Aehlig
of coordinating efforts and avoiding accidental conflicting
188 e408eb8a Klaus Aehlig
jobs. Everybody with appropriate credentials can modify the filter
189 e408eb8a Klaus Aehlig
rules, not just the originator of a rule. To avoid accidental
190 e408eb8a Klaus Aehlig
lock-out, requests modifying the queue are executed directly and not
191 e408eb8a Klaus Aehlig
going through the queue themselves.