5 .. contents:: :depth: 4
8 Current state and shortcomings
9 ==============================
11 Queries are used to retrieve information about the cluster, e.g. a list
12 of instances or nodes. For historical reasons they use a simple data
13 structure for their result. The client submits the fields it would like
14 to receive and the query returns a list for each item (instance, node,
15 etc.) available. Each item consists of another list representing the
18 This data structure has a few drawbacks. It can't associate a status
19 (e.g. “node offline”) with fields as using special values can lead to
20 ambiguities. Additionally it can't mark fields as “not found” as the
21 list of returned columns must match the fields requested.
25 >>> cli.GetClient().QueryNodes([], ["name", "pip", "mfree"], False)
27 ['node1.example.com', '192.0.2.18', 14800],
28 ['node2.example.com', '192.0.2.19', 31280]
31 There is no way for clients to determine the list of possible fields,
32 meaning they have to be hardcoded. Selecting unknown fields raises
35 >>> cli.GetClient().QueryNodes([], ["name", "UnknownField"], False)
36 ganeti.errors.OpPrereqError: (u'Unknown output fields selected: UnknownField', u'wrong_input')
38 The client must also know each fields' kind, that is whether a field is
39 numeric, boolean, describes a storage size, etc. Centralizing this
40 information in one place, the master daemon, is desirable.
46 The current query result format can not be changed as it's being used in
47 various places. Changing the format from one Ganeti version to another
48 would cause too much disruption. For this reason the ability to
49 explicitly request a new result format must be added while the old
50 format stays the default.
52 The implementation of query filters is planned for the future. To avoid
53 having to change the calls again, a (hopefully) future-compatible
54 interface will be implemented now.
56 In Python code, the objects described below will be implemented using
57 subclasses of ``objects.ConfigObject``, providing existing facilities
63 As it turned out, only very few fields for instances used regular
64 expressions, all of which can easily be turned into static field names.
65 Therefore their use in field names is dropped. Reasons:
67 - When regexps are used and a field name is not listed as a simple
68 string in the field dictionary, all keys in the field dictionary have
69 to be checked whether they're a regular expression object and if so,
70 matched (see ``utils.FindMatch``).
71 - Code becomes simpler. There would be no need anymore to care about
72 regular expressions as field names—they'd all be simple strings, even
73 if there are many more. The list of field names would be static once
74 built at module-load time.
75 - There's the issue of formatting titles for the clients. Should it be
76 done in the server? In the client? The field definition's title would
77 contain backreferences to the regexp groups in the field name
78 (``re.MatchObject.expand`` can be used). With just strings, the field
79 definitions can be passed directly to the client. They're static.
80 - Only a side note: In the memory consumed for 1'000
81 ``_sre.SRE_Pattern`` objects (as returned by ``re.compile`` for an
82 expression with one group) one can easily store 10'000 strings of the
83 same length (the regexp objects keep the expression string around, so
84 compiling the expression always uses more memory).
92 The proposal is to implement this new interface for the following
111 .. _data-query-request:
116 The request is a dictionary with the following entries:
118 ``what`` (string, required)
119 An :ref:`item type <item-types>`.
120 ``fields`` (list of strings, required)
121 List of names of fields to return. Example::
123 ["name", "mem", "nic0.ip", "disk0.size", "disk1.size"]
125 ``filter`` (optional)
126 This will be used to filter queries. In this implementation only names
127 can be filtered to replace the previous ``names`` parameter to
128 queries. An empty filter (``None``) will return all items. To retrieve
129 specific names, the filter must be specified as follows, with the
130 inner part repeated for each name::
132 ["|", ["=", "name", "node1"], ["=", "name", "node2"], …]
134 Filters consist of S-expressions (``["operator", <operants…>]``) and
135 extensions will be made in the future to allow for more operators and
136 fields. Such extensions might include a Python-style "in" operator,
137 but for simplicity only "=" is supported in this implementation.
139 To reiterate: Filters for this implementation must consist of exactly
140 one OR expression (``["|", …]``) and one or more name equality filters
141 (``["=", "name", "…"]``).
143 Support for synchronous queries, currently available in the interface
144 but disabled in the master daemon, will be dropped. Direct calls to
145 opcodes have to be used instead.
147 .. _data-query-response:
152 The result is a dictionary with the following entries:
154 ``fields`` (list of :ref:`field definitions <field-def>`)
155 In-order list of a :ref:`field definition <field-def>` for each
156 requested field, unknown fields are returned with the kind
157 ``unknown``. Length must be equal to number of requested fields.
158 ``data`` (list of lists of tuples)
159 List of lists, one list for each item found. Each item's list must
160 have one entry for each field listed in ``fields`` (meaning their
161 length is equal). Each field entry is a tuple of ``(status, value)``.
162 ``status`` must be one of the following values:
165 Value is available and matches the kind in the :ref:`field
166 definition <field-def>`.
167 Unknown field (numeric 1)
168 Field for this column is not known. Value must be ``None``.
170 Exact meaning depends on query, e.g. node is unreachable or marked
171 offline. Value must be ``None``.
172 Value unavailable for item (numeric 3)
173 Used if, for example, NIC 3 is requested for an instance with only
174 one network interface. Value must be ``None``.
175 Resource offline (numeric 4)
176 Used if resource is marked offline. Value must be ``None``.
178 Example response after requesting the fields ``name``, ``mfree``,
179 ``xyz``, ``mtotal``, ``nic0.ip``, ``nic1.ip`` and ``nic2.ip``::
183 { "name": "name", "title": "Name", "kind": "text", },
184 { "name": "mfree", "title": "MemFree", "kind": "unit", },
186 { "name": "xyz", "title": None, "kind": "unknown", },
187 { "name": "mtotal", "title": "MemTotal", "kind": "unit", },
188 { "name": "nic0.ip", "title": "Nic.IP/0", "kind": "text", },
189 { "name": "nic1.ip", "title": "Nic.IP/1", "kind": "text", },
190 { "name": "nic2.ip", "title": "Nic.IP/2", "kind": "text", },
194 [(0, "node1"), (0, 128), (1, None), (0, 4096),
195 (0, "192.0.2.1"), (0, "192.0.2.2"), (3, None)],
196 [(0, "node2"), (0, 96), (1, None), (0, 5000),
197 (0, "192.0.2.21"), (0, "192.0.2.39"), (3, "192.0.2.90")],
198 # Node not available, can't get "mfree" or "mtotal"
199 [(0, "node3"), (2, None), (1, None), (2, None),
200 (0, "192.0.2.30"), (3, None), (3, None)],
209 .. _fields-query-request:
214 The request is a dictionary with the following entries:
216 ``what`` (string, required)
217 An :ref:`item type <item-types>`.
218 ``fields`` (list of strings, optional)
219 List of names of fields to return. If not set, all fields are
222 ["name", "mem", "nic0.ip", "disk0.size", "disk1.size"]
224 .. _fields-query-response:
229 The result is a dictionary with the following entries:
231 ``fields`` (list of :ref:`field definitions <field-def>`)
232 List of a :ref:`field definition <field-def>` for each field. If
233 ``fields`` was set in the request and contained an unknown field, it
234 is returned as type ``unknown``.
240 { "name": "name", "title": "Name", "kind": "text", },
241 { "name": "mfree", "title": "MemFree", "kind": "unit", },
242 { "name": "mtotal", "title": "MemTotal", "kind": "unit", },
243 { "name": "nic0.ip", "title": "Nic.IP/0", "kind": "text", },
244 { "name": "nic1.ip", "title": "Nic.IP/1", "kind": "text", },
245 { "name": "nic2.ip", "title": "Nic.IP/2", "kind": "text", },
246 { "name": "nic3.ip", "title": "Nic.IP/3", "kind": "text", },
248 { "name": "disk0.size", "title": "Disk.Size/0", "kind": "unit", },
249 { "name": "disk1.size", "title": "Disk.Size/1", "kind": "unit", },
250 { "name": "disk2.size", "title": "Disk.Size/2", "kind": "unit", },
251 { "name": "disk3.size", "title": "Disk.Size/3", "kind": "unit", },
261 A field definition is a dictionary with the following entries:
264 Field name. Must only contain characters matching ``[a-z0-9/._]``.
266 Human-readable title to use in output. Must not contain whitespace.
268 Field type, one of the following:
279 Numeric, in megabytes
281 Unix timestamp in seconds since the epoch
283 Free-form type, depending on query
285 More types can be added in the future, so clients should default to
286 formatting any unknown types the same way as "other", which should be
287 a string representation in most cases.
290 Human-readable description. Must start with uppercase character and
291 must not end with punctuation or contain newlines.
293 .. TODO: Investigate whether there are fields with floating point
296 Example 1 (item name)::
304 Example 2 (free memory)::
312 Example 3 (list of primary instances)::
316 "title": "PrimaryInstances",
320 .. _old-result-format:
325 To limit the amount of code necessary, the :ref:`new result format
326 <data-query-response>` will be converted for clients calling the old
327 methods. Unavailable values are set to ``None``. If unknown fields were
328 requested, the whole query fails as the client expects exactly the
336 Currently query calls take a number of parameters, e.g. names, fields
337 and whether to use locking. These will continue to work and return the
338 :ref:`old result format <old-result-format>`. Only clients using the
339 new calls described below will be able to make use of new features such
340 as filters. Two new calls are introduced:
343 Execute a query on items, optionally filtered. Takes a single
344 parameter, a :ref:`query object <data-query-request>` encoded as a
345 dictionary and returns a :ref:`data query response
346 <data-query-response>`.
348 Return list of supported fields as :ref:`field definitions
349 <field-def>`. Takes a single parameter, a :ref:`fields query object
350 <fields-query-request>` encoded as a dictionary and returns a
351 :ref:`fields query response <fields-query-response>`.
357 The LUXI API is more or less mapped directly into Python. In addition to
358 the existing stub functions new ones will be added for the new query
364 The RAPI interface already returns dictionaries for each item, but to
365 not break compatibility no changes should be made to the structure (e.g.
366 to include field definitions). The proposal here is to add a new
367 parameter to allow clients to execute the requests described in this
368 proposal directly and to receive the unmodified result. The new formats
369 are a lot more verbose, flexible and extensible.
376 Command line programs might have difficulties to display the verbose
377 status data to the user. There are several options:
379 - Use colours to indicate missing values
380 - Display status as value in parentheses, e.g. "(unavailable)"
381 - Hide unknown columns from the result table and print a warning
382 - Exit with non-zero code to indicate failures and/or missing data
384 Some are better for interactive usage, some better for use by other
385 programs. It is expected that a combination will be used. The column
386 separator (``--separator=…``) can be used to differentiate between
387 interactive and programmatic usage.
390 Other discussed solutions
391 -------------------------
393 Another solution discussed was to add an additional column for each
394 non-static field containing the status. Clients interested in the status
395 could explicitly query for it.
397 .. vim: set textwidth=72 :