Revision 3301805f
b/doc/design-monitoring-agent.rst | ||
---|---|---|
24 | 24 |
monitoring system. Such status page will be exported on a network port |
25 | 25 |
and will be encoded in JSON (simple text) over HTTP. |
26 | 26 |
|
27 |
The choice of json is obvious as we already depend on it in Ganeti and
|
|
27 |
The choice of JSON is obvious as we already depend on it in Ganeti and
|
|
28 | 28 |
thus we don't need to add extra libraries to use it, as opposed to what |
29 | 29 |
would happen for XML or some other markup format. |
30 | 30 |
|
... | ... | |
48 | 48 |
- Node OS resources report (memory, CPU, network interfaces) |
49 | 49 |
- Information from a plugin system |
50 | 50 |
|
51 |
Format of the query |
|
52 |
------------------- |
|
53 |
|
|
54 |
The query will be an HTTP GET request on a particular port. At the |
|
55 |
beginning it will only be possible to query the full status report. |
|
56 |
|
|
57 |
Format of the report |
|
58 |
-------------------- |
|
59 |
|
|
60 |
The report of the will be in JSON format, and it will present an array |
|
61 |
of report objects. |
|
62 |
Each report object will be produced by a specific data collector. |
|
63 |
Each report object includes some mandatory fields, to be provided by all |
|
64 |
the data collectors: |
|
65 |
|
|
66 |
``name`` |
|
67 |
The name of the data collector that produced this part of the report. |
|
68 |
It is supposed to be unique inside a report. |
|
69 |
|
|
70 |
``version`` |
|
71 |
The version of the data collector that produces this part of the |
|
72 |
report. Built-in data collectors (as opposed to those implemented as |
|
73 |
plugins) should have "B" as the version number. |
|
74 |
|
|
75 |
``formatVersion`` |
|
76 |
The format of what is represented in the "data" field for each data |
|
77 |
collector might change over time. Every time this happens, the |
|
78 |
format_version should be changed, so that who reads the report knows |
|
79 |
what format to expect, and how to correctly interpret it. |
|
80 |
|
|
81 |
``timestamp`` |
|
82 |
The time when the reported data were gathered. Is has to be expressed |
|
83 |
in nanoseconds since the unix epoch (0:00:00 January 01, 1970). If not |
|
84 |
enough precision is available (or needed) it can be padded with |
|
85 |
zeroes. If a report object needs multiple timestamps, it can add more |
|
86 |
and/or override this one inside its own "data" section. |
|
87 |
|
|
88 |
``category`` |
|
89 |
A collector can belong to a given category of collectors (e.g.: storage |
|
90 |
collectors, daemon collector). This means that it will have to provide a |
|
91 |
minumum set of prescribed fields, as documented for each category. |
|
92 |
This field will contain the name of the category the collector belongs to, |
|
93 |
if any, or just the ``null`` value. |
|
94 |
|
|
95 |
``kind`` |
|
96 |
Two kinds of collectors are possible: |
|
97 |
`Performance reporting collectors`_ and `Status reporting collectors`_. |
|
98 |
The respective paragraphs will describe them and the value of this field. |
|
99 |
|
|
100 |
``data`` |
|
101 |
This field contains all the data generated by the specific data collector, |
|
102 |
in its own independently defined format. The monitoring agent could check |
|
103 |
this syntactically (according to the JSON specifications) but not |
|
104 |
semantically. |
|
105 |
|
|
106 |
Here follows a minimal example of a report:: |
|
107 |
|
|
108 |
[ |
|
109 |
{ |
|
110 |
"name" : "TheCollectorIdentifier", |
|
111 |
"version" : "1.2", |
|
112 |
"formatVersion" : 1, |
|
113 |
"timestamp" : 1351607182000000000, |
|
114 |
"category" : null, |
|
115 |
"kind" : 0, |
|
116 |
"data" : { "plugin_specific_data" : "go_here" } |
|
117 |
}, |
|
118 |
{ |
|
119 |
"name" : "AnotherDataCollector", |
|
120 |
"version" : "B", |
|
121 |
"formatVersion" : 7, |
|
122 |
"timestamp" : 1351609526123854000, |
|
123 |
"category" : "storage", |
|
124 |
"kind" : 1, |
|
125 |
"data" : { "status" : { "code" : 1, |
|
126 |
"message" : "Error on disk 2" |
|
127 |
}, |
|
128 |
"plugin_specific" : "data", |
|
129 |
"some_late_data" : { "timestamp" : 1351609526123942720, |
|
130 |
... |
|
131 |
} |
|
132 |
} |
|
133 |
} |
|
134 |
] |
|
135 |
|
|
136 |
Performance reporting collectors |
|
137 |
++++++++++++++++++++++++++++++++ |
|
138 |
|
|
139 |
These collectors only provide data about some component of the system, without |
|
140 |
giving any interpretation over their meaning. |
|
141 |
|
|
142 |
The value of the ``kind`` field of the report will be ``0``. |
|
143 |
|
|
144 |
Status reporting collectors |
|
145 |
+++++++++++++++++++++++++++ |
|
146 |
|
|
147 |
These collectors will provide information about the status of some |
|
148 |
component of ganeti, or managed by ganeti. |
|
149 |
|
|
150 |
The value of their ``kind`` field will be ``1``. |
|
151 |
|
|
152 |
The rationale behind this kind of collectors is that there are some situations |
|
153 |
where exporting data about the underlying subsystems would expose potential |
|
154 |
issues. But if Ganeti itself is able (and going) to fix the problem, conflicts |
|
155 |
might arise between Ganeti and something/somebody else trying to fix the same |
|
156 |
problem. |
|
157 |
Also, some external monitoring systems might not be aware of the internals of a |
|
158 |
particular subsystem (e.g.: DRBD) and might only exploit the high level |
|
159 |
response of its data collector, alerting an administrator if anything is wrong. |
|
160 |
Still, completely hiding the underlying data is not a good idea, as they might |
|
161 |
still be of use in some cases. So status reporting plugins will provide two |
|
162 |
output modes: one just exporting a high level information about the status, |
|
163 |
and one also exporting all the data they gathered. |
|
164 |
The default output mode will be the status-only one. Through a command line |
|
165 |
parameter (for stand-alone data collectors) or through the HTTP request to the |
|
166 |
monitoring agent |
|
167 |
(when collectors are executed as part of it) the verbose output mode providing |
|
168 |
all the data can be selected. |
|
169 |
|
|
170 |
When exporting just the status each status reporting collector will provide, |
|
171 |
in its ``data`` section, at least the following field: |
|
172 |
|
|
173 |
``status`` |
|
174 |
summarizes the status of the component being monitored and consists of two |
|
175 |
subfields: |
|
176 |
|
|
177 |
``code`` |
|
178 |
It assumes a numeric value, encoded in such a way to allow using a bitset |
|
179 |
to easily distinguish which states are currently present in the whole cluster. |
|
180 |
If the bitwise OR of all the ``status`` fields is 0, the cluster is |
|
181 |
completely healty. |
|
182 |
The status codes are as follows: |
|
183 |
|
|
184 |
``0`` |
|
185 |
The collector can determine that everything is working as |
|
186 |
intended. |
|
187 |
|
|
188 |
``1`` |
|
189 |
Something is temporarily wrong but it is being automatically fixed by |
|
190 |
Ganeti. |
|
191 |
There is no need of external intervention. |
|
192 |
|
|
193 |
``2`` |
|
194 |
The collector can determine that something is wrong and Ganeti has no |
|
195 |
way to fix it autonomously. External intervention is required. |
|
196 |
|
|
197 |
``4`` |
|
198 |
The collector has failed to understand whether the status is good or |
|
199 |
bad. Further analysis is required. Interpret this status as a |
|
200 |
potentially dangerous situation. |
|
201 |
|
|
202 |
``message`` |
|
203 |
A message to better explain the reason of the status. |
|
204 |
The exact format of the message string is data collector dependent. |
|
205 |
|
|
206 |
The field is mandatory, but the content can be ``null`` if the code is |
|
207 |
``0`` (working as intended) or ``1`` (being fixed automatically). |
|
208 |
|
|
209 |
If the status code is ``2``, the message should specify what has gone |
|
210 |
wrong. |
|
211 |
If the status code is ``4``, the message shoud explain why it was not |
|
212 |
possible to determine a proper status. |
|
213 |
|
|
214 |
The ``data`` section will also contain all the fields describing the gathered |
|
215 |
data, according to a collector-specific format. |
|
216 |
|
|
51 | 217 |
Instance status |
52 | 218 |
+++++++++++++++ |
53 | 219 |
|
... | ... | |
75 | 241 |
system, which then will export those themselves. |
76 | 242 |
|
77 | 243 |
Monitoring and auditing systems can then use the reason to understand |
78 |
the cause of an instance status, and they can use the object version to
|
|
244 |
the cause of an instance status, and they can use the timestamp to
|
|
79 | 245 |
understand the freshness of their data even in the absence of an atomic |
80 | 246 |
cross-node reporting: for example if they see an instance "up" on a node |
81 | 247 |
after seeing it running on a previous one, they can compare these values |
... | ... | |
86 | 252 |
upon. |
87 | 253 |
|
88 | 254 |
The instance status will be on each node, for the instances it is |
89 |
primary for and will contain at least: |
|
255 |
primary for, and its ``data`` section of the report will contain a list |
|
256 |
of instances, with at least the following fields for each instance: |
|
257 |
|
|
258 |
``name`` |
|
259 |
The name of the instance. |
|
260 |
|
|
261 |
``uuid`` |
|
262 |
The UUID of the instance (stable on name change). |
|
263 |
|
|
264 |
``admin_state`` |
|
265 |
The status of the instance (up/down/offline) as requested by the admin. |
|
266 |
|
|
267 |
``actual_state`` |
|
268 |
The actual status of the instance. It can be ``up``, ``down``, or |
|
269 |
``hung`` if the instance is up but it appears to be completely stuck. |
|
270 |
|
|
271 |
``uptime`` |
|
272 |
The uptime of the instance (if it is up, "null" otherwise). |
|
273 |
|
|
274 |
``mtime`` |
|
275 |
The timestamp of the last known change to the instance state. |
|
276 |
|
|
277 |
``state_reason`` |
|
278 |
The last known reason for state change, described according to the |
|
279 |
following subfields: |
|
280 |
|
|
281 |
``text`` |
|
282 |
Either a user-provided reason (if any), or the name of the command that |
|
283 |
triggered the state change, as a fallback. |
|
284 |
|
|
285 |
``jobID`` |
|
286 |
The ID of the job that caused the state change. |
|
90 | 287 |
|
91 |
- The instance name |
|
92 |
- The instance UUID (stable on name change) |
|
93 |
- The instance running status (up or down) |
|
94 |
- The uptime, as detected by the hypervisor |
|
95 |
- The timestamp of last known change |
|
96 |
- The timestamp of when the status was last checked (see caching, below) |
|
97 |
- The last known reason for change, if any |
|
288 |
``source`` |
|
289 |
Where the state change was triggered (RAPI, CLI). |
|
98 | 290 |
|
99 |
More information about all the fields and their type will be available |
|
100 |
in the "Format of the report" section. |
|
291 |
``status`` |
|
292 |
It represents the status of the instance, and its format is the same as that |
|
293 |
of the ``status`` field of `Status reporting collectors`_. |
|
294 |
|
|
295 |
Each hypervisor should provide its own instance status data collector, possibly |
|
296 |
with the addition of more, specific, fields. |
|
297 |
The ``category`` field of all of them will be ``instance``. |
|
298 |
The ``kind`` field will be ``1``. |
|
101 | 299 |
|
102 | 300 |
Note that as soon as a node knows it's not the primary anymore for an |
103 | 301 |
instance it will stop reporting status for it: this means the instance |
104 | 302 |
will either disappear, if it has been deleted, or appear on another |
105 | 303 |
node, if it's been moved. |
106 | 304 |
|
107 |
Instance Disk status
|
|
108 |
++++++++++++++++++++
|
|
305 |
The ``code`` of the ``status`` field of the report of the Instance status data
|
|
306 |
collector will be:
|
|
109 | 307 |
|
110 |
As for the instance status Ganeti has now only partial information about |
|
111 |
its instance disks: in particular each node is unaware of the disk to |
|
112 |
instance mapping, that exists only on the master. |
|
308 |
``0`` |
|
309 |
if ``status`` is ``0`` for all the instances it is reporting about. |
|
113 | 310 |
|
114 |
For this design doc we plan to fix this by changing all RPCs that create |
|
115 |
a backend storage or that put an already existing one in use and passing |
|
116 |
the relevant instance to the node. The node can then export these to the |
|
117 |
status reporting tool. |
|
311 |
``1`` |
|
312 |
otherwise. |
|
313 |
|
|
314 |
Storage status |
|
315 |
++++++++++++++ |
|
316 |
|
|
317 |
The storage status collectors will be a series of data collectors |
|
318 |
(drbd, rbd, plain, file) that will gather data about all the storage types |
|
319 |
for the current node (this is right now hardcoded to the enabled storage |
|
320 |
types, and in the future tied to the enabled storage pools for the nodegroup). |
|
321 |
|
|
322 |
The ``name`` of each of these collector will reflect what storage type each of |
|
323 |
them refers to. |
|
324 |
|
|
325 |
The ``category`` field of these collector will be ``storage``. |
|
326 |
|
|
327 |
The ``kind`` field will be ``1`` (`Status reporting collectors`_). |
|
328 |
|
|
329 |
The ``data`` section of the report will provide at least the following fields: |
|
330 |
|
|
331 |
``free`` |
|
332 |
The amount of free space (in KBytes). |
|
333 |
|
|
334 |
``used`` |
|
335 |
The amount of used space (in KBytes). |
|
336 |
|
|
337 |
``total`` |
|
338 |
The total visible space (in KBytes). |
|
339 |
|
|
340 |
Each specific storage type might provide more type-specific fields. |
|
341 |
|
|
342 |
In case of error, the ``message`` subfield of the ``status`` field of the |
|
343 |
report of the instance status collector will disclose the nature of the error |
|
344 |
as a type specific information. Examples of these are "backend pv unavailable" |
|
345 |
for lvm storage, "unreachable" for network based storage or "filesystem error" |
|
346 |
for filesystem based implementations. |
|
347 |
|
|
348 |
DRBD status |
|
349 |
*********** |
|
350 |
|
|
351 |
This data collector will run only on nodes where DRBD is actually |
|
352 |
present and it will gather information about DRBD devices. |
|
353 |
|
|
354 |
Its ``kind`` in the report will be ``1`` (`Status reporting collectors`_). |
|
355 |
|
|
356 |
Its ``category`` field in the report will contain the value ``storage``. |
|
357 |
|
|
358 |
When executed in verbose mode, the ``data`` section of the report of this |
|
359 |
collector will provide the following fields: |
|
360 |
|
|
361 |
``versionInfo`` |
|
362 |
Information about the DRBD version number, given by a combination of |
|
363 |
any (but at least one) of the following fields: |
|
364 |
|
|
365 |
``version`` |
|
366 |
The DRBD driver version. |
|
367 |
|
|
368 |
``api`` |
|
369 |
The API version number. |
|
370 |
|
|
371 |
``proto`` |
|
372 |
The protocol version. |
|
373 |
|
|
374 |
``srcversion`` |
|
375 |
The version of the source files. |
|
376 |
|
|
377 |
``gitHash`` |
|
378 |
Git hash of the source files. |
|
379 |
|
|
380 |
``buildBy`` |
|
381 |
Who built the binary, and, optionally, when. |
|
382 |
|
|
383 |
``device`` |
|
384 |
A list of structures, each describing a DRBD device (a minor) and containing |
|
385 |
the following fields: |
|
386 |
|
|
387 |
``minor`` |
|
388 |
The device minor number. |
|
389 |
|
|
390 |
``connectionState`` |
|
391 |
The state of the connection. If it is "Unconfigured", all the following |
|
392 |
fields are not present. |
|
393 |
|
|
394 |
``localRole`` |
|
395 |
The role of the local resource. |
|
396 |
|
|
397 |
``remoteRole`` |
|
398 |
The role of the remote resource. |
|
399 |
|
|
400 |
``localState`` |
|
401 |
The status of the local disk. |
|
402 |
|
|
403 |
``remoteState`` |
|
404 |
The status of the remote disk. |
|
405 |
|
|
406 |
``replicationProtocol`` |
|
407 |
The replication protocol being used. |
|
408 |
|
|
409 |
``ioFlags`` |
|
410 |
The input/output flags. |
|
411 |
|
|
412 |
``perfIndicators`` |
|
413 |
The performance indicators. This field will contain the following |
|
414 |
sub-fields: |
|
415 |
|
|
416 |
``networkSend`` |
|
417 |
KiB of data sent on the network. |
|
418 |
|
|
419 |
``networkReceive`` |
|
420 |
KiB of data received from the network. |
|
421 |
|
|
422 |
``diskWrite`` |
|
423 |
KiB of data written on local disk. |
|
424 |
|
|
425 |
``diskRead`` |
|
426 |
KiB of date read from the local disk. |
|
427 |
|
|
428 |
``activityLog`` |
|
429 |
Number of updates of the activity log. |
|
430 |
|
|
431 |
``bitMap`` |
|
432 |
Number of updates to the bitmap area of the metadata. |
|
433 |
|
|
434 |
``localCount`` |
|
435 |
Number of open requests to the local I/O subsystem. |
|
436 |
|
|
437 |
``pending`` |
|
438 |
Number of requests sent to the partner but not yet answered. |
|
439 |
|
|
440 |
``unacknowledged`` |
|
441 |
Number of requests received by the partner but still to be answered. |
|
442 |
|
|
443 |
``applicationPending`` |
|
444 |
Num of block input/output requests forwarded to DRBD but that have not yet |
|
445 |
been answered. |
|
446 |
|
|
447 |
``epochs`` |
|
448 |
(Optional) Number of epoch objects. Not provided by all DRBD versions. |
|
449 |
|
|
450 |
``writeOrder`` |
|
451 |
(Optional) Currently used write ordering method. Not provided by all DRBD |
|
452 |
versions. |
|
453 |
|
|
454 |
``outOfSync`` |
|
455 |
(Optional) KiB of storage currently out of sync. Not provided by all DRBD |
|
456 |
versions. |
|
457 |
|
|
458 |
``syncStatus`` |
|
459 |
(Optional) The status of the synchronization of the disk. This is present |
|
460 |
only if the disk is being synchronized, and includes the following fields: |
|
461 |
|
|
462 |
``percentage`` |
|
463 |
The percentage of synchronized data. |
|
464 |
|
|
465 |
``progress`` |
|
466 |
How far the synchronization is. Written as "x/y", where x and y are |
|
467 |
integer numbers expressed in the measurement unit stated in |
|
468 |
``progressUnit`` |
|
469 |
|
|
470 |
``progressUnit`` |
|
471 |
The measurement unit for the progress indicator. |
|
472 |
|
|
473 |
``timeToFinish`` |
|
474 |
The expected time before finishing the synchronization. |
|
475 |
|
|
476 |
``speed`` |
|
477 |
The speed of the synchronization. |
|
478 |
|
|
479 |
``want`` |
|
480 |
The desiderd speed of the synchronization. |
|
481 |
|
|
482 |
``speedUnit`` |
|
483 |
The measurement unit of the ``speed`` and ``want`` values. Expressed |
|
484 |
as "size/time". |
|
485 |
|
|
486 |
``instance`` |
|
487 |
The name of the Ganeti instance this disk is associated to. |
|
118 | 488 |
|
119 |
While we haven't implemented these RPC changes yet, we'll use confd to |
|
120 |
fetch this information in the data collector. |
|
121 |
|
|
122 |
Since Ganeti supports many type of disks for instances (drbd, rbd, |
|
123 |
plain, file) we will export both a "generic" status which will work for |
|
124 |
any type of disk and will be very opaque (at minimum just an "healthy" |
|
125 |
or "error" state, plus perhaps some human readable comment and a |
|
126 |
"per-type" status which will explain more about the internal details but |
|
127 |
will not be compatible between different storage types (and will for |
|
128 |
example export the drbd connection status, sync, and so on). |
|
129 |
|
|
130 |
Status of storage for instances |
|
131 |
+++++++++++++++++++++++++++++++ |
|
132 |
|
|
133 |
The node will also be reporting on all storage types it knows about for |
|
134 |
the current node (this is right now hardcoded to the enabled storage |
|
135 |
types, and in the future tied to the enabled storage pools for the |
|
136 |
nodegroup). For this kind of information also we will report both a |
|
137 |
generic health status (healthy or error) for each type of storage, and |
|
138 |
some more generic statistics (free space, used space, total visible |
|
139 |
space). In addition type specific information can be exported: for |
|
140 |
example, in case of error, the nature of the error can be disclosed as a |
|
141 |
type specific information. Examples of these are "backend pv |
|
142 |
unavailable" for lvm storage, "unreachable" for network based storage or |
|
143 |
"filesystem error" for filesystem based implementations. |
|
144 | 489 |
|
145 | 490 |
Ganeti daemons status |
146 | 491 |
+++++++++++++++++++++ |
147 | 492 |
|
148 |
Ganeti will report what information it has about its own daemons: this |
|
149 |
includes memory usage, uptime, CPU usage. This should allow identifying |
|
150 |
possible problems with the Ganeti system itself: for example memory |
|
151 |
leaks, crashes and high resource utilization should be evident by |
|
152 |
analyzing this information. |
|
493 |
Ganeti will report what information it has about its own daemons. |
|
494 |
This should allow identifying possible problems with the Ganeti system itself: |
|
495 |
for example memory leaks, crashes and high resource utilization should be |
|
496 |
evident by analyzing this information. |
|
497 |
|
|
498 |
The ``kind`` field will be ``1`` (`Status reporting collectors`_). |
|
499 |
|
|
500 |
Each daemon will have its own data collector, and each of them will have |
|
501 |
a ``category`` field valued ``daemon``. |
|
502 |
|
|
503 |
When executed in verbose mode, their data section will include at least: |
|
504 |
|
|
505 |
``memory`` |
|
506 |
The amount of used memory. |
|
507 |
|
|
508 |
``size_unit`` |
|
509 |
The measurement unit used for the memory. |
|
153 | 510 |
|
154 |
Ganeti daemons will also be able to export extra internal information to |
|
155 |
the status reporting, through the plugin system (see below). |
|
511 |
``uptime`` |
|
512 |
The uptime of the daemon. |
|
513 |
|
|
514 |
``CPU usage`` |
|
515 |
How much cpu the daemon is using (percentage). |
|
516 |
|
|
517 |
Any other daemon-specific information can be included as well in the ``data`` |
|
518 |
section. |
|
156 | 519 |
|
157 | 520 |
Hypervisor resources report |
158 | 521 |
+++++++++++++++++++++++++++ |
... | ... | |
164 | 527 |
specific" way. Each hypervisor can then add extra specific information |
165 | 528 |
that is not generic enough be abstracted. |
166 | 529 |
|
530 |
The ``kind`` field will be ``0`` (`Performance reporting collectors`_). |
|
531 |
|
|
532 |
Each of the hypervisor data collectory will be of ``category``: ``hypervisor``. |
|
533 |
|
|
167 | 534 |
Node OS resources report |
168 | 535 |
++++++++++++++++++++++++ |
169 | 536 |
|
170 | 537 |
Since Ganeti assumes it's running on Linux, it's useful to export some |
171 |
basic information as seen by the host system. This includes number and |
|
172 |
status of CPUs, memory, filesystems and network intefaces as well as the |
|
173 |
version of components Ganeti interacts with (Linux, drbd, hypervisor, |
|
174 |
etc). |
|
538 |
basic information as seen by the host system. |
|
175 | 539 |
|
176 |
Note that we won't go into any hardware specific details (e.g. querying a |
|
177 |
node RAID is outside the scope of this, and can be implemented as a |
|
178 |
plugin) but we can easily just report the information above, since it's |
|
179 |
standard enough across all systems. |
|
540 |
The ``category`` field of the report will be ``null``. |
|
180 | 541 |
|
181 |
Plugin system |
|
182 |
+++++++++++++ |
|
542 |
The ``kind`` field will be ``0`` (`Performance reporting collectors`_). |
|
183 | 543 |
|
184 |
The monitoring system will be equipped with a plugin system that can |
|
185 |
export specific local information through it. The plugin system will be |
|
186 |
in the form of either scripts whose output will be inserted in the |
|
187 |
report, plain text files which will be inserted into the report, or |
|
188 |
local unix or network sockets from which the information has to be read. |
|
189 |
This should allow most flexibility for implementing an efficient system, |
|
190 |
while being able to keep it as simple as possible. |
|
544 |
The ``data`` section will include: |
|
191 | 545 |
|
192 |
The plugin system is expected to be used by local installations to |
|
193 |
export any installation specific information that they want to be |
|
194 |
monitored, about either hardware or software on their systems. |
|
546 |
``cpu_number`` |
|
547 |
The number of available cpus. |
|
195 | 548 |
|
549 |
``cpus`` |
|
550 |
A list with one element per cpu, showing its average load. |
|
196 | 551 |
|
197 |
Format of the query
|
|
198 |
-------------------
|
|
552 |
``memory``
|
|
553 |
The current view of memory (free, used, cached, etc.)
|
|
199 | 554 |
|
200 |
The query will be an HTTP GET request on a particular port. At the |
|
201 |
beginning it will only be possible to query the full status report. |
|
555 |
``filesystem`` |
|
556 |
A list with one element per filesystem, showing a summary of the |
|
557 |
total/available space. |
|
202 | 558 |
|
559 |
``NICs`` |
|
560 |
A list with one element per network interface, showing the amount of |
|
561 |
sent/received data, error rate, IP address of the interface, etc. |
|
203 | 562 |
|
204 |
Format of the report |
|
205 |
-------------------- |
|
563 |
``versions`` |
|
564 |
A map using the name of a component Ganeti interacts (Linux, drbd, |
|
565 |
hypervisor, etc) as the key and its version number as the value. |
|
206 | 566 |
|
207 |
The report of the will be in JSON format, and it will present an array |
|
208 |
of report objects. |
|
209 |
Each report object will be produced by a specific data collector. |
|
210 |
Each report object includes some mandatory fields, to be provided by all |
|
211 |
the data collectors, and a field to contain data collector-specific |
|
212 |
data. |
|
567 |
Note that we won't go into any hardware specific details (e.g. querying a |
|
568 |
node RAID is outside the scope of this, and can be implemented as a |
|
569 |
plugin) but we can easily just report the information above, since it's |
|
570 |
standard enough across all systems. |
|
213 | 571 |
|
214 |
Here follows a minimal example of a report:: |
|
572 |
Instance disk status propagation |
|
573 |
-------------------------------- |
|
215 | 574 |
|
216 |
[ |
|
217 |
{ |
|
218 |
"name" : "TheCollectorIdentifier", |
|
219 |
"version" : "1.2", |
|
220 |
"format_version" : 1, |
|
221 |
"timestamp" : 1351607182000000000, |
|
222 |
"data" : { "plugin_specific_data" : "go_here" } |
|
223 |
}, |
|
224 |
{ |
|
225 |
"name" : "AnotherDataCollector", |
|
226 |
"version" : "B", |
|
227 |
"format_version" : 7, |
|
228 |
"timestamp" : 1351609526123854000, |
|
229 |
"data" : { "plugin_specific" : "data", |
|
230 |
"some_late_data" : { "timestamp" : "SPECIFIC_TIME", |
|
231 |
... } |
|
232 |
} |
|
233 |
} |
|
234 |
] |
|
575 |
As for the instance status Ganeti has now only partial information about |
|
576 |
its instance disks: in particular each node is unaware of the disk to |
|
577 |
instance mapping, that exists only on the master. |
|
235 | 578 |
|
236 |
Here is the description of the mandatory fields of each object: |
|
579 |
For this design doc we plan to fix this by changing all RPCs that create |
|
580 |
a backend storage or that put an already existing one in use and passing |
|
581 |
the relevant instance to the node. The node can then export these to the |
|
582 |
status reporting tool. |
|
237 | 583 |
|
238 |
name |
|
239 |
the name of the data collector that produced this part of the report. |
|
240 |
It is supposed to be unique inside a report. |
|
584 |
While we haven't implemented these RPC changes yet, we'll use Confd to |
|
585 |
fetch this information in the data collectors. |
|
241 | 586 |
|
242 |
version |
|
243 |
the version of the data collector that produces this part of the |
|
244 |
report. Built-in data collectors (as opposed to those implemented as |
|
245 |
plugins) should have "B" as the version number. |
|
587 |
Plugin system |
|
588 |
------------- |
|
246 | 589 |
|
247 |
format_version |
|
248 |
the format of what is represented in the "data" field for each data |
|
249 |
collector might change over time. Every time this happens, the |
|
250 |
format_version should be changed, so that who reads the report knows |
|
251 |
what format to expect, and how to correctly interpret it. |
|
590 |
The monitoring system will be equipped with a plugin system that can |
|
591 |
export specific local information through it. |
|
252 | 592 |
|
253 |
timestamp |
|
254 |
the time when the reported data were gathered. Is has to be expressed |
|
255 |
in nanoseconds since the unix epoch (0:00:00 January 01, 1970). If not |
|
256 |
enough precision is available (or needed) it can be padded with |
|
257 |
zeroes. If a report object needs multiple timestamps, it can add more |
|
258 |
and/or override this one inside its own "data" section. |
|
593 |
The plugin system is expected to be used by local installations to |
|
594 |
export any installation specific information that they want to be |
|
595 |
monitored, about either hardware or software on their systems. |
|
259 | 596 |
|
260 |
data |
|
261 |
this field contains all the data generated by the data collector, in |
|
262 |
its own independently defined format. The monitoring agent could check |
|
263 |
this syntactically (according to the JSON specifications) but not |
|
264 |
semantically. |
|
597 |
The plugin system will be in the form of either scripts or binaries whose output |
|
598 |
will be inserted in the report. |
|
265 | 599 |
|
600 |
Eventually support for other kinds of plugins might be added as well, such as |
|
601 |
plain text files which will be inserted into the report, or local unix or |
|
602 |
network sockets from which the information has to be read. This should allow |
|
603 |
most flexibility for implementing an efficient system, while being able to keep |
|
604 |
it as simple as possible. |
|
266 | 605 |
|
267 | 606 |
Data collectors |
268 | 607 |
--------------- |
269 | 608 |
|
270 | 609 |
In order to ease testing as well as to make it simple to reuse this |
271 | 610 |
subsystem it will be possible to run just the "data collectors" on each |
272 |
node without passing through the agent daemon. Each data collector will |
|
273 |
report specific data about its subsystem and will be documented |
|
274 |
separately. |
|
611 |
node without passing through the agent daemon. |
|
275 | 612 |
|
276 | 613 |
If a data collector is run independently, it should print on stdout its |
277 | 614 |
report, according to the format corresponding to a single data collector |
278 |
report object, as described in the previous paragraph. |
|
279 |
|
|
615 |
report object, as described in the previous paragraphs. |
|
280 | 616 |
|
281 | 617 |
Mode of operation |
282 | 618 |
----------------- |
... | ... | |
297 | 633 |
can use adaptively to query a certain resource faster or slower |
298 | 634 |
depending on those two parameters. |
299 | 635 |
|
636 |
When run as stand-alone binaries, the data collector will not using any |
|
637 |
caching system, and just fetch and return the data immediately. |
|
300 | 638 |
|
301 | 639 |
Implementation place |
302 | 640 |
-------------------- |
... | ... | |
315 | 653 |
|
316 | 654 |
We will implement the agent system in this order: |
317 | 655 |
|
318 |
- initial example data collectors (eg. for drbd and instance status. |
|
319 |
Data collector-specific report format TBD).
|
|
320 |
- initial daemon for exporting data
|
|
656 |
- initial example data collectors (eg. for drbd and instance status).
|
|
657 |
- initial daemon for exporting data, integrating the existing collectors
|
|
658 |
- plugin system
|
|
321 | 659 |
- RPC updates for instance status reasons and disk to instance mapping |
660 |
- cache layer for the daemon |
|
322 | 661 |
- more data collectors |
323 |
- cache layer for the daemon (if needed) |
|
324 | 662 |
|
325 | 663 |
|
326 | 664 |
Future work |
Also available in: Unified diff