root / doc / design-monitoring-agent.rst @ 41044e04
History | View | Annotate | Download (24.8 kB)
1 |
======================= |
---|---|
2 |
Ganeti monitoring agent |
3 |
======================= |
4 |
|
5 |
.. contents:: :depth: 4 |
6 |
|
7 |
This is a design document detailing the implementation of a Ganeti |
8 |
monitoring agent report system, that can be queried by a monitoring |
9 |
system to calculate health information for a Ganeti cluster. |
10 |
|
11 |
Current state and shortcomings |
12 |
============================== |
13 |
|
14 |
There is currently no monitoring support in Ganeti. While we don't want |
15 |
to build something like Nagios or Pacemaker as part of Ganeti, it would |
16 |
be useful if such tools could easily extract information from a Ganeti |
17 |
machine in order to take actions (example actions include logging an |
18 |
outage for future reporting or alerting a person or system about it). |
19 |
|
20 |
Proposed changes |
21 |
================ |
22 |
|
23 |
Each Ganeti node should export a status page that can be queried by a |
24 |
monitoring system. Such status page will be exported on a network port |
25 |
and will be encoded in JSON (simple text) over HTTP. |
26 |
|
27 |
The choice of JSON is obvious as we already depend on it in Ganeti and |
28 |
thus we don't need to add extra libraries to use it, as opposed to what |
29 |
would happen for XML or some other markup format. |
30 |
|
31 |
Location of agent report |
32 |
------------------------ |
33 |
|
34 |
The report will be available from all nodes, and be concerned for all |
35 |
node-local resources. This allows more real-time information to be |
36 |
available, at the cost of querying all nodes. |
37 |
|
38 |
Information reported |
39 |
-------------------- |
40 |
|
41 |
The monitoring agent system will report on the following basic information: |
42 |
|
43 |
- Instance status |
44 |
- Instance disk status |
45 |
- Status of storage for instances |
46 |
- Ganeti daemons status, CPU usage, memory footprint |
47 |
- Hypervisor resources report (memory, CPU, network interfaces) |
48 |
- Node OS resources report (memory, CPU, network interfaces) |
49 |
- Information from a plugin system |
50 |
|
51 |
Format of the report |
52 |
-------------------- |
53 |
|
54 |
The report of the will be in JSON format, and it will present an array |
55 |
of report objects. |
56 |
Each report object will be produced by a specific data collector. |
57 |
Each report object includes some mandatory fields, to be provided by all |
58 |
the data collectors: |
59 |
|
60 |
``name`` |
61 |
The name of the data collector that produced this part of the report. |
62 |
It is supposed to be unique inside a report. |
63 |
|
64 |
``version`` |
65 |
The version of the data collector that produces this part of the |
66 |
report. Built-in data collectors (as opposed to those implemented as |
67 |
plugins) should have "B" as the version number. |
68 |
|
69 |
``format_version`` |
70 |
The format of what is represented in the "data" field for each data |
71 |
collector might change over time. Every time this happens, the |
72 |
format_version should be changed, so that who reads the report knows |
73 |
what format to expect, and how to correctly interpret it. |
74 |
|
75 |
``timestamp`` |
76 |
The time when the reported data were gathered. It has to be expressed |
77 |
in nanoseconds since the unix epoch (0:00:00 January 01, 1970). If not |
78 |
enough precision is available (or needed) it can be padded with |
79 |
zeroes. If a report object needs multiple timestamps, it can add more |
80 |
and/or override this one inside its own "data" section. |
81 |
|
82 |
``category`` |
83 |
A collector can belong to a given category of collectors (e.g.: storage |
84 |
collectors, daemon collector). This means that it will have to provide a |
85 |
minumum set of prescribed fields, as documented for each category. |
86 |
This field will contain the name of the category the collector belongs to, |
87 |
if any, or just the ``null`` value. |
88 |
|
89 |
``kind`` |
90 |
Two kinds of collectors are possible: |
91 |
`Performance reporting collectors`_ and `Status reporting collectors`_. |
92 |
The respective paragraphs will describe them and the value of this field. |
93 |
|
94 |
``data`` |
95 |
This field contains all the data generated by the specific data collector, |
96 |
in its own independently defined format. The monitoring agent could check |
97 |
this syntactically (according to the JSON specifications) but not |
98 |
semantically. |
99 |
|
100 |
Here follows a minimal example of a report:: |
101 |
|
102 |
[ |
103 |
{ |
104 |
"name" : "TheCollectorIdentifier", |
105 |
"version" : "1.2", |
106 |
"format_version" : 1, |
107 |
"timestamp" : 1351607182000000000, |
108 |
"category" : null, |
109 |
"kind" : 0, |
110 |
"data" : { "plugin_specific_data" : "go_here" } |
111 |
}, |
112 |
{ |
113 |
"name" : "AnotherDataCollector", |
114 |
"version" : "B", |
115 |
"format_version" : 7, |
116 |
"timestamp" : 1351609526123854000, |
117 |
"category" : "storage", |
118 |
"kind" : 1, |
119 |
"data" : { "status" : { "code" : 1, |
120 |
"message" : "Error on disk 2" |
121 |
}, |
122 |
"plugin_specific" : "data", |
123 |
"some_late_data" : { "timestamp" : 1351609526123942720, |
124 |
... |
125 |
} |
126 |
} |
127 |
} |
128 |
] |
129 |
|
130 |
Performance reporting collectors |
131 |
++++++++++++++++++++++++++++++++ |
132 |
|
133 |
These collectors only provide data about some component of the system, without |
134 |
giving any interpretation over their meaning. |
135 |
|
136 |
The value of the ``kind`` field of the report will be ``0``. |
137 |
|
138 |
Status reporting collectors |
139 |
+++++++++++++++++++++++++++ |
140 |
|
141 |
These collectors will provide information about the status of some |
142 |
component of ganeti, or managed by ganeti. |
143 |
|
144 |
The value of their ``kind`` field will be ``1``. |
145 |
|
146 |
The rationale behind this kind of collectors is that there are some situations |
147 |
where exporting data about the underlying subsystems would expose potential |
148 |
issues. But if Ganeti itself is able (and going) to fix the problem, conflicts |
149 |
might arise between Ganeti and something/somebody else trying to fix the same |
150 |
problem. |
151 |
Also, some external monitoring systems might not be aware of the internals of a |
152 |
particular subsystem (e.g.: DRBD) and might only exploit the high level |
153 |
response of its data collector, alerting an administrator if anything is wrong. |
154 |
Still, completely hiding the underlying data is not a good idea, as they might |
155 |
still be of use in some cases. So status reporting plugins will provide two |
156 |
output modes: one just exporting a high level information about the status, |
157 |
and one also exporting all the data they gathered. |
158 |
The default output mode will be the status-only one. Through a command line |
159 |
parameter (for stand-alone data collectors) or through the HTTP request to the |
160 |
monitoring agent |
161 |
(when collectors are executed as part of it) the verbose output mode providing |
162 |
all the data can be selected. |
163 |
|
164 |
When exporting just the status each status reporting collector will provide, |
165 |
in its ``data`` section, at least the following field: |
166 |
|
167 |
``status`` |
168 |
summarizes the status of the component being monitored and consists of two |
169 |
subfields: |
170 |
|
171 |
``code`` |
172 |
It assumes a numeric value, encoded in such a way to allow using a bitset |
173 |
to easily distinguish which states are currently present in the whole cluster. |
174 |
If the bitwise OR of all the ``status`` fields is 0, the cluster is |
175 |
completely healty. |
176 |
The status codes are as follows: |
177 |
|
178 |
``0`` |
179 |
The collector can determine that everything is working as |
180 |
intended. |
181 |
|
182 |
``1`` |
183 |
Something is temporarily wrong but it is being automatically fixed by |
184 |
Ganeti. |
185 |
There is no need of external intervention. |
186 |
|
187 |
``2`` |
188 |
The collector has failed to understand whether the status is good or |
189 |
bad. Further analysis is required. Interpret this status as a |
190 |
potentially dangerous situation. |
191 |
|
192 |
``4`` |
193 |
The collector can determine that something is wrong and Ganeti has no |
194 |
way to fix it autonomously. External intervention is required. |
195 |
|
196 |
``message`` |
197 |
A message to better explain the reason of the status. |
198 |
The exact format of the message string is data collector dependent. |
199 |
|
200 |
The field is mandatory, but the content can be an empty string if the |
201 |
``code`` is ``0`` (working as intended) or ``1`` (being fixed |
202 |
automatically). |
203 |
|
204 |
If the status code is ``2``, the message should specify what has gone |
205 |
wrong. |
206 |
If the status code is ``4``, the message shoud explain why it was not |
207 |
possible to determine a proper status. |
208 |
|
209 |
The ``data`` section will also contain all the fields describing the gathered |
210 |
data, according to a collector-specific format. |
211 |
|
212 |
Instance status |
213 |
+++++++++++++++ |
214 |
|
215 |
At the moment each node knows which instances are running on it, which |
216 |
instances it is primary for, but not the cause why an instance might not |
217 |
be running. On the other hand we don't want to distribute full instance |
218 |
"admin" status information to all nodes, because of the performance |
219 |
impact this would have. |
220 |
|
221 |
As such we propose that: |
222 |
|
223 |
- Any operation that can affect instance status will have an optional |
224 |
"reason" attached to it (at opcode level). This can be used for |
225 |
example to distinguish an admin request, from a scheduled maintenance |
226 |
or an automated tool's work. If this reason is not passed, Ganeti will |
227 |
just use the information it has about the source of the request: for |
228 |
example a cli shutdown operation will have "cli:shutdown" as a reason, |
229 |
a cli failover operation will have "cli:failover". Operations coming |
230 |
from the remote API will use "rapi" instead of "cli". Of course |
231 |
setting a real site-specific reason is still preferred. |
232 |
- RPCs that affect the instance status will be changed so that the |
233 |
"reason" and the version of the config object they ran on is passed to |
234 |
them. They will then export the new expected instance status, together |
235 |
with the associated reason and object version to the status report |
236 |
system, which then will export those themselves. |
237 |
|
238 |
Monitoring and auditing systems can then use the reason to understand |
239 |
the cause of an instance status, and they can use the timestamp to |
240 |
understand the freshness of their data even in the absence of an atomic |
241 |
cross-node reporting: for example if they see an instance "up" on a node |
242 |
after seeing it running on a previous one, they can compare these values |
243 |
to understand which data is freshest, and repoll the "older" node. Of |
244 |
course if they keep seeing this status this represents an error (either |
245 |
an instance continuously "flapping" between nodes, or an instance is |
246 |
constantly up on more than one), which should be reported and acted |
247 |
upon. |
248 |
|
249 |
The instance status will be on each node, for the instances it is |
250 |
primary for, and its ``data`` section of the report will contain a list |
251 |
of instances, with at least the following fields for each instance: |
252 |
|
253 |
``name`` |
254 |
The name of the instance. |
255 |
|
256 |
``uuid`` |
257 |
The UUID of the instance (stable on name change). |
258 |
|
259 |
``admin_state`` |
260 |
The status of the instance (up/down/offline) as requested by the admin. |
261 |
|
262 |
``actual_state`` |
263 |
The actual status of the instance. It can be ``up``, ``down``, or |
264 |
``hung`` if the instance is up but it appears to be completely stuck. |
265 |
|
266 |
``uptime`` |
267 |
The uptime of the instance (if it is up, "null" otherwise). |
268 |
|
269 |
``mtime`` |
270 |
The timestamp of the last known change to the instance state. |
271 |
|
272 |
``state_reason`` |
273 |
The last known reason for state change, described according to the |
274 |
following subfields: |
275 |
|
276 |
``text`` |
277 |
Either a user-provided reason (if any), or the name of the command that |
278 |
triggered the state change, as a fallback. |
279 |
|
280 |
``jobID`` |
281 |
The ID of the job that caused the state change. |
282 |
|
283 |
``source`` |
284 |
Where the state change was triggered (RAPI, CLI). |
285 |
|
286 |
``status`` |
287 |
It represents the status of the instance, and its format is the same as that |
288 |
of the ``status`` field of `Status reporting collectors`_. |
289 |
|
290 |
Each hypervisor should provide its own instance status data collector, possibly |
291 |
with the addition of more, specific, fields. |
292 |
The ``category`` field of all of them will be ``instance``. |
293 |
The ``kind`` field will be ``1``. |
294 |
|
295 |
Note that as soon as a node knows it's not the primary anymore for an |
296 |
instance it will stop reporting status for it: this means the instance |
297 |
will either disappear, if it has been deleted, or appear on another |
298 |
node, if it's been moved. |
299 |
|
300 |
The ``code`` of the ``status`` field of the report of the Instance status data |
301 |
collector will be: |
302 |
|
303 |
``0`` |
304 |
if ``status`` is ``0`` for all the instances it is reporting about. |
305 |
|
306 |
``1`` |
307 |
otherwise. |
308 |
|
309 |
Storage status |
310 |
++++++++++++++ |
311 |
|
312 |
The storage status collectors will be a series of data collectors |
313 |
(drbd, rbd, plain, file) that will gather data about all the storage types |
314 |
for the current node (this is right now hardcoded to the enabled storage |
315 |
types, and in the future tied to the enabled storage pools for the nodegroup). |
316 |
|
317 |
The ``name`` of each of these collector will reflect what storage type each of |
318 |
them refers to. |
319 |
|
320 |
The ``category`` field of these collector will be ``storage``. |
321 |
|
322 |
The ``kind`` field will be ``1`` (`Status reporting collectors`_). |
323 |
|
324 |
The ``data`` section of the report will provide at least the following fields: |
325 |
|
326 |
``free`` |
327 |
The amount of free space (in KBytes). |
328 |
|
329 |
``used`` |
330 |
The amount of used space (in KBytes). |
331 |
|
332 |
``total`` |
333 |
The total visible space (in KBytes). |
334 |
|
335 |
Each specific storage type might provide more type-specific fields. |
336 |
|
337 |
In case of error, the ``message`` subfield of the ``status`` field of the |
338 |
report of the instance status collector will disclose the nature of the error |
339 |
as a type specific information. Examples of these are "backend pv unavailable" |
340 |
for lvm storage, "unreachable" for network based storage or "filesystem error" |
341 |
for filesystem based implementations. |
342 |
|
343 |
DRBD status |
344 |
*********** |
345 |
|
346 |
This data collector will run only on nodes where DRBD is actually |
347 |
present and it will gather information about DRBD devices. |
348 |
|
349 |
Its ``kind`` in the report will be ``1`` (`Status reporting collectors`_). |
350 |
|
351 |
Its ``category`` field in the report will contain the value ``storage``. |
352 |
|
353 |
When executed in verbose mode, the ``data`` section of the report of this |
354 |
collector will provide the following fields: |
355 |
|
356 |
``versionInfo`` |
357 |
Information about the DRBD version number, given by a combination of |
358 |
any (but at least one) of the following fields: |
359 |
|
360 |
``version`` |
361 |
The DRBD driver version. |
362 |
|
363 |
``api`` |
364 |
The API version number. |
365 |
|
366 |
``proto`` |
367 |
The protocol version. |
368 |
|
369 |
``srcversion`` |
370 |
The version of the source files. |
371 |
|
372 |
``gitHash`` |
373 |
Git hash of the source files. |
374 |
|
375 |
``buildBy`` |
376 |
Who built the binary, and, optionally, when. |
377 |
|
378 |
``device`` |
379 |
A list of structures, each describing a DRBD device (a minor) and containing |
380 |
the following fields: |
381 |
|
382 |
``minor`` |
383 |
The device minor number. |
384 |
|
385 |
``connectionState`` |
386 |
The state of the connection. If it is "Unconfigured", all the following |
387 |
fields are not present. |
388 |
|
389 |
``localRole`` |
390 |
The role of the local resource. |
391 |
|
392 |
``remoteRole`` |
393 |
The role of the remote resource. |
394 |
|
395 |
``localState`` |
396 |
The status of the local disk. |
397 |
|
398 |
``remoteState`` |
399 |
The status of the remote disk. |
400 |
|
401 |
``replicationProtocol`` |
402 |
The replication protocol being used. |
403 |
|
404 |
``ioFlags`` |
405 |
The input/output flags. |
406 |
|
407 |
``perfIndicators`` |
408 |
The performance indicators. This field will contain the following |
409 |
sub-fields: |
410 |
|
411 |
``networkSend`` |
412 |
KiB of data sent on the network. |
413 |
|
414 |
``networkReceive`` |
415 |
KiB of data received from the network. |
416 |
|
417 |
``diskWrite`` |
418 |
KiB of data written on local disk. |
419 |
|
420 |
``diskRead`` |
421 |
KiB of date read from the local disk. |
422 |
|
423 |
``activityLog`` |
424 |
Number of updates of the activity log. |
425 |
|
426 |
``bitMap`` |
427 |
Number of updates to the bitmap area of the metadata. |
428 |
|
429 |
``localCount`` |
430 |
Number of open requests to the local I/O subsystem. |
431 |
|
432 |
``pending`` |
433 |
Number of requests sent to the partner but not yet answered. |
434 |
|
435 |
``unacknowledged`` |
436 |
Number of requests received by the partner but still to be answered. |
437 |
|
438 |
``applicationPending`` |
439 |
Num of block input/output requests forwarded to DRBD but that have not yet |
440 |
been answered. |
441 |
|
442 |
``epochs`` |
443 |
(Optional) Number of epoch objects. Not provided by all DRBD versions. |
444 |
|
445 |
``writeOrder`` |
446 |
(Optional) Currently used write ordering method. Not provided by all DRBD |
447 |
versions. |
448 |
|
449 |
``outOfSync`` |
450 |
(Optional) KiB of storage currently out of sync. Not provided by all DRBD |
451 |
versions. |
452 |
|
453 |
``syncStatus`` |
454 |
(Optional) The status of the synchronization of the disk. This is present |
455 |
only if the disk is being synchronized, and includes the following fields: |
456 |
|
457 |
``percentage`` |
458 |
The percentage of synchronized data. |
459 |
|
460 |
``progress`` |
461 |
How far the synchronization is. Written as "x/y", where x and y are |
462 |
integer numbers expressed in the measurement unit stated in |
463 |
``progressUnit`` |
464 |
|
465 |
``progressUnit`` |
466 |
The measurement unit for the progress indicator. |
467 |
|
468 |
``timeToFinish`` |
469 |
The expected time before finishing the synchronization. |
470 |
|
471 |
``speed`` |
472 |
The speed of the synchronization. |
473 |
|
474 |
``want`` |
475 |
The desiderd speed of the synchronization. |
476 |
|
477 |
``speedUnit`` |
478 |
The measurement unit of the ``speed`` and ``want`` values. Expressed |
479 |
as "size/time". |
480 |
|
481 |
``instance`` |
482 |
The name of the Ganeti instance this disk is associated to. |
483 |
|
484 |
|
485 |
Ganeti daemons status |
486 |
+++++++++++++++++++++ |
487 |
|
488 |
Ganeti will report what information it has about its own daemons. |
489 |
This should allow identifying possible problems with the Ganeti system itself: |
490 |
for example memory leaks, crashes and high resource utilization should be |
491 |
evident by analyzing this information. |
492 |
|
493 |
The ``kind`` field will be ``1`` (`Status reporting collectors`_). |
494 |
|
495 |
Each daemon will have its own data collector, and each of them will have |
496 |
a ``category`` field valued ``daemon``. |
497 |
|
498 |
When executed in verbose mode, their data section will include at least: |
499 |
|
500 |
``memory`` |
501 |
The amount of used memory. |
502 |
|
503 |
``size_unit`` |
504 |
The measurement unit used for the memory. |
505 |
|
506 |
``uptime`` |
507 |
The uptime of the daemon. |
508 |
|
509 |
``CPU usage`` |
510 |
How much cpu the daemon is using (percentage). |
511 |
|
512 |
Any other daemon-specific information can be included as well in the ``data`` |
513 |
section. |
514 |
|
515 |
Hypervisor resources report |
516 |
+++++++++++++++++++++++++++ |
517 |
|
518 |
Each hypervisor has a view of system resources that sometimes is |
519 |
different than the one the OS sees (for example in Xen the Node OS, |
520 |
running as Dom0, has access to only part of those resources). In this |
521 |
section we'll report all information we can in a "non hypervisor |
522 |
specific" way. Each hypervisor can then add extra specific information |
523 |
that is not generic enough be abstracted. |
524 |
|
525 |
The ``kind`` field will be ``0`` (`Performance reporting collectors`_). |
526 |
|
527 |
Each of the hypervisor data collectory will be of ``category``: ``hypervisor``. |
528 |
|
529 |
Node OS resources report |
530 |
++++++++++++++++++++++++ |
531 |
|
532 |
Since Ganeti assumes it's running on Linux, it's useful to export some |
533 |
basic information as seen by the host system. |
534 |
|
535 |
The ``category`` field of the report will be ``null``. |
536 |
|
537 |
The ``kind`` field will be ``0`` (`Performance reporting collectors`_). |
538 |
|
539 |
The ``data`` section will include: |
540 |
|
541 |
``cpu_number`` |
542 |
The number of available cpus. |
543 |
|
544 |
``cpus`` |
545 |
A list with one element per cpu, showing its average load. |
546 |
|
547 |
``memory`` |
548 |
The current view of memory (free, used, cached, etc.) |
549 |
|
550 |
``filesystem`` |
551 |
A list with one element per filesystem, showing a summary of the |
552 |
total/available space. |
553 |
|
554 |
``NICs`` |
555 |
A list with one element per network interface, showing the amount of |
556 |
sent/received data, error rate, IP address of the interface, etc. |
557 |
|
558 |
``versions`` |
559 |
A map using the name of a component Ganeti interacts (Linux, drbd, |
560 |
hypervisor, etc) as the key and its version number as the value. |
561 |
|
562 |
Note that we won't go into any hardware specific details (e.g. querying a |
563 |
node RAID is outside the scope of this, and can be implemented as a |
564 |
plugin) but we can easily just report the information above, since it's |
565 |
standard enough across all systems. |
566 |
|
567 |
Format of the query |
568 |
------------------- |
569 |
|
570 |
The queries to the monitoring agent will be HTTP GET requests on port 1815. |
571 |
The answer will be encoded in JSON format and will depend on the specific |
572 |
accessed resource. |
573 |
|
574 |
If a request is sent to a non-existing resource, a 404 error will be returned by |
575 |
the HTTP server. |
576 |
|
577 |
The following paragraphs will present the existing resources supported by the |
578 |
current protocol version, that is version 1. |
579 |
|
580 |
``/`` |
581 |
+++++ |
582 |
The root resource. It will return the list of the supported protocol version |
583 |
numbers. |
584 |
|
585 |
Currently, this will include only version 1. |
586 |
|
587 |
``/1`` |
588 |
++++++ |
589 |
Not an actual resource per-se, it is the root of all the resources of protocol |
590 |
version 1. |
591 |
|
592 |
If requested through GET, the null JSON value will be returned. |
593 |
|
594 |
``/1/list/collectors`` |
595 |
++++++++++++++++++++++ |
596 |
Returns a list of tuples (kind, category, name) showing all the collectors |
597 |
available in the system. |
598 |
|
599 |
``/1/report/all`` |
600 |
+++++++++++++++++ |
601 |
A list of the reports of all the data collectors, as described in the section |
602 |
`Format of the report`_. |
603 |
|
604 |
`Status reporting collectors`_ will provide their output in non-verbose format. |
605 |
The verbose format can be requested by adding the parameter ``verbose=1`` to the |
606 |
request. |
607 |
|
608 |
``/1/report/[category]/[collector_name]`` |
609 |
+++++++++++++++++++++++++++++++++++++++++ |
610 |
Returns the report of the collector ``[collector_name]`` that belongs to the |
611 |
specified ``[category]``. |
612 |
|
613 |
If a collector does not belong to any category, ``collector`` will be used as |
614 |
the value for ``[category]``. |
615 |
|
616 |
`Status reporting collectors`_ will provide their output in non-verbose format. |
617 |
The verbose format can be requested by adding the parameter ``verbose=1`` to the |
618 |
request. |
619 |
|
620 |
Instance disk status propagation |
621 |
-------------------------------- |
622 |
|
623 |
As for the instance status Ganeti has now only partial information about |
624 |
its instance disks: in particular each node is unaware of the disk to |
625 |
instance mapping, that exists only on the master. |
626 |
|
627 |
For this design doc we plan to fix this by changing all RPCs that create |
628 |
a backend storage or that put an already existing one in use and passing |
629 |
the relevant instance to the node. The node can then export these to the |
630 |
status reporting tool. |
631 |
|
632 |
While we haven't implemented these RPC changes yet, we'll use Confd to |
633 |
fetch this information in the data collectors. |
634 |
|
635 |
Plugin system |
636 |
------------- |
637 |
|
638 |
The monitoring system will be equipped with a plugin system that can |
639 |
export specific local information through it. |
640 |
|
641 |
The plugin system is expected to be used by local installations to |
642 |
export any installation specific information that they want to be |
643 |
monitored, about either hardware or software on their systems. |
644 |
|
645 |
The plugin system will be in the form of either scripts or binaries whose output |
646 |
will be inserted in the report. |
647 |
|
648 |
Eventually support for other kinds of plugins might be added as well, such as |
649 |
plain text files which will be inserted into the report, or local unix or |
650 |
network sockets from which the information has to be read. This should allow |
651 |
most flexibility for implementing an efficient system, while being able to keep |
652 |
it as simple as possible. |
653 |
|
654 |
Data collectors |
655 |
--------------- |
656 |
|
657 |
In order to ease testing as well as to make it simple to reuse this |
658 |
subsystem it will be possible to run just the "data collectors" on each |
659 |
node without passing through the agent daemon. |
660 |
|
661 |
If a data collector is run independently, it should print on stdout its |
662 |
report, according to the format corresponding to a single data collector |
663 |
report object, as described in the previous paragraphs. |
664 |
|
665 |
Mode of operation |
666 |
----------------- |
667 |
|
668 |
In order to be able to report information fast the monitoring agent |
669 |
daemon will keep an in-memory or on-disk cache of the status, which will |
670 |
be returned when queries are made. The status system will then |
671 |
periodically check resources to make sure the status is up to date. |
672 |
|
673 |
Different parts of the report will be queried at different speeds. These |
674 |
will depend on: |
675 |
- how often they vary (or we expect them to vary) |
676 |
- how fast they are to query |
677 |
- how important their freshness is |
678 |
|
679 |
Of course the last parameter is installation specific, and while we'll |
680 |
try to have defaults, it will be configurable. The first two instead we |
681 |
can use adaptively to query a certain resource faster or slower |
682 |
depending on those two parameters. |
683 |
|
684 |
When run as stand-alone binaries, the data collector will not using any |
685 |
caching system, and just fetch and return the data immediately. |
686 |
|
687 |
Implementation place |
688 |
-------------------- |
689 |
|
690 |
The status daemon will be implemented as a standalone Haskell daemon. In |
691 |
the future it should be easy to merge multiple daemons into one with |
692 |
multiple entry points, should we find out it saves resources and doesn't |
693 |
impact functionality. |
694 |
|
695 |
The libekg library should be looked at for easily providing metrics in |
696 |
json format. |
697 |
|
698 |
Implementation order |
699 |
-------------------- |
700 |
|
701 |
We will implement the agent system in this order: |
702 |
|
703 |
- initial example data collectors (eg. for drbd and instance status). |
704 |
- initial daemon for exporting data, integrating the existing collectors |
705 |
- plugin system |
706 |
- RPC updates for instance status reasons and disk to instance mapping |
707 |
- cache layer for the daemon |
708 |
- more data collectors |
709 |
|
710 |
|
711 |
Future work |
712 |
=========== |
713 |
|
714 |
As a future step it can be useful to "centralize" all this reporting |
715 |
data on a single place. This for example can be just the master node, or |
716 |
all the master candidates. We will evaluate doing this after the first |
717 |
node-local version has been developed and tested. |
718 |
|
719 |
Another possible change is replacing the "read-only" RPCs with queries |
720 |
to the agent system, thus having only one way of collecting information |
721 |
from the nodes from a monitoring system and for Ganeti itself. |
722 |
|
723 |
One extra feature we may need is a way to query for only sub-parts of |
724 |
the report (eg. instances status only). This can be done by passing |
725 |
arguments to the HTTP GET, which will be defined when we get to this |
726 |
funtionality. |
727 |
|
728 |
Finally the :doc:`autorepair system design <design-autorepair>`. system |
729 |
(see its design) can be expanded to use the monitoring agent system as a |
730 |
source of information to decide which repairs it can perform. |
731 |
|
732 |
.. vim: set textwidth=72 : |
733 |
.. Local Variables: |
734 |
.. mode: rst |
735 |
.. fill-column: 72 |
736 |
.. End: |