Statistics
| Branch: | Tag: | Revision:

root / doc / design-monitoring-agent.rst @ 42b50796

History | View | Annotate | Download (24.6 kB)

1
=======================
2
Ganeti monitoring agent
3
=======================
4

    
5
.. contents:: :depth: 4
6

    
7
This is a design document detailing the implementation of a Ganeti
8
monitoring agent report system, that can be queried by a monitoring
9
system to calculate health information for a Ganeti cluster.
10

    
11
Current state and shortcomings
12
==============================
13

    
14
There is currently no monitoring support in Ganeti. While we don't want
15
to build something like Nagios or Pacemaker as part of Ganeti, it would
16
be useful if such tools could easily extract information from a Ganeti
17
machine in order to take actions (example actions include logging an
18
outage for future reporting or alerting a person or system about it).
19

    
20
Proposed changes
21
================
22

    
23
Each Ganeti node should export a status page that can be queried by a
24
monitoring system. Such status page will be exported on a network port
25
and will be encoded in JSON (simple text) over HTTP.
26

    
27
The choice of JSON is obvious as we already depend on it in Ganeti and
28
thus we don't need to add extra libraries to use it, as opposed to what
29
would happen for XML or some other markup format.
30

    
31
Location of agent report
32
------------------------
33

    
34
The report will be available from all nodes, and be concerned for all
35
node-local resources. This allows more real-time information to be
36
available, at the cost of querying all nodes.
37

    
38
Information reported
39
--------------------
40

    
41
The monitoring agent system will report on the following basic information:
42

    
43
- Instance status
44
- Instance disk status
45
- Status of storage for instances
46
- Ganeti daemons status, CPU usage, memory footprint
47
- Hypervisor resources report (memory, CPU, network interfaces)
48
- Node OS resources report (memory, CPU, network interfaces)
49
- Information from a plugin system
50

    
51
Format of the report
52
--------------------
53

    
54
The report of the will be in JSON format, and it will present an array
55
of report objects.
56
Each report object will be produced by a specific data collector.
57
Each report object includes some mandatory fields, to be provided by all
58
the data collectors:
59

    
60
``name``
61
  The name of the data collector that produced this part of the report.
62
  It is supposed to be unique inside a report.
63

    
64
``version``
65
  The version of the data collector that produces this part of the
66
  report. Built-in data collectors (as opposed to those implemented as
67
  plugins) should have "B" as the version number.
68

    
69
``format_version``
70
  The format of what is represented in the "data" field for each data
71
  collector might change over time. Every time this happens, the
72
  format_version should be changed, so that who reads the report knows
73
  what format to expect, and how to correctly interpret it.
74

    
75
``timestamp``
76
  The time when the reported data were gathered. It has to be expressed
77
  in nanoseconds since the unix epoch (0:00:00 January 01, 1970). If not
78
  enough precision is available (or needed) it can be padded with
79
  zeroes. If a report object needs multiple timestamps, it can add more
80
  and/or override this one inside its own "data" section.
81

    
82
``category``
83
  A collector can belong to a given category of collectors (e.g.: storage
84
  collectors, daemon collector). This means that it will have to provide a
85
  minumum set of prescribed fields, as documented for each category.
86
  This field will contain the name of the category the collector belongs to,
87
  if any, or just the ``null`` value.
88

    
89
``kind``
90
  Two kinds of collectors are possible:
91
  `Performance reporting collectors`_ and `Status reporting collectors`_.
92
  The respective paragraphs will describe them and the value of this field.
93

    
94
``data``
95
  This field contains all the data generated by the specific data collector,
96
  in its own independently defined format. The monitoring agent could check
97
  this syntactically (according to the JSON specifications) but not
98
  semantically.
99

    
100
Here follows a minimal example of a report::
101

    
102
  [
103
  {
104
      "name" : "TheCollectorIdentifier",
105
      "version" : "1.2",
106
      "format_version" : 1,
107
      "timestamp" : 1351607182000000000,
108
      "category" : null,
109
      "kind" : 0,
110
      "data" : { "plugin_specific_data" : "go_here" }
111
  },
112
  {
113
      "name" : "AnotherDataCollector",
114
      "version" : "B",
115
      "format_version" : 7,
116
      "timestamp" : 1351609526123854000,
117
      "category" : "storage",
118
      "kind" : 1,
119
      "data" : { "status" : { "code" : 1,
120
                              "message" : "Error on disk 2"
121
                            },
122
                 "plugin_specific" : "data",
123
                 "some_late_data" : { "timestamp" : 1351609526123942720,
124
                                      ...
125
                                    }
126
               }
127
  }
128
  ]
129

    
130
Performance reporting collectors
131
++++++++++++++++++++++++++++++++
132

    
133
These collectors only provide data about some component of the system, without
134
giving any interpretation over their meaning.
135

    
136
The value of the ``kind`` field of the report will be ``0``.
137

    
138
Status reporting collectors
139
+++++++++++++++++++++++++++
140

    
141
These collectors will provide information about the status of some
142
component of ganeti, or managed by ganeti.
143

    
144
The value of their ``kind`` field will be ``1``.
145

    
146
The rationale behind this kind of collectors is that there are some situations
147
where exporting data about the underlying subsystems would expose potential
148
issues. But if Ganeti itself is able (and going) to fix the problem, conflicts
149
might arise between Ganeti and something/somebody else trying to fix the same
150
problem.
151
Also, some external monitoring systems might not be aware of the internals of a
152
particular subsystem (e.g.: DRBD) and might only exploit the high level
153
response of its data collector, alerting an administrator if anything is wrong.
154
Still, completely hiding the underlying data is not a good idea, as they might
155
still be of use in some cases. So status reporting plugins will provide two
156
output modes: one just exporting a high level information about the status,
157
and one also exporting all the data they gathered.
158
The default output mode will be the status-only one. Through a command line
159
parameter (for stand-alone data collectors) or through the HTTP request to the
160
monitoring agent
161
(when collectors are executed as part of it) the verbose output mode providing
162
all the data can be selected.
163

    
164
When exporting just the status each status reporting collector will provide,
165
in its ``data`` section, at least the following field:
166

    
167
``status``
168
  summarizes the status of the component being monitored and consists of two
169
  subfields:
170

    
171
  ``code``
172
    It assumes a numeric value, encoded in such a way to allow using a bitset
173
    to easily distinguish which states are currently present in the whole cluster.
174
    If the bitwise OR of all the ``status`` fields is 0, the cluster is
175
    completely healty.
176
    The status codes are as follows:
177

    
178
    ``0``
179
      The collector can determine that everything is working as
180
      intended.
181

    
182
    ``1``
183
      Something is temporarily wrong but it is being automatically fixed by
184
      Ganeti.
185
      There is no need of external intervention.
186

    
187
    ``2``
188
      The collector has failed to understand whether the status is good or
189
      bad. Further analysis is required. Interpret this status as a
190
      potentially dangerous situation.
191

    
192
    ``4``
193
      The collector can determine that something is wrong and Ganeti has no
194
      way to fix it autonomously. External intervention is required.
195

    
196
  ``message``
197
    A message to better explain the reason of the status.
198
    The exact format of the message string is data collector dependent.
199

    
200
    The field is mandatory, but the content can be an empty string if the
201
    ``code`` is ``0`` (working as intended) or ``1`` (being fixed
202
    automatically).
203

    
204
    If the status code is ``2``, the message should specify what has gone
205
    wrong.
206
    If the status code is ``4``, the message shoud explain why it was not
207
    possible to determine a proper status.
208

    
209
The ``data`` section will also contain all the fields describing the gathered
210
data, according to a collector-specific format.
211

    
212
Instance status
213
+++++++++++++++
214

    
215
At the moment each node knows which instances are running on it, which
216
instances it is primary for, but not the cause why an instance might not
217
be running. On the other hand we don't want to distribute full instance
218
"admin" status information to all nodes, because of the performance
219
impact this would have.
220

    
221
As such we propose that:
222

    
223
- Any operation that can affect instance status will have an optional
224
  "reason" attached to it (at opcode level). This can be used for
225
  example to distinguish an admin request, from a scheduled maintenance
226
  or an automated tool's work. If this reason is not passed, Ganeti will
227
  just use the information it has about the source of the request.
228
  This reason information will be structured according to the
229
  :doc:`Ganeti reason trail <design-reason-trail>` design document.
230
- RPCs that affect the instance status will be changed so that the
231
  "reason" and the version of the config object they ran on is passed to
232
  them. They will then export the new expected instance status, together
233
  with the associated reason and object version to the status report
234
  system, which then will export those themselves.
235

    
236
Monitoring and auditing systems can then use the reason to understand
237
the cause of an instance status, and they can use the timestamp to
238
understand the freshness of their data even in the absence of an atomic
239
cross-node reporting: for example if they see an instance "up" on a node
240
after seeing it running on a previous one, they can compare these values
241
to understand which data is freshest, and repoll the "older" node. Of
242
course if they keep seeing this status this represents an error (either
243
an instance continuously "flapping" between nodes, or an instance is
244
constantly up on more than one), which should be reported and acted
245
upon.
246

    
247
The instance status will be on each node, for the instances it is
248
primary for, and its ``data`` section of the report will contain a list
249
of instances, named ``instances``, with at least the following fields for
250
each instance:
251

    
252
``name``
253
  The name of the instance.
254

    
255
``uuid``
256
  The UUID of the instance (stable on name change).
257

    
258
``admin_state``
259
  The status of the instance (up/down/offline) as requested by the admin.
260

    
261
``actual_state``
262
  The actual status of the instance. It can be ``up``, ``down``, or
263
  ``hung`` if the instance is up but it appears to be completely stuck.
264

    
265
``uptime``
266
  The uptime of the instance (if it is up, "null" otherwise).
267

    
268
``mtime``
269
  The timestamp of the last known change to the instance state.
270

    
271
``state_reason``
272
  The last known reason for state change of the instance, described according
273
  to the JSON representation of a reason trail, as detailed in the :doc:`reason
274
  trail design document <design-reason-trail>`.
275

    
276
``status``
277
  It represents the status of the instance, and its format is the same as that
278
  of the ``status`` field of `Status reporting collectors`_.
279

    
280
Each hypervisor should provide its own instance status data collector, possibly
281
with the addition of more, specific, fields.
282
The ``category`` field of all of them will be ``instance``.
283
The ``kind`` field will be ``1``.
284

    
285
Note that as soon as a node knows it's not the primary anymore for an
286
instance it will stop reporting status for it: this means the instance
287
will either disappear, if it has been deleted, or appear on another
288
node, if it's been moved.
289

    
290
The ``code`` of the ``status`` field of the report of the Instance status data
291
collector will be:
292

    
293
``0``
294
  if ``status`` is ``0`` for all the instances it is reporting about.
295

    
296
``1``
297
  otherwise.
298

    
299
Storage status
300
++++++++++++++
301

    
302
The storage status collectors will be a series of data collectors
303
(drbd, rbd, plain, file) that will gather data about all the storage types
304
for the current node (this is right now hardcoded to the enabled storage
305
types, and in the future tied to the enabled storage pools for the nodegroup).
306

    
307
The ``name`` of each of these collector will reflect what storage type each of
308
them refers to.
309

    
310
The ``category`` field of these collector will be ``storage``.
311

    
312
The ``kind`` field will be ``1`` (`Status reporting collectors`_).
313

    
314
The ``data`` section of the report will provide at least the following fields:
315

    
316
``free``
317
  The amount of free space (in KBytes).
318

    
319
``used``
320
  The amount of used space (in KBytes).
321

    
322
``total``
323
  The total visible space (in KBytes).
324

    
325
Each specific storage type might provide more type-specific fields.
326

    
327
In case of error, the ``message`` subfield of the ``status`` field of the
328
report of the instance status collector will disclose the nature of the error
329
as a type specific information. Examples of these are "backend pv unavailable"
330
for lvm storage, "unreachable" for network based storage or "filesystem error"
331
for filesystem based implementations.
332

    
333
DRBD status
334
***********
335

    
336
This data collector will run only on nodes where DRBD is actually
337
present and it will gather information about DRBD devices.
338

    
339
Its ``kind`` in the report will be ``1`` (`Status reporting collectors`_).
340

    
341
Its ``category`` field in the report will contain the value ``storage``.
342

    
343
When executed in verbose mode, the ``data`` section of the report of this
344
collector will provide the following fields:
345

    
346
``versionInfo``
347
  Information about the DRBD version number, given by a combination of
348
  any (but at least one) of the following fields:
349

    
350
  ``version``
351
    The DRBD driver version.
352

    
353
  ``api``
354
    The API version number.
355

    
356
  ``proto``
357
    The protocol version.
358

    
359
  ``srcversion``
360
    The version of the source files.
361

    
362
  ``gitHash``
363
    Git hash of the source files.
364

    
365
  ``buildBy``
366
    Who built the binary, and, optionally, when.
367

    
368
``device``
369
  A list of structures, each describing a DRBD device (a minor) and containing
370
  the following fields:
371

    
372
  ``minor``
373
    The device minor number.
374

    
375
  ``connectionState``
376
    The state of the connection. If it is "Unconfigured", all the following
377
    fields are not present.
378

    
379
  ``localRole``
380
    The role of the local resource.
381

    
382
  ``remoteRole``
383
    The role of the remote resource.
384

    
385
  ``localState``
386
    The status of the local disk.
387

    
388
  ``remoteState``
389
    The status of the remote disk.
390

    
391
  ``replicationProtocol``
392
    The replication protocol being used.
393

    
394
  ``ioFlags``
395
    The input/output flags.
396

    
397
  ``perfIndicators``
398
    The performance indicators. This field will contain the following
399
    sub-fields:
400

    
401
    ``networkSend``
402
      KiB of data sent on the network.
403

    
404
    ``networkReceive``
405
      KiB of data received from the network.
406

    
407
    ``diskWrite``
408
      KiB of data written on local disk.
409

    
410
    ``diskRead``
411
      KiB of date read from the local disk.
412

    
413
    ``activityLog``
414
      Number of updates of the activity log.
415

    
416
    ``bitMap``
417
      Number of updates to the bitmap area of the metadata.
418

    
419
    ``localCount``
420
      Number of open requests to the local I/O subsystem.
421

    
422
    ``pending``
423
      Number of requests sent to the partner but not yet answered.
424

    
425
    ``unacknowledged``
426
      Number of requests received by the partner but still to be answered.
427

    
428
    ``applicationPending``
429
      Num of block input/output requests forwarded to DRBD but that have not yet
430
      been answered.
431

    
432
    ``epochs``
433
      (Optional) Number of epoch objects. Not provided by all DRBD versions.
434

    
435
    ``writeOrder``
436
      (Optional) Currently used write ordering method. Not provided by all DRBD
437
      versions.
438

    
439
    ``outOfSync``
440
      (Optional) KiB of storage currently out of sync. Not provided by all DRBD
441
      versions.
442

    
443
  ``syncStatus``
444
    (Optional) The status of the synchronization of the disk. This is present
445
    only if the disk is being synchronized, and includes the following fields:
446

    
447
    ``percentage``
448
      The percentage of synchronized data.
449

    
450
    ``progress``
451
      How far the synchronization is. Written as "x/y", where x and y are
452
      integer numbers expressed in the measurement unit stated in
453
      ``progressUnit``
454

    
455
    ``progressUnit``
456
      The measurement unit for the progress indicator.
457

    
458
    ``timeToFinish``
459
      The expected time before finishing the synchronization.
460

    
461
    ``speed``
462
      The speed of the synchronization.
463

    
464
    ``want``
465
      The desiderd speed of the synchronization.
466

    
467
    ``speedUnit``
468
      The measurement unit of the ``speed`` and ``want`` values. Expressed
469
      as "size/time".
470

    
471
  ``instance``
472
    The name of the Ganeti instance this disk is associated to.
473

    
474

    
475
Ganeti daemons status
476
+++++++++++++++++++++
477

    
478
Ganeti will report what information it has about its own daemons.
479
This should allow identifying possible problems with the Ganeti system itself:
480
for example memory leaks, crashes and high resource utilization should be
481
evident by analyzing this information.
482

    
483
The ``kind`` field will be ``1`` (`Status reporting collectors`_).
484

    
485
Each daemon will have its own data collector, and each of them will have
486
a ``category`` field valued ``daemon``.
487

    
488
When executed in verbose mode, their data section will include at least:
489

    
490
``memory``
491
  The amount of used memory.
492

    
493
``size_unit``
494
  The measurement unit used for the memory.
495

    
496
``uptime``
497
  The uptime of the daemon.
498

    
499
``CPU usage``
500
  How much cpu the daemon is using (percentage).
501

    
502
Any other daemon-specific information can be included as well in the ``data``
503
section.
504

    
505
Hypervisor resources report
506
+++++++++++++++++++++++++++
507

    
508
Each hypervisor has a view of system resources that sometimes is
509
different than the one the OS sees (for example in Xen the Node OS,
510
running as Dom0, has access to only part of those resources). In this
511
section we'll report all information we can in a "non hypervisor
512
specific" way. Each hypervisor can then add extra specific information
513
that is not generic enough be abstracted.
514

    
515
The ``kind`` field will be ``0`` (`Performance reporting collectors`_).
516

    
517
Each of the hypervisor data collectory will be of ``category``: ``hypervisor``.
518

    
519
Node OS resources report
520
++++++++++++++++++++++++
521

    
522
Since Ganeti assumes it's running on Linux, it's useful to export some
523
basic information as seen by the host system.
524

    
525
The ``category`` field of the report will be ``null``.
526

    
527
The ``kind`` field will be ``0`` (`Performance reporting collectors`_).
528

    
529
The ``data`` section will include:
530

    
531
``cpu_number``
532
  The number of available cpus.
533

    
534
``cpus``
535
  A list with one element per cpu, showing its average load.
536

    
537
``memory``
538
  The current view of memory (free, used, cached, etc.)
539

    
540
``filesystem``
541
  A list with one element per filesystem, showing a summary of the
542
  total/available space.
543

    
544
``NICs``
545
  A list with one element per network interface, showing the amount of
546
  sent/received data, error rate, IP address of the interface, etc.
547

    
548
``versions``
549
  A map using the name of a component Ganeti interacts (Linux, drbd,
550
  hypervisor, etc) as the key and its version number as the value.
551

    
552
Note that we won't go into any hardware specific details (e.g. querying a
553
node RAID is outside the scope of this, and can be implemented as a
554
plugin) but we can easily just report the information above, since it's
555
standard enough across all systems.
556

    
557
Format of the query
558
-------------------
559

    
560
The queries to the monitoring agent will be HTTP GET requests on port 1815.
561
The answer will be encoded in JSON format and will depend on the specific
562
accessed resource.
563

    
564
If a request is sent to a non-existing resource, a 404 error will be returned by
565
the HTTP server.
566

    
567
The following paragraphs will present the existing resources supported by the
568
current protocol version, that is version 1.
569

    
570
``/``
571
+++++
572
The root resource. It will return the list of the supported protocol version
573
numbers.
574

    
575
Currently, this will include only version 1.
576

    
577
``/1``
578
++++++
579
Not an actual resource per-se, it is the root of all the resources of protocol
580
version 1.
581

    
582
If requested through GET, the null JSON value will be returned.
583

    
584
``/1/list/collectors``
585
++++++++++++++++++++++
586
Returns a list of tuples (kind, category, name) showing all the collectors
587
available in the system.
588

    
589
``/1/report/all``
590
+++++++++++++++++
591
A list of the reports of all the data collectors, as described in the section
592
`Format of the report`_.
593

    
594
`Status reporting collectors`_ will provide their output in non-verbose format.
595
The verbose format can be requested by adding the parameter ``verbose=1`` to the
596
request.
597

    
598
``/1/report/[category]/[collector_name]``
599
+++++++++++++++++++++++++++++++++++++++++
600
Returns the report of the collector ``[collector_name]`` that belongs to the
601
specified ``[category]``.
602

    
603
The ``category`` has to be written in lowercase.
604

    
605
If a collector does not belong to any category, ``default`` will have to be
606
used as the value for ``[category]``.
607

    
608
`Status reporting collectors`_ will provide their output in non-verbose format.
609
The verbose format can be requested by adding the parameter ``verbose=1`` to the
610
request.
611

    
612
Instance disk status propagation
613
--------------------------------
614

    
615
As for the instance status Ganeti has now only partial information about
616
its instance disks: in particular each node is unaware of the disk to
617
instance mapping, that exists only on the master.
618

    
619
For this design doc we plan to fix this by changing all RPCs that create
620
a backend storage or that put an already existing one in use and passing
621
the relevant instance to the node. The node can then export these to the
622
status reporting tool.
623

    
624
While we haven't implemented these RPC changes yet, we'll use Confd to
625
fetch this information in the data collectors.
626

    
627
Plugin system
628
-------------
629

    
630
The monitoring system will be equipped with a plugin system that can
631
export specific local information through it.
632

    
633
The plugin system is expected to be used by local installations to
634
export any installation specific information that they want to be
635
monitored, about either hardware or software on their systems.
636

    
637
The plugin system will be in the form of either scripts or binaries whose output
638
will be inserted in the report.
639

    
640
Eventually support for other kinds of plugins might be added as well, such as
641
plain text files which will be inserted into the report, or local unix or
642
network sockets from which the information has to be read.  This should allow
643
most flexibility for implementing an efficient system, while being able to keep
644
it as simple as possible.
645

    
646
Data collectors
647
---------------
648

    
649
In order to ease testing as well as to make it simple to reuse this
650
subsystem it will be possible to run just the "data collectors" on each
651
node without passing through the agent daemon.
652

    
653
If a data collector is run independently, it should print on stdout its
654
report, according to the format corresponding to a single data collector
655
report object, as described in the previous paragraphs.
656

    
657
Mode of operation
658
-----------------
659

    
660
In order to be able to report information fast the monitoring agent
661
daemon will keep an in-memory or on-disk cache of the status, which will
662
be returned when queries are made. The status system will then
663
periodically check resources to make sure the status is up to date.
664

    
665
Different parts of the report will be queried at different speeds. These
666
will depend on:
667
- how often they vary (or we expect them to vary)
668
- how fast they are to query
669
- how important their freshness is
670

    
671
Of course the last parameter is installation specific, and while we'll
672
try to have defaults, it will be configurable. The first two instead we
673
can use adaptively to query a certain resource faster or slower
674
depending on those two parameters.
675

    
676
When run as stand-alone binaries, the data collector will not using any
677
caching system, and just fetch and return the data immediately.
678

    
679
Implementation place
680
--------------------
681

    
682
The status daemon will be implemented as a standalone Haskell daemon. In
683
the future it should be easy to merge multiple daemons into one with
684
multiple entry points, should we find out it saves resources and doesn't
685
impact functionality.
686

    
687
The libekg library should be looked at for easily providing metrics in
688
json format.
689

    
690
Implementation order
691
--------------------
692

    
693
We will implement the agent system in this order:
694

    
695
- initial example data collectors (eg. for drbd and instance status).
696
- initial daemon for exporting data, integrating the existing collectors
697
- plugin system
698
- RPC updates for instance status reasons and disk to instance mapping
699
- cache layer for the daemon
700
- more data collectors
701

    
702

    
703
Future work
704
===========
705

    
706
As a future step it can be useful to "centralize" all this reporting
707
data on a single place. This for example can be just the master node, or
708
all the master candidates. We will evaluate doing this after the first
709
node-local version has been developed and tested.
710

    
711
Another possible change is replacing the "read-only" RPCs with queries
712
to the agent system, thus having only one way of collecting information
713
from the nodes from a monitoring system and for Ganeti itself.
714

    
715
One extra feature we may need is a way to query for only sub-parts of
716
the report (eg. instances status only). This can be done by passing
717
arguments to the HTTP GET, which will be defined when we get to this
718
funtionality.
719

    
720
Finally the :doc:`autorepair system design <design-autorepair>`. system
721
(see its design) can be expanded to use the monitoring agent system as a
722
source of information to decide which repairs it can perform.
723

    
724
.. vim: set textwidth=72 :
725
.. Local Variables:
726
.. mode: rst
727
.. fill-column: 72
728
.. End: