1 =======================
2 Ganeti monitoring agent
3 =======================
5 .. contents:: :depth: 4
7 This is a design document detailing the implementation of a Ganeti
8 monitoring agent report system, that can be queried by a monitoring
9 system to calculate health information for a Ganeti cluster.
11 Current state and shortcomings
12 ==============================
14 There is currently no monitoring support in Ganeti. While we don't want
15 to build something like Nagios or Pacemaker as part of Ganeti, it would
16 be useful if such tools could easily extract information from a Ganeti
17 machine in order to take actions (example actions include logging an
18 outage for future reporting or alerting a person or system about it).
23 Each Ganeti node should export a status page that can be queried by a
24 monitoring system. Such status page will be exported on a network port
25 and will be encoded in JSON (simple text) over HTTP.
27 The choice of JSON is obvious as we already depend on it in Ganeti and
28 thus we don't need to add extra libraries to use it, as opposed to what
29 would happen for XML or some other markup format.
31 Location of agent report
32 ------------------------
34 The report will be available from all nodes, and be concerned for all
35 node-local resources. This allows more real-time information to be
36 available, at the cost of querying all nodes.
41 The monitoring agent system will report on the following basic information:
44 - Instance disk status
45 - Status of storage for instances
46 - Ganeti daemons status, CPU usage, memory footprint
47 - Hypervisor resources report (memory, CPU, network interfaces)
48 - Node OS resources report (memory, CPU, network interfaces)
49 - Node OS CPU load average report
50 - Information from a plugin system
55 The report of the will be in JSON format, and it will present an array
57 Each report object will be produced by a specific data collector.
58 Each report object includes some mandatory fields, to be provided by all
62 The name of the data collector that produced this part of the report.
63 It is supposed to be unique inside a report.
66 The version of the data collector that produces this part of the
67 report. Built-in data collectors (as opposed to those implemented as
68 plugins) should have "B" as the version number.
71 The format of what is represented in the "data" field for each data
72 collector might change over time. Every time this happens, the
73 format_version should be changed, so that who reads the report knows
74 what format to expect, and how to correctly interpret it.
77 The time when the reported data were gathered. It has to be expressed
78 in nanoseconds since the unix epoch (0:00:00 January 01, 1970). If not
79 enough precision is available (or needed) it can be padded with
80 zeroes. If a report object needs multiple timestamps, it can add more
81 and/or override this one inside its own "data" section.
84 A collector can belong to a given category of collectors (e.g.: storage
85 collectors, daemon collector). This means that it will have to provide a
86 minumum set of prescribed fields, as documented for each category.
87 This field will contain the name of the category the collector belongs to,
88 if any, or just the ``null`` value.
91 Two kinds of collectors are possible:
92 `Performance reporting collectors`_ and `Status reporting collectors`_.
93 The respective paragraphs will describe them and the value of this field.
96 This field contains all the data generated by the specific data collector,
97 in its own independently defined format. The monitoring agent could check
98 this syntactically (according to the JSON specifications) but not
101 Here follows a minimal example of a report::
105 "name" : "TheCollectorIdentifier",
107 "format_version" : 1,
108 "timestamp" : 1351607182000000000,
111 "data" : { "plugin_specific_data" : "go_here" }
114 "name" : "AnotherDataCollector",
116 "format_version" : 7,
117 "timestamp" : 1351609526123854000,
118 "category" : "storage",
120 "data" : { "status" : { "code" : 1,
121 "message" : "Error on disk 2"
123 "plugin_specific" : "data",
124 "some_late_data" : { "timestamp" : 1351609526123942720,
131 Performance reporting collectors
132 ++++++++++++++++++++++++++++++++
134 These collectors only provide data about some component of the system, without
135 giving any interpretation over their meaning.
137 The value of the ``kind`` field of the report will be ``0``.
139 Status reporting collectors
140 +++++++++++++++++++++++++++
142 These collectors will provide information about the status of some
143 component of ganeti, or managed by ganeti.
145 The value of their ``kind`` field will be ``1``.
147 The rationale behind this kind of collectors is that there are some situations
148 where exporting data about the underlying subsystems would expose potential
149 issues. But if Ganeti itself is able (and going) to fix the problem, conflicts
150 might arise between Ganeti and something/somebody else trying to fix the same
152 Also, some external monitoring systems might not be aware of the internals of a
153 particular subsystem (e.g.: DRBD) and might only exploit the high level
154 response of its data collector, alerting an administrator if anything is wrong.
155 Still, completely hiding the underlying data is not a good idea, as they might
156 still be of use in some cases. So status reporting plugins will provide two
157 output modes: one just exporting a high level information about the status,
158 and one also exporting all the data they gathered.
159 The default output mode will be the status-only one. Through a command line
160 parameter (for stand-alone data collectors) or through the HTTP request to the
162 (when collectors are executed as part of it) the verbose output mode providing
163 all the data can be selected.
165 When exporting just the status each status reporting collector will provide,
166 in its ``data`` section, at least the following field:
169 summarizes the status of the component being monitored and consists of two
173 It assumes a numeric value, encoded in such a way to allow using a bitset
174 to easily distinguish which states are currently present in the whole
175 cluster. If the bitwise OR of all the ``status`` fields is 0, the cluster
176 is completely healty.
177 The status codes are as follows:
180 The collector can determine that everything is working as
184 Something is temporarily wrong but it is being automatically fixed by
186 There is no need of external intervention.
189 The collector has failed to understand whether the status is good or
190 bad. Further analysis is required. Interpret this status as a
191 potentially dangerous situation.
194 The collector can determine that something is wrong and Ganeti has no
195 way to fix it autonomously. External intervention is required.
198 A message to better explain the reason of the status.
199 The exact format of the message string is data collector dependent.
201 The field is mandatory, but the content can be an empty string if the
202 ``code`` is ``0`` (working as intended) or ``1`` (being fixed
205 If the status code is ``2``, the message should specify what has gone
207 If the status code is ``4``, the message shoud explain why it was not
208 possible to determine a proper status.
210 The ``data`` section will also contain all the fields describing the gathered
211 data, according to a collector-specific format.
216 At the moment each node knows which instances are running on it, which
217 instances it is primary for, but not the cause why an instance might not
218 be running. On the other hand we don't want to distribute full instance
219 "admin" status information to all nodes, because of the performance
220 impact this would have.
222 As such we propose that:
224 - Any operation that can affect instance status will have an optional
225 "reason" attached to it (at opcode level). This can be used for
226 example to distinguish an admin request, from a scheduled maintenance
227 or an automated tool's work. If this reason is not passed, Ganeti will
228 just use the information it has about the source of the request.
229 This reason information will be structured according to the
230 :doc:`Ganeti reason trail <design-reason-trail>` design document.
231 - RPCs that affect the instance status will be changed so that the
232 "reason" and the version of the config object they ran on is passed to
233 them. They will then export the new expected instance status, together
234 with the associated reason and object version to the status report
235 system, which then will export those themselves.
237 Monitoring and auditing systems can then use the reason to understand
238 the cause of an instance status, and they can use the timestamp to
239 understand the freshness of their data even in the absence of an atomic
240 cross-node reporting: for example if they see an instance "up" on a node
241 after seeing it running on a previous one, they can compare these values
242 to understand which data is freshest, and repoll the "older" node. Of
243 course if they keep seeing this status this represents an error (either
244 an instance continuously "flapping" between nodes, or an instance is
245 constantly up on more than one), which should be reported and acted
248 The instance status will be on each node, for the instances it is
249 primary for, and its ``data`` section of the report will contain a list
250 of instances, named ``instances``, with at least the following fields for
254 The name of the instance.
257 The UUID of the instance (stable on name change).
260 The status of the instance (up/down/offline) as requested by the admin.
263 The actual status of the instance. It can be ``up``, ``down``, or
264 ``hung`` if the instance is up but it appears to be completely stuck.
267 The uptime of the instance (if it is up, "null" otherwise).
270 The timestamp of the last known change to the instance state.
273 The last known reason for state change of the instance, described according
274 to the JSON representation of a reason trail, as detailed in the :doc:`reason
275 trail design document <design-reason-trail>`.
278 It represents the status of the instance, and its format is the same as that
279 of the ``status`` field of `Status reporting collectors`_.
281 Each hypervisor should provide its own instance status data collector, possibly
282 with the addition of more, specific, fields.
283 The ``category`` field of all of them will be ``instance``.
284 The ``kind`` field will be ``1``.
286 Note that as soon as a node knows it's not the primary anymore for an
287 instance it will stop reporting status for it: this means the instance
288 will either disappear, if it has been deleted, or appear on another
289 node, if it's been moved.
291 The ``code`` of the ``status`` field of the report of the Instance status data
295 if ``status`` is ``0`` for all the instances it is reporting about.
303 The storage collectors will be a series of data collectors
304 that will gather data about storage for the current node. The collection
305 will be performed at different granularity and abstraction levels, from
306 the physical disks, to partitions, logical volumes and to the specific
307 storage types used by Ganeti itself (drbd, rbd, plain, file).
309 The ``name`` of each of these collector will reflect what storage type each of
312 The ``category`` field of these collector will be ``storage``.
314 The ``kind`` field will depend on the specific collector.
316 Each ``storage`` collector's ``data`` section will provide collector-specific
319 The various storage collectors will provide keys to join the data they provide,
320 in order to allow the user to get a better understanding of the system. E.g.:
321 through device names, or instance names.
326 This storage data collector will gather information about the status of the
327 disks installed in the system, as listed in the /proc/diskstats file. This means
328 that not only physical hard drives, but also ramdisks and loopback devices will
331 Its ``kind`` in the report will be ``0`` (`Performance reporting collectors`_).
333 Its ``category`` field in the report will contain the value ``storage``.
335 When executed in verbose mode, the ``data`` section of the report of this
336 collector will be a list of items, each representing one disk, each providing
337 the following fields:
340 The major number of the device.
343 The minor number of the device.
346 The name of the device.
349 This is the total number of reads completed successfully.
352 Reads which are adjacent to each other may be merged for efficiency. Thus
353 two 4K reads may become one 8K read before it is ultimately handed to the
354 disk, and so it will be counted (and queued) as only one I/O. This field
355 specifies how often this was done.
358 This is the total number of sectors read successfully.
361 This is the total number of milliseconds spent by all reads.
364 This is the total number of writes completed successfully.
367 Writes which are adjacent to each other may be merged for efficiency. Thus
368 two 4K writes may become one 8K read before it is ultimately handed to the
369 disk, and so it will be counted (and queued) as only one I/O. This field
370 specifies how often this was done.
373 This is the total number of sectors written successfully.
376 This is the total number of milliseconds spent by all writes.
379 The number of I/Os currently in progress.
380 The only field that should go to zero, it is incremented as requests are
381 given to appropriate struct request_queue and decremented as they finish.
384 The number of milliseconds spent doing I/Os. This field increases so long
385 as field ``IOs`` is nonzero.
388 The weighted number of milliseconds spent doing I/Os.
389 This field is incremented at each I/O start, I/O completion, I/O merge,
390 or read of these stats by the number of I/Os in progress (field ``IOs``)
391 times the number of milliseconds spent doing I/O since the last update of
392 this field. This can provide an easy measure of both I/O completion time
393 and the backlog that may be accumulating.
395 Logical Volume collector
396 ************************
398 This data collector will gather information about the attributes of logical
399 volumes present in the system.
401 Its ``kind`` in the report will be ``0`` (`Performance reporting collectors`_).
403 Its ``category`` field in the report will contain the value ``storage``.
405 The ``data`` section of the report of this collector will be a list of items,
406 each representing one logical volume and providing the following fields:
409 The UUID of the logical volume.
412 The name of the logical volume.
415 The attributes of the logical volume.
418 Persistent major number or -1 if not persistent.
421 Persistent minor number or -1 if not persistent.
424 Currently assigned major number or -1 if LV is not active.
427 Currently assigned minor number or -1 if LV is not active.
433 Number of segments in LV.
439 Kernel device-mapper modules required for this LV, if any.
442 Unique identifier of the volume group.
445 Name of the volume group.
451 Offset within the LVto the start of the segment in bytes.
454 Offset within the LV to the start of the segment in physical extents.
457 Size of the segment in bytes.
460 Tags for the segment, if any.
463 Ranges of Physical Extents of underlying devices in lvs command line format.
466 Underlying devices used with starting extent numbers.
469 The name of the instance this LV is used by, or ``null`` if it was not
470 possible to determine it.
475 This data collector will run only on nodes where DRBD is actually
476 present and it will gather information about DRBD devices.
478 Its ``kind`` in the report will be ``1`` (`Status reporting collectors`_).
480 Its ``category`` field in the report will contain the value ``storage``.
482 When executed in verbose mode, the ``data`` section of the report of this
483 collector will provide the following fields:
486 Information about the DRBD version number, given by a combination of
487 any (but at least one) of the following fields:
490 The DRBD driver version.
493 The API version number.
496 The protocol version.
499 The version of the source files.
502 Git hash of the source files.
505 Who built the binary, and, optionally, when.
508 A list of structures, each describing a DRBD device (a minor) and containing
509 the following fields:
512 The device minor number.
515 The state of the connection. If it is "Unconfigured", all the following
516 fields are not present.
519 The role of the local resource.
522 The role of the remote resource.
525 The status of the local disk.
528 The status of the remote disk.
530 ``replicationProtocol``
531 The replication protocol being used.
534 The input/output flags.
537 The performance indicators. This field will contain the following
541 KiB of data sent on the network.
544 KiB of data received from the network.
547 KiB of data written on local disk.
550 KiB of date read from the local disk.
553 Number of updates of the activity log.
556 Number of updates to the bitmap area of the metadata.
559 Number of open requests to the local I/O subsystem.
562 Number of requests sent to the partner but not yet answered.
565 Number of requests received by the partner but still to be answered.
567 ``applicationPending``
568 Num of block input/output requests forwarded to DRBD but that have not yet
572 (Optional) Number of epoch objects. Not provided by all DRBD versions.
575 (Optional) Currently used write ordering method. Not provided by all DRBD
579 (Optional) KiB of storage currently out of sync. Not provided by all DRBD
583 (Optional) The status of the synchronization of the disk. This is present
584 only if the disk is being synchronized, and includes the following fields:
587 The percentage of synchronized data.
590 How far the synchronization is. Written as "x/y", where x and y are
591 integer numbers expressed in the measurement unit stated in
595 The measurement unit for the progress indicator.
598 The expected time before finishing the synchronization.
601 The speed of the synchronization.
604 The desiderd speed of the synchronization.
607 The measurement unit of the ``speed`` and ``want`` values. Expressed
611 The name of the Ganeti instance this disk is associated to.
614 Ganeti daemons status
615 +++++++++++++++++++++
617 Ganeti will report what information it has about its own daemons.
618 This should allow identifying possible problems with the Ganeti system itself:
619 for example memory leaks, crashes and high resource utilization should be
620 evident by analyzing this information.
622 The ``kind`` field will be ``1`` (`Status reporting collectors`_).
624 Each daemon will have its own data collector, and each of them will have
625 a ``category`` field valued ``daemon``.
627 When executed in verbose mode, their data section will include at least:
630 The amount of used memory.
633 The measurement unit used for the memory.
636 The uptime of the daemon.
639 How much cpu the daemon is using (percentage).
641 Any other daemon-specific information can be included as well in the ``data``
644 Hypervisor resources report
645 +++++++++++++++++++++++++++
647 Each hypervisor has a view of system resources that sometimes is
648 different than the one the OS sees (for example in Xen the Node OS,
649 running as Dom0, has access to only part of those resources). In this
650 section we'll report all information we can in a "non hypervisor
651 specific" way. Each hypervisor can then add extra specific information
652 that is not generic enough be abstracted.
654 The ``kind`` field will be ``0`` (`Performance reporting collectors`_).
656 Each of the hypervisor data collectory will be of ``category``: ``hypervisor``.
658 Node OS resources report
659 ++++++++++++++++++++++++
661 Since Ganeti assumes it's running on Linux, it's useful to export some
662 basic information as seen by the host system.
664 The ``category`` field of the report will be ``null``.
666 The ``kind`` field will be ``0`` (`Performance reporting collectors`_).
668 The ``data`` section will include:
671 The number of available cpus.
674 A list with one element per cpu, showing its average load.
677 The current view of memory (free, used, cached, etc.)
680 A list with one element per filesystem, showing a summary of the
681 total/available space.
684 A list with one element per network interface, showing the amount of
685 sent/received data, error rate, IP address of the interface, etc.
688 A map using the name of a component Ganeti interacts (Linux, drbd,
689 hypervisor, etc) as the key and its version number as the value.
691 Note that we won't go into any hardware specific details (e.g. querying a
692 node RAID is outside the scope of this, and can be implemented as a
693 plugin) but we can easily just report the information above, since it's
694 standard enough across all systems.
696 Node OS CPU load average report
697 +++++++++++++++++++++++++++++++
699 This data collector will export CPU load statistics as seen by the host
700 system. Apart from using the data from an external monitoring system we
701 can also use the data to improve instance allocation and/or the Ganeti
702 cluster balance. To compute the CPU load average we will use a number of
703 values collected inside a time window. The collection process will be
704 done by an independent thread (see `Mode of Operation`_).
706 This report is a subset of the previous report (`Node OS resources
707 report`_) and they might eventually get merged, once reporting for the
708 other fields (memory, filesystem, NICs) gets implemented too.
712 The ``category`` field of the report will be ``null``.
714 The ``kind`` field will be ``0`` (`Performance reporting collectors`_).
716 The ``data`` section will include:
719 The number of available cpus.
722 A list with one element per cpu, showing its average load.
725 The total CPU load average as a sum of the all separate cpus.
727 The CPU load report function will get N values, collected by the
728 CPU load collection function and calculate the above averages. Please
729 see the section `Mode of Operation`_ for more information one how the
730 two functions of the data collector interact.
735 .. include:: monitoring-query-format.rst
737 Instance disk status propagation
738 --------------------------------
740 As for the instance status Ganeti has now only partial information about
741 its instance disks: in particular each node is unaware of the disk to
742 instance mapping, that exists only on the master.
744 For this design doc we plan to fix this by changing all RPCs that create
745 a backend storage or that put an already existing one in use and passing
746 the relevant instance to the node. The node can then export these to the
747 status reporting tool.
749 While we haven't implemented these RPC changes yet, we'll use Confd to
750 fetch this information in the data collectors.
755 The monitoring system will be equipped with a plugin system that can
756 export specific local information through it.
758 The plugin system is expected to be used by local installations to
759 export any installation specific information that they want to be
760 monitored, about either hardware or software on their systems.
762 The plugin system will be in the form of either scripts or binaries whose output
763 will be inserted in the report.
765 Eventually support for other kinds of plugins might be added as well, such as
766 plain text files which will be inserted into the report, or local unix or
767 network sockets from which the information has to be read. This should allow
768 most flexibility for implementing an efficient system, while being able to keep
769 it as simple as possible.
774 In order to ease testing as well as to make it simple to reuse this
775 subsystem it will be possible to run just the "data collectors" on each
776 node without passing through the agent daemon.
778 If a data collector is run independently, it should print on stdout its
779 report, according to the format corresponding to a single data collector
780 report object, as described in the previous paragraphs.
785 In order to be able to report information fast the monitoring agent
786 daemon will keep an in-memory or on-disk cache of the status, which will
787 be returned when queries are made. The status system will then
788 periodically check resources to make sure the status is up to date.
790 Different parts of the report will be queried at different speeds. These
792 - how often they vary (or we expect them to vary)
793 - how fast they are to query
794 - how important their freshness is
796 Of course the last parameter is installation specific, and while we'll
797 try to have defaults, it will be configurable. The first two instead we
798 can use adaptively to query a certain resource faster or slower
799 depending on those two parameters.
801 When run as stand-alone binaries, the data collector will not using any
802 caching system, and just fetch and return the data immediately.
804 Since some performance collectors have to operate on a number of values
805 collected in previous times, we need a mechanism independent of the data
806 collector which will trigger the collection of those values and also
807 store them, so that they are available for calculation by the data
810 To collect data periodically, a thread will be created by the monitoring
811 agent which will run the collection function of every data collector
812 that provides one. The values returned by the collection function of
813 the data collector will be saved in an appropriate map, associating each
814 value to the corresponding collector, using the collector's name as the
815 key of the map. This map will be stored in mond's memory.
817 For example: the collection function of the CPU load collector will
818 collect a CPU load value and save it in the map mentioned above. The
819 collection function will be called by the collector thread every t
820 milliseconds. When the report function of the collector is called, it
821 will process the last N values of the map and calculate the
822 corresponding average.
827 The status daemon will be implemented as a standalone Haskell daemon. In
828 the future it should be easy to merge multiple daemons into one with
829 multiple entry points, should we find out it saves resources and doesn't
830 impact functionality.
832 The libekg library should be looked at for easily providing metrics in
838 We will implement the agent system in this order:
840 - initial example data collectors (eg. for drbd and instance status).
841 - initial daemon for exporting data, integrating the existing collectors
843 - RPC updates for instance status reasons and disk to instance mapping
844 - cache layer for the daemon
845 - more data collectors
851 As a future step it can be useful to "centralize" all this reporting
852 data on a single place. This for example can be just the master node, or
853 all the master candidates. We will evaluate doing this after the first
854 node-local version has been developed and tested.
856 Another possible change is replacing the "read-only" RPCs with queries
857 to the agent system, thus having only one way of collecting information
858 from the nodes from a monitoring system and for Ganeti itself.
860 One extra feature we may need is a way to query for only sub-parts of
861 the report (eg. instances status only). This can be done by passing
862 arguments to the HTTP GET, which will be defined when we get to this
865 Finally the :doc:`autorepair system design <design-autorepair>`. system
866 (see its design) can be expanded to use the monitoring agent system as a
867 source of information to decide which repairs it can perform.
869 .. vim: set textwidth=72 :