- Ganeti daemons status, CPU usage, memory footprint
- Hypervisor resources report (memory, CPU, network interfaces)
- Node OS resources report (memory, CPU, network interfaces)
+- Node OS CPU load average report
- Information from a plugin system
Format of the report
report. Built-in data collectors (as opposed to those implemented as
plugins) should have "B" as the version number.
-``formatVersion``
+``format_version``
The format of what is represented in the "data" field for each data
collector might change over time. Every time this happens, the
format_version should be changed, so that who reads the report knows
what format to expect, and how to correctly interpret it.
``timestamp``
- The time when the reported data were gathered. Is has to be expressed
+ The time when the reported data were gathered. It has to be expressed
in nanoseconds since the unix epoch (0:00:00 January 01, 1970). If not
enough precision is available (or needed) it can be padded with
zeroes. If a report object needs multiple timestamps, it can add more
{
"name" : "TheCollectorIdentifier",
"version" : "1.2",
- "formatVersion" : 1,
+ "format_version" : 1,
"timestamp" : 1351607182000000000,
"category" : null,
"kind" : 0,
{
"name" : "AnotherDataCollector",
"version" : "B",
- "formatVersion" : 7,
+ "format_version" : 7,
"timestamp" : 1351609526123854000,
"category" : "storage",
"kind" : 1,
``code``
It assumes a numeric value, encoded in such a way to allow using a bitset
- to easily distinguish which states are currently present in the whole cluster.
- If the bitwise OR of all the ``status`` fields is 0, the cluster is
- completely healty.
+ to easily distinguish which states are currently present in the whole
+ cluster. If the bitwise OR of all the ``status`` fields is 0, the cluster
+ is completely healty.
The status codes are as follows:
``0``
There is no need of external intervention.
``2``
- The collector can determine that something is wrong and Ganeti has no
- way to fix it autonomously. External intervention is required.
-
- ``4``
The collector has failed to understand whether the status is good or
bad. Further analysis is required. Interpret this status as a
potentially dangerous situation.
+ ``4``
+ The collector can determine that something is wrong and Ganeti has no
+ way to fix it autonomously. External intervention is required.
+
``message``
A message to better explain the reason of the status.
The exact format of the message string is data collector dependent.
- The field is mandatory, but the content can be ``null`` if the code is
- ``0`` (working as intended) or ``1`` (being fixed automatically).
+ The field is mandatory, but the content can be an empty string if the
+ ``code`` is ``0`` (working as intended) or ``1`` (being fixed
+ automatically).
If the status code is ``2``, the message should specify what has gone
wrong.
"reason" attached to it (at opcode level). This can be used for
example to distinguish an admin request, from a scheduled maintenance
or an automated tool's work. If this reason is not passed, Ganeti will
- just use the information it has about the source of the request: for
- example a cli shutdown operation will have "cli:shutdown" as a reason,
- a cli failover operation will have "cli:failover". Operations coming
- from the remote API will use "rapi" instead of "cli". Of course
- setting a real site-specific reason is still preferred.
+ just use the information it has about the source of the request.
+ This reason information will be structured according to the
+ :doc:`Ganeti reason trail <design-reason-trail>` design document.
- RPCs that affect the instance status will be changed so that the
"reason" and the version of the config object they ran on is passed to
them. They will then export the new expected instance status, together
The instance status will be on each node, for the instances it is
primary for, and its ``data`` section of the report will contain a list
-of instances, with at least the following fields for each instance:
+of instances, named ``instances``, with at least the following fields for
+each instance:
``name``
The name of the instance.
The timestamp of the last known change to the instance state.
``state_reason``
- The last known reason for state change, described according to the
- following subfields:
-
- ``text``
- Either a user-provided reason (if any), or the name of the command that
- triggered the state change, as a fallback.
-
- ``jobID``
- The ID of the job that caused the state change.
-
- ``source``
- Where the state change was triggered (RAPI, CLI).
+ The last known reason for state change of the instance, described according
+ to the JSON representation of a reason trail, as detailed in the :doc:`reason
+ trail design document <design-reason-trail>`.
``status``
It represents the status of the instance, and its format is the same as that
``1``
otherwise.
-Storage status
-++++++++++++++
+Storage collectors
+++++++++++++++++++
-The storage status collectors will be a series of data collectors
-(drbd, rbd, plain, file) that will gather data about all the storage types
-for the current node (this is right now hardcoded to the enabled storage
-types, and in the future tied to the enabled storage pools for the nodegroup).
+The storage collectors will be a series of data collectors
+that will gather data about storage for the current node. The collection
+will be performed at different granularity and abstraction levels, from
+the physical disks, to partitions, logical volumes and to the specific
+storage types used by Ganeti itself (drbd, rbd, plain, file).
The ``name`` of each of these collector will reflect what storage type each of
them refers to.
The ``category`` field of these collector will be ``storage``.
-The ``kind`` field will be ``1`` (`Status reporting collectors`_).
+The ``kind`` field will depend on the specific collector.
+
+Each ``storage`` collector's ``data`` section will provide collector-specific
+fields.
+
+The various storage collectors will provide keys to join the data they provide,
+in order to allow the user to get a better understanding of the system. E.g.:
+through device names, or instance names.
+
+Diskstats collector
+*******************
+
+This storage data collector will gather information about the status of the
+disks installed in the system, as listed in the /proc/diskstats file. This means
+that not only physical hard drives, but also ramdisks and loopback devices will
+be listed.
+
+Its ``kind`` in the report will be ``0`` (`Performance reporting collectors`_).
+
+Its ``category`` field in the report will contain the value ``storage``.
+
+When executed in verbose mode, the ``data`` section of the report of this
+collector will be a list of items, each representing one disk, each providing
+the following fields:
+
+``major``
+ The major number of the device.
+
+``minor``
+ The minor number of the device.
+
+``name``
+ The name of the device.
+
+``readsNum``
+ This is the total number of reads completed successfully.
+
+``mergedReads``
+ Reads which are adjacent to each other may be merged for efficiency. Thus
+ two 4K reads may become one 8K read before it is ultimately handed to the
+ disk, and so it will be counted (and queued) as only one I/O. This field
+ specifies how often this was done.
+
+``secRead``
+ This is the total number of sectors read successfully.
+
+``timeRead``
+ This is the total number of milliseconds spent by all reads.
+
+``writes``
+ This is the total number of writes completed successfully.
+
+``mergedWrites``
+ Writes which are adjacent to each other may be merged for efficiency. Thus
+ two 4K writes may become one 8K read before it is ultimately handed to the
+ disk, and so it will be counted (and queued) as only one I/O. This field
+ specifies how often this was done.
+
+``secWritten``
+ This is the total number of sectors written successfully.
+
+``timeWrite``
+ This is the total number of milliseconds spent by all writes.
+
+``ios``
+ The number of I/Os currently in progress.
+ The only field that should go to zero, it is incremented as requests are
+ given to appropriate struct request_queue and decremented as they finish.
+
+``timeIO``
+ The number of milliseconds spent doing I/Os. This field increases so long
+ as field ``IOs`` is nonzero.
+
+``wIOmillis``
+ The weighted number of milliseconds spent doing I/Os.
+ This field is incremented at each I/O start, I/O completion, I/O merge,
+ or read of these stats by the number of I/Os in progress (field ``IOs``)
+ times the number of milliseconds spent doing I/O since the last update of
+ this field. This can provide an easy measure of both I/O completion time
+ and the backlog that may be accumulating.
-The ``data`` section of the report will provide at least the following fields:
+Logical Volume collector
+************************
-``free``
- The amount of free space (in KBytes).
+This data collector will gather information about the attributes of logical
+volumes present in the system.
-``used``
- The amount of used space (in KBytes).
+Its ``kind`` in the report will be ``0`` (`Performance reporting collectors`_).
-``total``
- The total visible space (in KBytes).
+Its ``category`` field in the report will contain the value ``storage``.
+
+The ``data`` section of the report of this collector will be a list of items,
+each representing one logical volume and providing the following fields:
+
+``uuid``
+ The UUID of the logical volume.
+
+``name``
+ The name of the logical volume.
+
+``attr``
+ The attributes of the logical volume.
+
+``major``
+ Persistent major number or -1 if not persistent.
+
+``minor``
+ Persistent minor number or -1 if not persistent.
+
+``kernel_major``
+ Currently assigned major number or -1 if LV is not active.
+
+``kernel_minor``
+ Currently assigned minor number or -1 if LV is not active.
+
+``size``
+ Size of LV in bytes.
+
+``seg_count``
+ Number of segments in LV.
+
+``tags``
+ Tags, if any.
+
+``modules``
+ Kernel device-mapper modules required for this LV, if any.
+
+``vg_uuid``
+ Unique identifier of the volume group.
+
+``vg_name``
+ Name of the volume group.
+
+``segtype``
+ Type of LV segment.
-Each specific storage type might provide more type-specific fields.
+``seg_start``
+ Offset within the LVto the start of the segment in bytes.
-In case of error, the ``message`` subfield of the ``status`` field of the
-report of the instance status collector will disclose the nature of the error
-as a type specific information. Examples of these are "backend pv unavailable"
-for lvm storage, "unreachable" for network based storage or "filesystem error"
-for filesystem based implementations.
+``seg_start_pe``
+ Offset within the LV to the start of the segment in physical extents.
+
+``seg_size``
+ Size of the segment in bytes.
+
+``seg_tags``
+ Tags for the segment, if any.
+
+``seg_pe_ranges``
+ Ranges of Physical Extents of underlying devices in lvs command line format.
+
+``devices``
+ Underlying devices used with starting extent numbers.
+
+``instance``
+ The name of the instance this LV is used by, or ``null`` if it was not
+ possible to determine it.
DRBD status
***********
plugin) but we can easily just report the information above, since it's
standard enough across all systems.
-Format of the query
--------------------
+Node OS CPU load average report
++++++++++++++++++++++++++++++++
-The queries to the monitoring agent will be HTTP GET requests on port 1815.
-The answer will be encoded in JSON format and will depend on the specific
-accessed resource.
+This data collector will export CPU load statistics as seen by the host
+system. Apart from using the data from an external monitoring system we
+can also use the data to improve instance allocation and/or the Ganeti
+cluster balance. To compute the CPU load average we will use a number of
+values collected inside a time window. The collection process will be
+done by an independent thread (see `Mode of Operation`_).
-If a request is sent to a non-existing resource, a 404 error will be returned by
-the HTTP server.
+This report is a subset of the previous report (`Node OS resources
+report`_) and they might eventually get merged, once reporting for the
+other fields (memory, filesystem, NICs) gets implemented too.
-The following paragraphs will present the existing resources supported by the
-current protocol version, that is version 1.
+Specifically:
-``/``
-+++++
-The root resource. It will return the list of the supported protocol version
-numbers.
-
-Currently, this will include only version 1.
+The ``category`` field of the report will be ``null``.
-``/1``
-++++++
-Not an actual resource per-se, it is the root of all the resources of protocol
-version 1.
+The ``kind`` field will be ``0`` (`Performance reporting collectors`_).
-If requested through GET, the null JSON value will be returned.
+The ``data`` section will include:
-``/1/list/collectors``
-++++++++++++++++++++++
-Returns a list of tuples (kind, category, name) showing all the collectors
-available in the system.
+``cpu_number``
+ The number of available cpus.
-``/1/report/all``
-+++++++++++++++++
-A list of the reports of all the data collectors, as described in the section
-`Format of the report`_.
+``cpus``
+ A list with one element per cpu, showing its average load.
-`Status reporting collectors`_ will provide their output in non-verbose format.
-The verbose format can be requested by adding the parameter ``verbose=1`` to the
-request.
+``cpu_total``
+ The total CPU load average as a sum of the all separate cpus.
-``/1/report/[category]/[collector_name]``
-+++++++++++++++++++++++++++++++++++++++++
-Returns the report of the collector ``[collector_name]`` that belongs to the
-specified ``[category]``.
+The CPU load report function will get N values, collected by the
+CPU load collection function and calculate the above averages. Please
+see the section `Mode of Operation`_ for more information one how the
+two functions of the data collector interact.
-If a collector does not belong to any category, ``collector`` will be used as
-the value for ``[category]``.
+Format of the query
+-------------------
-`Status reporting collectors`_ will provide their output in non-verbose format.
-The verbose format can be requested by adding the parameter ``verbose=1`` to the
-request.
+.. include:: monitoring-query-format.rst
Instance disk status propagation
--------------------------------
When run as stand-alone binaries, the data collector will not using any
caching system, and just fetch and return the data immediately.
+Since some performance collectors have to operate on a number of values
+collected in previous times, we need a mechanism independent of the data
+collector which will trigger the collection of those values and also
+store them, so that they are available for calculation by the data
+collectors.
+
+To collect data periodically, a thread will be created by the monitoring
+agent which will run the collection function of every data collector
+that provides one. The values returned by the collection function of
+the data collector will be saved in an appropriate map, associating each
+value to the corresponding collector, using the collector's name as the
+key of the map. This map will be stored in mond's memory.
+
+For example: the collection function of the CPU load collector will
+collect a CPU load value and save it in the map mentioned above. The
+collection function will be called by the collector thread every t
+milliseconds. When the report function of the collector is called, it
+will process the last N values of the map and calculate the
+corresponding average.
+
Implementation place
--------------------
projects / ganeti-local / blobdiff