|
.BI "[ -L[" path "] [-X]]"
|
-.BI "[ -n " nodes-file " ]"
-.BI "[ -i " instances-file " ]"
+.BI "[ -t " data-file " ]"
.TP
Algorithm options:
.BI "[ -e " score " ]"
.BI "[ -O " name... " ]"
.B "[ --no-disk-moves ]"
+.BI "[ -U " util-file " ]"
+.B "[ --evac-mode ]"
.TP
Reporting options:
.BI "[ -C[" file "] ]"
-.B "[ -p ]"
+.BI "[ -p[" fields "] ]"
.B "[ --print-instances ]"
.B "[ -o ]"
.B "[ -v... | -q ]"
instance placement and computes a series of steps designed to bring
the cluster into a better state.
-The algorithm to do so is designed to be stable (i.e. it will give you
-the same results when restarting it from the middle of the solution)
-and reasonably fast. It is not, however, designed to be a perfect
-algorithm - it is possible to make it go into a corner from which it
-can find no improvement, because it only look one "step" ahead.
+The algorithm used is designed to be stable (i.e. it will give you the
+same results when restarting it from the middle of the solution) and
+reasonably fast. It is not, however, designed to be a perfect
+algorithm \(em it is possible to make it go into a corner from which
+it can find no improvement, because it looks only one "step" ahead.
By default, the program will show the solution incrementally as it is
computed, in a somewhat cryptic format; for getting the actual Ganeti
is O(n*n) in the number of nodes. Furthermore, it doesn't seems to
give better scores but will result in more disk replacements.
+.SS PLACEMENT RESTRICTIONS
+
+At each step, we prevent an instance move if it would cause:
+
+.RS 4
+.TP 3
+\(em
+a node to go into N+1 failure state
+.TP
+\(em
+an instance to move onto an offline node (offline nodes are either
+read from the cluster or declared with \fI-O\fR)
+.TP
+\(em
+an exclusion-tag based conflict (exclusion tags are read from the
+cluster and/or defined via the \fI--exclusion-tags\fR option)
+.TP
+\(em
+a max vcpu/pcpu ratio to be exceeded (configured via \fI--max-cpu\fR)
+.TP
+\(em
+min disk free percentage to go below the configured limit (configured
+via \fI--min-disk\fR)
+
.SS CLUSTER SCORING
As said before, the algorithm tries to minimise the cluster score at
.RS 4
.TP 3
\(em
-coefficient of variance of the percent of free memory
+standard deviation of the percent of free memory
.TP
\(em
-coefficient of variance of the percent of reserved memory
+standard deviation of the percent of reserved memory
.TP
\(em
-coefficient of variance of the percent of free disk
+standard deviation of the percent of free disk
.TP
\(em
-percentage of nodes failing N+1 check
+count of nodes failing N+1 check
.TP
\(em
-percentage of instances living (either as primary or secondary) on
+count of instances living (either as primary or secondary) on
offline nodes
.TP
\(em
-coefficent of variance of the ratio of virtual-to-physical cpus (for
-primary instaces of the node)
+count of instances living (as primary) on offline nodes; this differs
+from the above metric by helping failover of such instances in 2-node
+clusters
+.TP
+\(em
+standard deviation of the ratio of virtual-to-physical cpus (for
+primary instances of the node)
+.TP
+\(em
+standard deviation of the dynamic load on the nodes, for cpus,
+memory, disk and network
.RE
The free memory and free disk values help ensure that all nodes are
N+1. And finally, the N+1 percentage helps guide the algorithm towards
eliminating N+1 failures, if possible.
-Except for the N+1 failures and offline instances percentage, we use
-the coefficient of variance since this brings the values into the same
-unit so to speak, and with a restrict domain of values (between zero
-and one). The percentage of N+1 failures, while also in this numeric
-range, doesn't actually has the same meaning, but it has shown to work
-well.
-
-The other alternative, using for N+1 checks the coefficient of
-variance of (N+1 fail=1, N+1 pass=0) across nodes could hint the
-algorithm to make more N+1 failures if most nodes are N+1 fail
-already. Since this (making N+1 failures) is not allowed by other
-rules of the algorithm, so the N+1 checks would simply not work
-anymore in this case.
-
-The offline instances percentage (meaning the percentage of instances
-living on offline nodes) will cause the algorithm to actively move
-instances away from offline nodes. This, coupled with the restriction
-on placement given by offline nodes, will cause evacuation of such
-nodes.
+Except for the N+1 failures and offline instances counts, we use the
+standard deviation since when used with values within a fixed range
+(we use percents expressed as values between zero and one) it gives
+consistent results across all metrics (there are some small issues
+related to different means, but it works generally well). The 'count'
+type values will have higher score and thus will matter more for
+balancing; thus these are better for hard constraints (like evacuating
+nodes and fixing N+1 failures). For example, the offline instances
+count (i.e. the number of instances living on offline nodes) will
+cause the algorithm to actively move instances away from offline
+nodes. This, coupled with the restriction on placement given by
+offline nodes, will cause evacuation of such nodes.
+
+The dynamic load values need to be read from an external file (Ganeti
+doesn't supply them), and are computed for each node as: sum of
+primary instance cpu load, sum of primary instance memory load, sum of
+primary and secondary instance disk load (as DRBD generates write load
+on secondary nodes too in normal case and in degraded scenarios also
+read load), and sum of primary instance network load. An example of
+how to generate these values for input to hbal would be to track "xm
+list" for instance over a day and by computing the delta of the cpu
+values, and feed that via the \fI-U\fR option for all instances (and
+keep the other metrics as one). For the algorithm to work, all that is
+needed is that the values are consistent for a metric across all
+instances (e.g. all instances use cpu% to report cpu usage, and not
+something related to number of CPU seconds used if the CPUs are
+different), and that they are normalised to between zero and one. Note
+that it's recommended to not have zero as the load value for any
+instance metric since then secondary instances are not well balanced.
On a perfectly balanced cluster (all nodes the same size, all
-instances the same size and spread across the nodes equally), all
-values would be zero. This doesn't happen too often in practice :)
+instances the same size and spread across the nodes equally), the
+values for all metrics would be zero. This doesn't happen too often in
+practice :)
.SS OFFLINE INSTANCES
memory size of down instances from the free node memory of their
primary node, in effect simulating the startup of such instances.
-.SS OTHER POSSIBLE METRICS
+.SS EXCLUSION TAGS
+
+The exclusion tags mechanism is designed to prevent instances which
+run the same workload (e.g. two DNS servers) to land on the same node,
+which would make the respective node a SPOF for the given service.
+
+It works by tagging instances with certain tags and then building
+exclusion maps based on these. Which tags are actually used is
+configured either via the command line (option \fI--exclusion-tags\fR)
+or via adding them to the cluster tags:
-It would be desirable to add more metrics to the algorithm, especially
-dynamically-computed metrics, such as:
-.RS 4
-.TP 3
-\(em
-CPU usage of instances
.TP
-\(em
-Disk IO usage
+.B --exclusion-tags=a,b
+This will make all instance tags of the form \fIa:*\fR, \fIb:*\fR be
+considered for the exclusion map
+
.TP
-\(em
-Network IO
-.RE
+cluster tags \fBhtools:iextags:a\fR, \fBhtools:iextags:b\fR
+This will make instance tags \fIa:*\fR, \fIb:*\fR be considered for
+the exclusion map. More precisely, the suffix of cluster tags starting
+with \fBhtools:iextags:\fR will become the prefix of the exclusion
+tags.
+
+.P
+Both the above forms mean that two instances both having (e.g.) the
+tag \fIa:foo\fR or \fIb:bar\fR won't end on the same node.
.SH OPTIONS
The options that can be passed to the program are as follows:
"jobsets", but only for visual inspection, not for actually
parallelisation. It is not possible to parallelise these directly when
executed via "gnt-instance" commands, since a compound command
-(e.g. failover and replace-disks) must be executed serially. Parallel
+(e.g. failover and replace\-disks) must be executed serially. Parallel
execution is only possible when using the Luxi backend and the
\fI-L\fR option.
Prints the before and after node status, in a format designed to allow
the user to understand the node's most important parameters.
-.TP
-.B --print-instances
-Prints the before and after instance map. This is less useful as the
-node status, but it can help in understanding instance moves.
-
-The node list will contain these informations:
+It is possible to customise the listed information by passing a
+comma\(hyseparated list of field names to this option (the field list is
+currently undocumented). By default, the node list will contain these
+informations:
.RS
.TP
.B F
-a character denoting the status of the node, with '-' meaning an
+a character denoting the status of the node, with '\-' meaning an
offline node, '*' meaning N+1 failure and blank meaning a good node
.TP
.B Name
.TP
.B r_cpu
ratio of virtual to physical cpus
+.TP
+.B lCpu
+the dynamic CPU load (if the information is available)
+.TP
+.B lMem
+the dynamic memory load (if the information is available)
+.TP
+.B lDsk
+the dynamic disk load (if the information is available)
+.TP
+.B lNet
+the dynamic net load (if the information is available)
.RE
.TP
+.B --print-instances
+Prints the before and after instance map. This is less useful as the
+node status, but it can help in understanding instance moves.
+
+.TP
.B -o, --oneline
-Only shows a one-line output from the program, designed for the case
+Only shows a one\(hyline output from the program, designed for the case
when one wants to look at multiple clusters at once and check their
status.
the percentage of instances on offline nodes)
.RE
Note that hbal will also mark as offline any nodes which are reported
-by RAPI as such, or that have "?" in file-based input in any numeric
+by RAPI as such, or that have "?" in file\(hybased input in any numeric
fields.
.RE
.TP
.BI "--no-disk-moves"
-This parameter prevent hbal from using disk move (i.e. "gnt-instance
-replace-disks") operations. This will result in a much quicker
+This parameter prevents hbal from using disk move (i.e. "gnt\-instance
+replace\-disks") operations. This will result in a much quicker
balancing, but of course the improvements are limited. It is up to the
user to decide when to use one or another.
.TP
-.BI "-n" nodefile ", --nodes=" nodefile
-The name of the file holding node information (if not collecting via
-RAPI), instead of the default \fInodes\fR file (but see below how to
-customize the default value via the environment).
+.B "--evac-mode"
+This parameter restricts the list of instances considered for moving
+to the ones living on offline/drained nodes. It can be used as a
+(bulk) replacement for Ganeti's own \fIgnt-node evacuate\fR, with the
+note that it doesn't guarantee full evacuation.
+
+.TP
+.BI "-U" util-file
+This parameter specifies a file holding instance dynamic utilisation
+information that will be used to tweak the balancing algorithm to
+equalise load on the nodes (as opposed to static resource usage). The
+file is in the format "instance_name cpu_util mem_util disk_util
+net_util" where the "_util" parameters are interpreted as numbers and
+the instance name must match exactly the instance as read from
+Ganeti. In case of unknown instance names, the program will abort.
+
+If not given, the default values are one for all metrics and thus
+dynamic utilisation has only one effect on the algorithm: the
+equalisation of the secondary instances across nodes (this is the only
+metric that is not tracked by another, dedicated value, and thus the
+disk load of instances will cause secondary instance
+equalisation). Note that value of one will also influence slightly the
+primary instance count, but that is already tracked via other metrics
+and thus the influence of the dynamic utilisation will be practically
+insignificant.
.TP
-.BI "-i" instancefile ", --instances=" instancefile
-The name of the file holding instance information (if not collecting
-via RAPI), instead of the default \fIinstances\fR file (but see below
-how to customize the default value via the environment).
+.BI "-t" datafile ", --text-data=" datafile
+The name of the file holding node and instance information (if not
+collecting via RAPI or LUXI). This or one of the other backends must
+be selected.
.TP
.BI "-m" cluster
-Collect data not from files but directly from the
+Collect data directly from the
.I cluster
given as an argument via RAPI. If the argument doesn't contain a colon
-(:), then it is converted into a fully-built URL via prepending
+(:), then it is converted into a fully\(hybuilt URL via prepending
https:// and appending the default RAPI port, otherwise it's
-considered a fully-specified URL and is used as-is.
+considered a fully\(hyspecified URL and is used as\(hyis.
.TP
.BI "-L[" path "]"
-Collect data not from files but directly from the master daemon, which
-is to be contacted via the luxi (an internal Ganeti protocol). An
-optional \fIpath\fR argument is interpreted as the path to the unix
-socket on which the master daemon listens; otherwise, the default path
-used by ganeti when installed with "--localstatedir=/var" is used.
+Collect data directly from the master daemon, which is to be contacted
+via the luxi (an internal Ganeti protocol). An optional \fIpath\fR
+argument is interpreted as the path to the unix socket on which the
+master daemon listens; otherwise, the default path used by ganeti when
+installed with \fI--localstatedir=/var\fR is used.
.TP
.B "-X"
.TP
.BI "--max-cpu " cpu-ratio
-The maximum virtual-to-physical cpu ratio, as a floating point number
-between zero and one. For example, specifying \fIcpu-ratio\fR as
-\fB2.5\fR means that, for a 4-cpu machine, a maximum of 10 virtual
-cpus should be allowed to be in use for primary instances. A value of
-one doesn't make sense though, as that means no disk space can be used
-on it.
+The maximum virtual\(hyto\(hyphysical cpu ratio, as a floating point
+number between zero and one. For example, specifying \fIcpu-ratio\fR
+as \fB2.5\fR means that, for a 4\(hycpu machine, a maximum of 10
+virtual cpus should be allowed to be in use for primary instances. A
+value of one doesn't make sense though, as that means no disk space
+can be used on it.
.TP
.BI "--min-disk " disk-ratio
If the variables \fBHTOOLS_NODES\fR and \fBHTOOLS_INSTANCES\fR are
present in the environment, they will override the default names for
the nodes and instances files. These will have of course no effect
-when RAPI is used.
+when the RAPI or Luxi backends are used.
.SH BUGS
The algorithm is not perfect.
The output format is not easily scriptable, and the program should
-feed moves directly into Ganeti (either via RAPI or via a gnt-debug
+feed moves directly into Ganeti (either via RAPI or via a gnt\-debug
input file).
.SH EXAMPLE
Commands to run to reach the above solution:
echo step 1
- echo gnt-instance migrate instance14
- echo gnt-instance replace-disks -n node16 instance14
- echo gnt-instance migrate instance14
+ echo gnt\-instance migrate instance14
+ echo gnt\-instance replace\-disks \-n node16 instance14
+ echo gnt\-instance migrate instance14
echo step 2
- echo gnt-instance migrate instance54
- echo gnt-instance replace-disks -n node16 instance54
- echo gnt-instance migrate instance54
+ echo gnt\-instance migrate instance54
+ echo gnt\-instance replace\-disks \-n node16 instance54
+ echo gnt\-instance migrate instance54
echo step 3
- echo gnt-instance migrate instance4
- echo gnt-instance replace-disks -n node16 instance4
+ echo gnt\-instance migrate instance4
+ echo gnt\-instance replace\-disks \-n node16 instance4
echo step 4
- echo gnt-instance replace-disks -n node2 instance48
- echo gnt-instance migrate instance48
+ echo gnt\-instance replace\-disks \-n node2 instance48
+ echo gnt\-instance migrate instance48
echo step 5
- echo gnt-instance replace-disks -n node16 instance93
- echo gnt-instance migrate instance93
+ echo gnt\-instance replace\-disks \-n node16 instance93
+ echo gnt\-instance migrate instance93
echo step 6
- echo gnt-instance replace-disks -n node2 instance89
- echo gnt-instance migrate instance89
+ echo gnt\-instance replace\-disks \-n node2 instance89
+ echo gnt\-instance migrate instance89
echo step 7
- echo gnt-instance replace-disks -n node16 instance5
- echo gnt-instance migrate instance5
+ echo gnt\-instance replace\-disks \-n node16 instance5
+ echo gnt\-instance migrate instance5
echo step 8
- echo gnt-instance migrate instance94
- echo gnt-instance replace-disks -n node16 instance94
+ echo gnt\-instance migrate instance94
+ echo gnt\-instance replace\-disks \-n node16 instance94
echo step 9
- echo gnt-instance migrate instance44
- echo gnt-instance replace-disks -n node15 instance44
+ echo gnt\-instance migrate instance44
+ echo gnt\-instance replace\-disks \-n node15 instance44
echo step 10
- echo gnt-instance replace-disks -n node16 instance62
+ echo gnt\-instance replace\-disks \-n node16 instance62
echo step 11
- echo gnt-instance replace-disks -n node16 instance13
+ echo gnt\-instance replace\-disks \-n node16 instance13
echo step 12
- echo gnt-instance replace-disks -n node7 instance19
+ echo gnt\-instance replace\-disks \-n node7 instance19
echo step 13
- echo gnt-instance replace-disks -n node1 instance43
+ echo gnt\-instance replace\-disks \-n node1 instance43
echo step 14
- echo gnt-instance replace-disks -n node4 instance1
+ echo gnt\-instance replace\-disks \-n node4 instance1
echo step 15
- echo gnt-instance replace-disks -n node17 instance58
+ echo gnt\-instance replace\-disks \-n node17 instance58
Final cluster status:
N1 Name t_mem f_mem r_mem t_dsk f_dsk pri sec p_fmem p_fdsk
we see which nodes are not N+1 compliant.
The algorithm is stable as long as each step above is fully completed,
-e.g. in step 8, both the migrate and the replace-disks are
+e.g. in step 8, both the migrate and the replace\-disks are
done. Otherwise, if only the migrate is done, the input data is
changed in a way that the program will output a different solution
list (but hopefully will end in the same state).
projects / ganeti-local / blobdiff