1 .TH HBAL 1 2009-03-23 htools "Ganeti H-tools"
3 hbal \- Cluster balancer for Ganeti
7 .B "[backend options...]"
8 .B "[algorithm options...]"
9 .B "[reporting options...]"
16 .BI "[ -m " cluster " ]"
18 .BI "[ -L[" path "] [-X]]"
20 .BI "[ -n " nodes-file " ]"
21 .BI "[ -i " instances-file " ]"
25 .BI "[ --max-cpu " cpu-ratio " ]"
26 .BI "[ --min-disk " disk-ratio " ]"
27 .BI "[ -l " limit " ]"
28 .BI "[ -e " score " ]"
29 .BI "[ -O " name... " ]"
30 .B "[ --no-disk-moves ]"
31 .BI "[ -U " util-file " ]"
35 .BI "[ -C[" file "] ]"
36 .BI "[ -p[" fields "] ]"
37 .B "[ --print-instances ]"
43 hbal is a cluster balancer that looks at the current state of the
44 cluster (nodes with their total and free disk, memory, etc.) and
45 instance placement and computes a series of steps designed to bring
46 the cluster into a better state.
48 The algorithm used is designed to be stable (i.e. it will give you the
49 same results when restarting it from the middle of the solution) and
50 reasonably fast. It is not, however, designed to be a perfect
51 algorithm \(em it is possible to make it go into a corner from which
52 it can find no improvement, because it looks only one "step" ahead.
54 By default, the program will show the solution incrementally as it is
55 computed, in a somewhat cryptic format; for getting the actual Ganeti
56 command list, use the \fB-C\fR option.
60 The program works in independent steps; at each step, we compute the
61 best instance move that lowers the cluster score.
63 The possible move type for an instance are combinations of
64 failover/migrate and replace-disks such that we change one of the
65 instance nodes, and the other one remains (but possibly with changed
66 role, e.g. from primary it becomes secondary). The list is:
76 replace primary, a composite move (f, r, f)
79 failover and replace secondary, also composite (f, r)
82 replace secondary and failover, also composite (r, f)
85 We don't do the only remaining possibility of replacing both nodes
86 (r,f,r,f or the equivalent f,r,f,r) since these move needs an
87 exhaustive search over both candidate primary and secondary nodes, and
88 is O(n*n) in the number of nodes. Furthermore, it doesn't seems to
89 give better scores but will result in more disk replacements.
91 .SS PLACEMENT RESTRICTIONS
93 At each step, we prevent an instance move if it would cause:
98 a node to go into N+1 failure state
101 an instance to move onto an offline node (offline nodes are either
102 read from the cluster or declared with \fI-O\fR)
105 an exclusion-tag based conflict (exclusion tags are read from the
106 cluster and/or defined via the \fI--exclusion-tags\fR option)
109 a max vcpu/pcpu ratio to be exceeded (configured via \fI--max-cpu\fR)
112 min disk free percentage to go below the configured limit (configured
113 via \fI--min-disk\fR)
117 As said before, the algorithm tries to minimise the cluster score at
118 each step. Currently this score is computed as a sum of the following
123 standard deviation of the percent of free memory
126 standard deviation of the percent of reserved memory
129 standard deviation of the percent of free disk
132 count of nodes failing N+1 check
135 count of instances living (either as primary or secondary) on
139 count of instances living (as primary) on offline nodes; this differs
140 from the above metric by helping failover of such instances in 2-node
144 standard deviation of the ratio of virtual-to-physical cpus (for
145 primary instances of the node)
148 standard deviation of the dynamic load on the nodes, for cpus,
149 memory, disk and network
152 The free memory and free disk values help ensure that all nodes are
153 somewhat balanced in their resource usage. The reserved memory helps
154 to ensure that nodes are somewhat balanced in holding secondary
155 instances, and that no node keeps too much memory reserved for
156 N+1. And finally, the N+1 percentage helps guide the algorithm towards
157 eliminating N+1 failures, if possible.
159 Except for the N+1 failures and offline instances counts, we use the
160 standard deviation since when used with values within a fixed range
161 (we use percents expressed as values between zero and one) it gives
162 consistent results across all metrics (there are some small issues
163 related to different means, but it works generally well). The 'count'
164 type values will have higher score and thus will matter more for
165 balancing; thus these are better for hard constraints (like evacuating
166 nodes and fixing N+1 failures). For example, the offline instances
167 count (i.e. the number of instances living on offline nodes) will
168 cause the algorithm to actively move instances away from offline
169 nodes. This, coupled with the restriction on placement given by
170 offline nodes, will cause evacuation of such nodes.
172 The dynamic load values need to be read from an external file (Ganeti
173 doesn't supply them), and are computed for each node as: sum of
174 primary instance cpu load, sum of primary instance memory load, sum of
175 primary and secondary instance disk load (as DRBD generates write load
176 on secondary nodes too in normal case and in degraded scenarios also
177 read load), and sum of primary instance network load. An example of
178 how to generate these values for input to hbal would be to track "xm
179 list" for instance over a day and by computing the delta of the cpu
180 values, and feed that via the \fI-U\fR option for all instances (and
181 keep the other metrics as one). For the algorithm to work, all that is
182 needed is that the values are consistent for a metric across all
183 instances (e.g. all instances use cpu% to report cpu usage, and not
184 something related to number of CPU seconds used if the CPUs are
185 different), and that they are normalised to between zero and one. Note
186 that it's recommended to not have zero as the load value for any
187 instance metric since then secondary instances are not well balanced.
189 On a perfectly balanced cluster (all nodes the same size, all
190 instances the same size and spread across the nodes equally), the
191 values for all metrics would be zero. This doesn't happen too often in
194 .SS OFFLINE INSTANCES
196 Since current Ganeti versions do not report the memory used by offline
197 (down) instances, ignoring the run status of instances will cause
198 wrong calculations. For this reason, the algorithm subtracts the
199 memory size of down instances from the free node memory of their
200 primary node, in effect simulating the startup of such instances.
204 The exclusion tags mechanism is designed to prevent instances which
205 run the same workload (e.g. two DNS servers) to land on the same node,
206 which would make the respective node a SPOF for the given service.
208 It works by tagging instances with certain tags and then building
209 exclusion maps based on these. Which tags are actually used is
210 configured either via the command line (option \fI--exclusion-tags\fR)
211 or via adding them to the cluster tags:
214 .B --exclusion-tags=a,b
215 This will make all instance tags of the form \fIa:*\fR, \fIb:*\fR be
216 considered for the exclusion map
219 cluster tags \fBhtools:iextags:a\fR, \fBhtools:iextags:b\fR
220 This will make instance tags \fIa:*\fR, \fIb:*\fR be considered for
221 the exclusion map. More precisely, the suffix of cluster tags starting
222 with \fBhtools:iextags:\fR will become the prefix of the exclusion
226 Both the above forms mean that two instances both having (e.g.) the
227 tag \fIa:foo\fR or \fIb:bar\fR won't end on the same node.
230 The options that can be passed to the program are as follows:
232 .B -C, --print-commands
233 Print the command list at the end of the run. Without this, the
234 program will only show a shorter, but cryptic output.
236 Note that the moves list will be split into independent steps, called
237 "jobsets", but only for visual inspection, not for actually
238 parallelisation. It is not possible to parallelise these directly when
239 executed via "gnt-instance" commands, since a compound command
240 (e.g. failover and replace\-disks) must be executed serially. Parallel
241 execution is only possible when using the Luxi backend and the
244 The algorithm for splitting the moves into jobsets is by accumulating
245 moves until the next move is touching nodes already touched by the
246 current moves; this means we can't execute in parallel (due to
247 resource allocation in Ganeti) and thus we start a new jobset.
251 Prints the before and after node status, in a format designed to allow
252 the user to understand the node's most important parameters.
254 It is possible to customise the listed information by passing a
255 comma\(hyseparated list of field names to this option (the field list is
256 currently undocumented). By default, the node list will contain these
261 a character denoting the status of the node, with '\-' meaning an
262 offline node, '*' meaning N+1 failure and blank meaning a good node
268 the total node memory
271 the memory used by the node itself
274 the memory used by instances
277 amount memory which seems to be in use but cannot be determined why or
278 by which instance; usually this means that the hypervisor has some
279 overhead or that there are other reporting errors
285 the reserved node memory, which is the amount of free memory needed
295 the number of physical cpus on the node
298 the number of virtual cpus allocated to primary instances
301 number of primary instances
304 number of secondary instances
307 percent of free memory
313 ratio of virtual to physical cpus
316 the dynamic CPU load (if the information is available)
319 the dynamic memory load (if the information is available)
322 the dynamic disk load (if the information is available)
325 the dynamic net load (if the information is available)
330 Prints the before and after instance map. This is less useful as the
331 node status, but it can help in understanding instance moves.
335 Only shows a one\(hyline output from the program, designed for the case
336 when one wants to look at multiple clusters at once and check their
339 The line will contain four fields:
344 initial cluster score
347 number of steps in the solution
353 improvement in the cluster score
359 This option (which can be given multiple times) will mark nodes as
360 being \fIoffline\fR. This means a couple of things:
365 instances won't be placed on these nodes, not even temporarily;
366 e.g. the \fIreplace primary\fR move is not available if the secondary
367 node is offline, since this move requires a failover.
370 these nodes will not be included in the score calculation (except for
371 the percentage of instances on offline nodes)
373 Note that hbal will also mark as offline any nodes which are reported
374 by RAPI as such, or that have "?" in file\(hybased input in any numeric
379 .BI "-e" score ", --min-score=" score
380 This parameter denotes the minimum score we are happy with and alters
381 the computation in two ways:
386 if the cluster has the initial score lower than this value, then we
387 don't enter the algorithm at all, and exit with success
390 during the iterative process, if we reach a score lower than this
391 value, we exit the algorithm
393 The default value of the parameter is currently \fI1e-9\fR (chosen
398 .BI "--no-disk-moves"
399 This parameter prevents hbal from using disk move (i.e. "gnt\-instance
400 replace\-disks") operations. This will result in a much quicker
401 balancing, but of course the improvements are limited. It is up to the
402 user to decide when to use one or another.
406 This parameter specifies a file holding instance dynamic utilisation
407 information that will be used to tweak the balancing algorithm to
408 equalise load on the nodes (as opposed to static resource usage). The
409 file is in the format "instance_name cpu_util mem_util disk_util
410 net_util" where the "_util" parameters are interpreted as numbers and
411 the instance name must match exactly the instance as read from
412 Ganeti. In case of unknown instance names, the program will abort.
414 If not given, the default values are one for all metrics and thus
415 dynamic utilisation has only one effect on the algorithm: the
416 equalisation of the secondary instances across nodes (this is the only
417 metric that is not tracked by another, dedicated value, and thus the
418 disk load of instances will cause secondary instance
419 equalisation). Note that value of one will also influence slightly the
420 primary instance count, but that is already tracked via other metrics
421 and thus the influence of the dynamic utilisation will be practically
425 .BI "-n" nodefile ", --nodes=" nodefile
426 The name of the file holding node information (if not collecting via
427 RAPI), instead of the default \fInodes\fR file (but see below how to
428 customize the default value via the environment).
431 .BI "-i" instancefile ", --instances=" instancefile
432 The name of the file holding instance information (if not collecting
433 via RAPI), instead of the default \fIinstances\fR file (but see below
434 how to customize the default value via the environment).
438 Collect data not from files but directly from the
440 given as an argument via RAPI. If the argument doesn't contain a colon
441 (:), then it is converted into a fully\(hybuilt URL via prepending
442 https:// and appending the default RAPI port, otherwise it's
443 considered a fully\(hyspecified URL and is used as\(hyis.
447 Collect data not from files but directly from the master daemon, which
448 is to be contacted via the luxi (an internal Ganeti protocol). An
449 optional \fIpath\fR argument is interpreted as the path to the unix
450 socket on which the master daemon listens; otherwise, the default path
451 used by ganeti when installed with \fI--localstatedir=/var\fR is used.
455 When using the Luxi backend, hbal can also execute the given
456 commands. The execution method is to execute the individual jobsets
457 (see the \fI-C\fR option for details) in separate stages, aborting if
458 at any time a jobset doesn't have all jobs successful. Each step in
459 the balancing solution will be translated into exactly one Ganeti job
460 (having between one and three OpCodes), and all the steps in a jobset
461 will be executed in parallel. The jobsets themselves are executed
465 .BI "-l" N ", --max-length=" N
466 Restrict the solution to this length. This can be used for example to
467 automate the execution of the balancing.
470 .BI "--max-cpu " cpu-ratio
471 The maximum virtual\(hyto\(hyphysical cpu ratio, as a floating point
472 number between zero and one. For example, specifying \fIcpu-ratio\fR
473 as \fB2.5\fR means that, for a 4\(hycpu machine, a maximum of 10
474 virtual cpus should be allowed to be in use for primary instances. A
475 value of one doesn't make sense though, as that means no disk space
479 .BI "--min-disk " disk-ratio
480 The minimum amount of free disk space remaining, as a floating point
481 number. For example, specifying \fIdisk-ratio\fR as \fB0.25\fR means
482 that at least one quarter of disk space should be left free on nodes.
486 Increase the output verbosity. Each usage of this option will increase
487 the verbosity (currently more than 2 doesn't make sense) from the
492 Decrease the output verbosity. Each usage of this option will decrease
493 the verbosity (less than zero doesn't make sense) from the default of
498 Just show the program version and exit.
502 The exist status of the command will be zero, unless for some reason
503 the algorithm fatally failed (e.g. wrong node or instance data).
507 If the variables \fBHTOOLS_NODES\fR and \fBHTOOLS_INSTANCES\fR are
508 present in the environment, they will override the default names for
509 the nodes and instances files. These will have of course no effect
510 when the RAPI or Luxi backends are used.
514 The program does not check its input data for consistency, and aborts
515 with cryptic errors messages in this case.
517 The algorithm is not perfect.
519 The output format is not easily scriptable, and the program should
520 feed moves directly into Ganeti (either via RAPI or via a gnt\-debug
525 Note that this example are not for the latest version (they don't have
530 With the default options, the program shows each individual step and
531 the improvements it brings in cluster score:
536 Loaded 20 nodes, 80 instances
537 Cluster is not N+1 happy, continuing but no guarantee that the cluster will end N+1 happy.
538 Initial score: 0.52329131
539 Trying to minimize the CV...
540 1. instance14 node1:node10 => node16:node10 0.42109120 a=f r:node16 f
541 2. instance54 node4:node15 => node16:node15 0.31904594 a=f r:node16 f
542 3. instance4 node5:node2 => node2:node16 0.26611015 a=f r:node16
543 4. instance48 node18:node20 => node2:node18 0.21361717 a=r:node2 f
544 5. instance93 node19:node18 => node16:node19 0.16166425 a=r:node16 f
545 6. instance89 node3:node20 => node2:node3 0.11005629 a=r:node2 f
546 7. instance5 node6:node2 => node16:node6 0.05841589 a=r:node16 f
547 8. instance94 node7:node20 => node20:node16 0.00658759 a=f r:node16
548 9. instance44 node20:node2 => node2:node15 0.00438740 a=f r:node15
549 10. instance62 node14:node18 => node14:node16 0.00390087 a=r:node16
550 11. instance13 node11:node14 => node11:node16 0.00361787 a=r:node16
551 12. instance19 node10:node11 => node10:node7 0.00336636 a=r:node7
552 13. instance43 node12:node13 => node12:node1 0.00305681 a=r:node1
553 14. instance1 node1:node2 => node1:node4 0.00263124 a=r:node4
554 15. instance58 node19:node20 => node19:node17 0.00252594 a=r:node17
555 Cluster score improved from 0.52329131 to 0.00252594
559 In the above output, we can see:
560 - the input data (here from files) shows a cluster with 20 nodes and
562 - the cluster is not initially N+1 compliant
563 - the initial score is 0.52329131
565 The step list follows, showing the instance, its initial
566 primary/secondary nodes, the new primary secondary, the cluster list,
567 and the actions taken in this step (with 'f' denoting failover/migrate
568 and 'r' denoting replace secondary).
570 Finally, the program shows the improvement in cluster score.
572 A more detailed output is obtained via the \fB-C\fR and \fB-p\fR options:
577 Loaded 20 nodes, 80 instances
578 Cluster is not N+1 happy, continuing but no guarantee that the cluster will end N+1 happy.
579 Initial cluster status:
580 N1 Name t_mem f_mem r_mem t_dsk f_dsk pri sec p_fmem p_fdsk
581 * node1 32762 1280 6000 1861 1026 5 3 0.03907 0.55179
582 node2 32762 31280 12000 1861 1026 0 8 0.95476 0.55179
583 * node3 32762 1280 6000 1861 1026 5 3 0.03907 0.55179
584 * node4 32762 1280 6000 1861 1026 5 3 0.03907 0.55179
585 * node5 32762 1280 6000 1861 978 5 5 0.03907 0.52573
586 * node6 32762 1280 6000 1861 1026 5 3 0.03907 0.55179
587 * node7 32762 1280 6000 1861 1026 5 3 0.03907 0.55179
588 node8 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
589 node9 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
590 * node10 32762 7280 12000 1861 1026 4 4 0.22221 0.55179
591 node11 32762 7280 6000 1861 922 4 5 0.22221 0.49577
592 node12 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
593 node13 32762 7280 6000 1861 922 4 5 0.22221 0.49577
594 node14 32762 7280 6000 1861 922 4 5 0.22221 0.49577
595 * node15 32762 7280 12000 1861 1131 4 3 0.22221 0.60782
596 node16 32762 31280 0 1861 1860 0 0 0.95476 1.00000
597 node17 32762 7280 6000 1861 1106 5 3 0.22221 0.59479
598 * node18 32762 1280 6000 1396 561 5 3 0.03907 0.40239
599 * node19 32762 1280 6000 1861 1026 5 3 0.03907 0.55179
600 node20 32762 13280 12000 1861 689 3 9 0.40535 0.37068
602 Initial score: 0.52329131
603 Trying to minimize the CV...
604 1. instance14 node1:node10 => node16:node10 0.42109120 a=f r:node16 f
605 2. instance54 node4:node15 => node16:node15 0.31904594 a=f r:node16 f
606 3. instance4 node5:node2 => node2:node16 0.26611015 a=f r:node16
607 4. instance48 node18:node20 => node2:node18 0.21361717 a=r:node2 f
608 5. instance93 node19:node18 => node16:node19 0.16166425 a=r:node16 f
609 6. instance89 node3:node20 => node2:node3 0.11005629 a=r:node2 f
610 7. instance5 node6:node2 => node16:node6 0.05841589 a=r:node16 f
611 8. instance94 node7:node20 => node20:node16 0.00658759 a=f r:node16
612 9. instance44 node20:node2 => node2:node15 0.00438740 a=f r:node15
613 10. instance62 node14:node18 => node14:node16 0.00390087 a=r:node16
614 11. instance13 node11:node14 => node11:node16 0.00361787 a=r:node16
615 12. instance19 node10:node11 => node10:node7 0.00336636 a=r:node7
616 13. instance43 node12:node13 => node12:node1 0.00305681 a=r:node1
617 14. instance1 node1:node2 => node1:node4 0.00263124 a=r:node4
618 15. instance58 node19:node20 => node19:node17 0.00252594 a=r:node17
619 Cluster score improved from 0.52329131 to 0.00252594
621 Commands to run to reach the above solution:
623 echo gnt\-instance migrate instance14
624 echo gnt\-instance replace\-disks \-n node16 instance14
625 echo gnt\-instance migrate instance14
627 echo gnt\-instance migrate instance54
628 echo gnt\-instance replace\-disks \-n node16 instance54
629 echo gnt\-instance migrate instance54
631 echo gnt\-instance migrate instance4
632 echo gnt\-instance replace\-disks \-n node16 instance4
634 echo gnt\-instance replace\-disks \-n node2 instance48
635 echo gnt\-instance migrate instance48
637 echo gnt\-instance replace\-disks \-n node16 instance93
638 echo gnt\-instance migrate instance93
640 echo gnt\-instance replace\-disks \-n node2 instance89
641 echo gnt\-instance migrate instance89
643 echo gnt\-instance replace\-disks \-n node16 instance5
644 echo gnt\-instance migrate instance5
646 echo gnt\-instance migrate instance94
647 echo gnt\-instance replace\-disks \-n node16 instance94
649 echo gnt\-instance migrate instance44
650 echo gnt\-instance replace\-disks \-n node15 instance44
652 echo gnt\-instance replace\-disks \-n node16 instance62
654 echo gnt\-instance replace\-disks \-n node16 instance13
656 echo gnt\-instance replace\-disks \-n node7 instance19
658 echo gnt\-instance replace\-disks \-n node1 instance43
660 echo gnt\-instance replace\-disks \-n node4 instance1
662 echo gnt\-instance replace\-disks \-n node17 instance58
664 Final cluster status:
665 N1 Name t_mem f_mem r_mem t_dsk f_dsk pri sec p_fmem p_fdsk
666 node1 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
667 node2 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
668 node3 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
669 node4 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
670 node5 32762 7280 6000 1861 1078 4 5 0.22221 0.57947
671 node6 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
672 node7 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
673 node8 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
674 node9 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
675 node10 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
676 node11 32762 7280 6000 1861 1022 4 4 0.22221 0.54951
677 node12 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
678 node13 32762 7280 6000 1861 1022 4 4 0.22221 0.54951
679 node14 32762 7280 6000 1861 1022 4 4 0.22221 0.54951
680 node15 32762 7280 6000 1861 1031 4 4 0.22221 0.55408
681 node16 32762 7280 6000 1861 1060 4 4 0.22221 0.57007
682 node17 32762 7280 6000 1861 1006 5 4 0.22221 0.54105
683 node18 32762 7280 6000 1396 761 4 2 0.22221 0.54570
684 node19 32762 7280 6000 1861 1026 4 4 0.22221 0.55179
685 node20 32762 13280 6000 1861 1089 3 5 0.40535 0.58565
690 Here we see, beside the step list, the initial and final cluster
691 status, with the final one showing all nodes being N+1 compliant, and
692 the command list to reach the final solution. In the initial listing,
693 we see which nodes are not N+1 compliant.
695 The algorithm is stable as long as each step above is fully completed,
696 e.g. in step 8, both the migrate and the replace\-disks are
697 done. Otherwise, if only the migrate is done, the input data is
698 changed in a way that the program will output a different solution
699 list (but hopefully will end in the same state).
702 .BR hspace "(1), " hscan "(1), " hail "(1), "
703 .BR ganeti "(7), " gnt-instance "(8), " gnt-node "(8)"
707 Copyright (C) 2009 Google Inc. Permission is granted to copy,
708 distribute and/or modify under the terms of the GNU General Public
709 License as published by the Free Software Foundation; either version 2
710 of the License, or (at your option) any later version.
712 On Debian systems, the complete text of the GNU General Public License
713 can be found in /usr/share/common-licenses/GPL.
projects / ganeti-local / blob