| Branch: | Tag: | Revision:

root / Ganeti / HTools @ 9b8fac3d

# Date Author Comment
9b8fac3d 04/15/2010 12:50 pm Iustin Pop

Add a new function to compute allocation deltas

Given two cluster states, the new function can answer the following

- how much resources currently allocated
- how much resources finally allocated (delta from above is how much we
can actually allocate on the cluster)...

86ecce4a 04/15/2010 12:27 pm Iustin Pop

Introduce total vcpu tracking in CStats

We add a new field that tracks the available virtual cpus (expressed as
node cpus times the vcpu ratio).

bfefb674 04/14/2010 03:44 pm Iustin Pop

Merge branch 'master' into next

  • master:
    Fix iallocator crash when no solutions exist
    Fix IAllocator multi-evacuate message
57587760 03/31/2010 12:54 pm Iustin Pop

Fix iallocator crash when no solutions exist

Commit 5436576 added an un-guarded `head' call, which crashes with
“Prelude.head: empty list” when no results exists for the per-instance
allocation/relocation calls.

This patch fixes this, and also adds another check for an unguarded...

934c62dc 03/31/2010 12:51 pm Iustin Pop

Fix IAllocator multi-evacuate message

Since Ganeti passes full host names (not common-suffix-stripped), we
need to remove the suffix from the evac_nodes keys too. In case one node
is not part of the cluster, it will lead to a wrong error message, but
for now it fixes the problem.

e41f4ba0 03/09/2010 04:40 pm Iustin Pop

Fix iallocator crash when no solutions exist

Commit 5436576 added an un-guarded `head' call, which crashes with
“Prelude.head: empty list” when no results exists for the per-instance
allocation/relocation calls.

This patch fixes this, and also adds another check for an unguarded...

be811997 02/26/2010 03:42 pm Iustin Pop

Fix a haddock comment issue

For some versions of haddock, this can create problems.

a46f34d7 02/25/2010 03:47 pm Iustin Pop

Abstract instance running states into a list

This removes some manual checks from a few places in the code with a
single list defined once.

5182e970 02/25/2010 03:39 pm Iustin Pop

A number of small fixes from hlint

c939b58e 02/25/2010 02:35 pm Iustin Pop

Fix unused-do-binds for ghc 6.12

GHC 6.12 has some new warnings, which are valid in most cases except
(IMHO) printf usage.

0903280b 02/25/2010 02:34 pm Iustin Pop

Fix unused imports for ghc 6.12

GHC 6.12 has become more picky about unused imports, so we need to
remove/tighten some of them.

ba9349b8 02/23/2010 07:10 pm Iustin Pop

hscan: implement LUXI backend scanning

This allows hscan to work also with NO_CURL (but only for the local
machine, of course).

5ab2b771 02/23/2010 02:53 pm Iustin Pop

Loader: abort for unknown to-be-excluded instances

c424cdc8 02/23/2010 02:13 pm Iustin Pop

balance function: use the movable flag directly

Instead of deciding based on secondary node, use the new flag.

39f979b8 02/23/2010 02:09 pm Iustin Pop

Update the loader pipeline to set the movable flag

This updates the movable flag on instances if they have only one node
(we don't rely on OpMoveInstance) or if they are set so via the command
line options.

This doesn't yet enable the use of the new flag.

a182df55 02/23/2010 01:56 pm Iustin Pop

Add a 'movable' flag on instances

This will be used instead of checking for no secondary and for
simplifying 'do not touch' instances.

10f396e1 02/23/2010 11:40 am Iustin Pop

Add an option for excluding instances from moves

12b0511d 02/22/2010 04:19 pm Iustin Pop

Add a tryEvac function

This will be used by the node evacuate IAllocator request type.

Signed-off-by: Iustin Pop <>

54365762 02/22/2010 04:19 pm Iustin Pop

Implement IAllocator node evacuate request

This patch adds the new request loading/execution (trivial), but the
actual response formatting becomes more difficult as now the response
type differs by request.

Signed-off-by: Iustin Pop <>

20c891d0 02/22/2010 04:19 pm Iustin Pop

IAllocator: move some keys into per-request data

Since not all structures will have these keys in the future, we move
them into per-structure keys.

Signed-off-by: Iustin Pop <>

23f9ab76 02/22/2010 04:19 pm Iustin Pop

Change an internal type from Maybe to list

In preparation for multiple responses, we change from Maybe to List
(both used in the container sense).

This allows us to keep the same workflow for all kind of requests.

Signed-off-by: Iustin Pop <>

1fe81531 02/22/2010 04:19 pm Iustin Pop

Move a type declaration to Node.hs

We'll need AllocElement in both Cluster and IAlloc in the future, so we
move it to Node.hs which is imported by both.

Signed-off-by: Iustin Pop <>

2e28ac32 02/22/2010 03:50 pm Iustin Pop

Implement evacuation mode in hbal

This mode restricts the list of instances to be moved to the instances
living on the offline (and drained) nodes.

Signed-off-by: Iustin Pop <>

f0f21ec4 02/22/2010 03:50 pm Iustin Pop

Add an evac mode CLI option

Signed-off-by: Iustin Pop <>

df18fdfe 02/22/2010 03:50 pm Iustin Pop

Reorder options in CLI.hs

This should be no code change, just reordering of the options.

Signed-off-by: Iustin Pop <>

146b37eb 02/03/2010 01:29 pm Iustin Pop

Fix secondary node selection for existing N+1

In case a secondary node is already N+1 failed, currently the node
selection will accept a node that cannot start (at all) the new instance
as valid. This is wrong, so we add a new simple check to prevent the...

a4a6e623 02/03/2010 10:24 am Iustin Pop

Rewrite the node add checks for simpler layout

This will make it clearer than many if…then choices.

a804261a 01/14/2010 06:38 pm Iustin Pop

Move instance relocation test upper in the chain

Currently we test each instance for relocation in checkMove; however, it
is a little more clear if we pass only the relocatable instances to
checkMove. The patch also slightly rewrites (indendation/style) the...

5ad86777 01/14/2010 06:05 pm Iustin Pop

Split the balancing function in two parts

Currently in the balancing function we do two thing:

- take the decision where to do a new balancing round or not
- and actually computing the balancing round

This is not nice, as the two parts are conceptually separate, so this...

71e635f3 01/12/2010 12:18 pm René Nussbaumer

Fixing a typo in option description

Signed-off-by: René Nussbaumer <>
Reviewed-by: Michael Hanselmann <>
Signed-off-by: Iustin Pop <>

16c2369c 01/07/2010 01:57 pm Iustin Pop

Switch the text file format to single-file

This patch changes from the two separate files to a single file, with
sections separated by a blank line. Currently only the node and instance
data is accepted, later the cluster tags will be read too via this

f5197d89 01/07/2010 12:44 pm Iustin Pop

Change the signatures of the text loader slightly

This is in preparation for the text format changes.

0c860cff 12/11/2009 07:01 pm Iustin Pop

Convert n1_score metric from % to count

This increases the priority of fixing N+1 failures compared to balancing

8ce618f3 12/11/2009 06:54 pm Iustin Pop

Merge branch 'master' into next

  • master:
    Use the oper_ram field if available
    rapi, luxi: treat drained nodes as offline
673f0f00 12/11/2009 06:47 pm Iustin Pop

Metric: count of primary instances/offline nodes

This helps with evacuation/failover of instances on 2-node clusters with
one one offline.

e4d31268 12/11/2009 06:43 pm Iustin Pop

Offline instance metric: change from % to count

Currently we use the offline instance percentage (with range [0, 1]),
but this is not good, since we want the evacuation of such instances to
have a high priority; therefore we change this to a count of offline...

6402a260 12/11/2009 06:17 pm Iustin Pop

Use the oper_ram field if available

For the RAPI and LUXI backends, we can get the actual memory usage (if
instances are running) via the oper_ram, whereas backend/memory only
tell what the instance will use at the next boot.

Not using oper_ram means that the node model is flawed and we consider...

b45222ce 12/09/2009 12:30 pm Iustin Pop

rapi, luxi: treat drained nodes as offline

Commit e97f211 changed the iallocator backend to handle drained nodes as
offline. This commit completes that change by making the rapi and luxi
backend do the same (the text backend ignores any '?' values which are...

1cea2e1e 12/02/2009 04:58 pm Iustin Pop

Fix typo breaking LUXI backend

This really shows the need for actual dist-time full testing (not

434c15d5 12/02/2009 12:57 pm Iustin Pop

Fix unittests after instance tags addition

f5e67f55 12/01/2009 02:49 pm Iustin Pop

Configure exclusion tags via the cluster tags

This patch adds reading of the exclusion tags from the cluster tags: any
tags starting with htools:iextags: will convert their suffix into an
exclusion tags prefix. In other words, "htools:iextags:service" will...

669ea132 12/01/2009 01:24 pm Iustin Pop

Read cluster tags in the IAllocator backend

f89235f1 12/01/2009 12:47 pm Iustin Pop

Read cluster tags in the LUXI backend

ea017cbc 12/01/2009 11:53 am Iustin Pop

Read cluster tags in the RAPI backend

This also shows them in hbal in verbose mode.

94e05c32 11/27/2009 05:13 pm Iustin Pop

Introduce support for reading the cluster tags

While these are not actually populated from the backends, and all the
programs ignore them, this patch contains the changes in the function
types required.

0f15cc76 11/17/2009 11:44 am Iustin Pop

Add a command-line option to filter exclusion tags

Since we don't want all instance tags to be used for exclusion, we add a
command line option to filter on these. Since the iallocator protocol
cannot accept command line options, currently it's not possible to...

b2999982 11/17/2009 11:44 am Iustin Pop

Add a new node list field

This patch adds a new node list field (ptags), showing the primary
instance tags.

1e4b5230 11/17/2009 11:44 am Iustin Pop

Node: add function for conflicting primary count

d844fe88 11/17/2009 11:44 am Iustin Pop

Use conflicting primaries count in cluster score

This small patch adds the number of conflicting primaries in the cluster
score. This is different from the other non-CV metrics where we usually
compute the percentage of failing instances (for that metric); but for a...

e27eb8ab 11/17/2009 11:44 am Iustin Pop

Specialize the math functions

The statistics functions are currently defined as polymorphic with a
Floating constraint. Changing this to monomorphic on Double type makes
them stricter and much more performant (~70% speedup). This is a cheap
way to recoup some of the loses incurred by the recent proliferation of...

185297fa 11/17/2009 11:44 am Iustin Pop

Collapse the statistical functions into one

This allows us to get rid of two duplicate list length computations,
with a minor speedup.

5f0b9579 11/17/2009 11:44 am Iustin Pop

Introduce tag-based exclusion of primary instances

This patch introduces exclusion of primary instances based on tags. This
is incomplete as currently all tags are being excluded, and we don't
optimise towards relocation of instances sharing tags on the same node.

17e7af2b 11/11/2009 12:10 pm Iustin Pop

Add a tags attribute to instances

… and read it in all the loaders. hscan is modified to save it to the
files it generates.

The attribute is not yet used in any place.

27671a61 11/11/2009 11:39 am Iustin Pop

Small change in some list arguments

This is simpler than the concat operator.

e98fb766 11/10/2009 02:59 pm Iustin Pop

Allow overriding the field list in -p

The print nodes option can now accept an optional field list to
customise the output. This is ugly, since the field names do not match
the header names, but it is at least barely customisable (at runtime).

76354e11 11/09/2009 05:49 pm Iustin Pop

Move more node-listing functionality in Node.hs

This will prepare for the runtime-selectable field list.

a7a8f280 11/09/2009 04:51 pm Iustin Pop

Change the default dynamic usage to baseUtil

This fixed the unbalanced secondary instances on partially empty
clusters, and helps in general for the cases where real utilisation data
is not available.

daee4bed 11/09/2009 03:43 pm Iustin Pop

Add a few comments in the scoring function

117dc2d8 11/06/2009 07:23 pm Iustin Pop

Enhance the error reporting for Rapi and Luxi

Currently the JSON conversion in Rapi and Luxi are giving something
Error: failed to load data. Details:
Unable to read Double

This doesn't tell one where the error is (in a node specification? and...

262f3e6c 11/06/2009 07:23 pm Iustin Pop

Change the Utils.fromObj signature

Currently the fromObj function takes a JSON object which is then
converted into a list of (String, JSValue) in which we make a lookup.
However, most of the callers of this function call it repeatedly on the
same object, which means we do the object→list conversion repeatedly....

7f4e37f0 11/06/2009 07:13 pm Iustin Pop

Make some CLI options more consistent

Both the simulate and the tiered allocation mode take a machine spec on
input via a comma-separated list. This patch makes this a little bit
more consistent (always use disk,ram,cpu in this order).

83ad1f3c 11/06/2009 07:13 pm Iustin Pop

hspace: show tiered-alloc stats in the output

This is a first attempt to get a readable output of tiered allocation
stats in hspace's output. Not very nice, but it should be somewhat

12e6776a 11/06/2009 07:13 pm Iustin Pop

A small style change in Node.hs

This imports PeerMap as P and reindents some lines.

c8db97e5 11/06/2009 07:12 pm Iustin Pop

Add support for shrinking instance specs

This patch adds a function that, for some given failure modes, shrinks a
given instance in the hope that allocation will succeed when retried
with the new spec.

2f567ac0 10/30/2009 01:23 pm Iustin Pop

Convert option parsing to a monadic flow

This allows us to do verification of option arguments in the assignment
functions themselves.

1f9066c0 10/30/2009 01:23 pm Iustin Pop

Rework the instance spec CLI options

This patch reworks the internal handling of the instance spec CLI
option, and adds a tiered spec option that will be used in hspace to
enable the (auxiliary) tiered-spec allocation mode.

It also introduces a new data type for holding the instance...

ce0392e6 10/21/2009 02:06 pm Iustin Pop

Some cleanup of Loader.mergeData

This doesn't need to be a monadic function, let's make it a simpler one.

a5f8dcdc 10/21/2009 11:58 am Iustin Pop

hbal: ignore unknown instance in dynload file

Since the utilisation file might be generated at a different time from
the hbal run, and instances could dissapear in the meantime, it's better
to simply ignore unknown instances rather than abort.

30ff0c73 10/21/2009 11:47 am Iustin Pop

Expand the --print-instances output

This adds run status, resource parameters and load parameters for

e7724ccc 10/19/2009 01:08 am Iustin Pop

Change the Container.findByName function

This patch changes the signature and implementation of the function;
returning the item makes more sense (saves a lookup later again in the
container, and applying idx is cheap), and the previous implementation
was ugly.

bbd8efd2 10/19/2009 01:08 am Iustin Pop

Some small style fixes

8c9af2f0 10/19/2009 12:17 am Iustin Pop

Simplify the cstats initializer

Since all values are initialized to zero, the exact ordering is not
important and thus we can use the positional mode for simpler code.

The patch also adds docstrings to the cstats functions.

668c03b3 10/19/2009 12:11 am Iustin Pop

Simplify Cluster.computeMoves

Since we now have an actual type for describing the instance moves
(IMove), it's simpler to convert this into the move description/move
commands, rather than re-computing the move based on initial and final
nodes. This makes the shell commands computation and over-Luxi command...

eb2598ab 10/18/2009 11:20 pm Iustin Pop

Remove obsolete export

The ‘Placement’ type has been moved to Types.hs but we kept exporting it
from Cluster, which is not needed.

c5f7412e 10/18/2009 08:21 pm Iustin Pop

Generalise the node/instance listing

This patch introduces a generic formatTable function (based on, and
similar to the Ganeti one, but different and more FP in style) and
changes the node and instance listing to it.

The node list (due to the many variables) is still a little bit hackish...

ad6cffe4 10/18/2009 07:38 pm Iustin Pop

Fix instance listing for non-redundant case

c4ef235b 10/16/2009 05:24 pm Iustin Pop

Fix two haddoc/happy docstring issues

ee9724b9 10/16/2009 04:59 pm Iustin Pop

Start using the utilisation scores in balancing

This enables the per-node load/total available capacity scores to be
used in balancing. Note that the total available capacity is currently
fixed at zero and cannot be changed by the user.

aa8d2e71 10/16/2009 04:59 pm Iustin Pop

Add loading and processing of utilisation data

This patch adds loading and processing the utilisation data during
instance moves. While the data is not yet used, it is correctly modified
by instance changes between nodes.

hbal has the new ‘-U’ command line argument for this. The format of the...

4f83a560 10/16/2009 02:54 pm Iustin Pop

Add an option to input utilisation data

a488a217 10/16/2009 02:43 pm Iustin Pop

Merge the Node.setPri and Node.addCpus functions

The latter is only used right after the former in the Loader module, and
we'll need more of this 'update not with the data of this instance'
functionality (which is different than addPri where all information must...

5b763470 10/16/2009 10:41 am Iustin Pop

Move some utility functions to Utils.hs

These were already duplicate (Text and Simu) and we need tryRead in more places.

183a9c3d 10/16/2009 10:09 am Iustin Pop

Show the load on nodes in node lists

The strange printf usage is due to some limitation (it seems) in ghc for
very long argument lists. The whole printout should be rewritten later.

2180829f 10/15/2009 05:05 pm Iustin Pop

Add initial structure for utilisation balancing

This patch adds the datatypes and modifies the nodes and instance types to have
such attributes. They are not used yet in any way.

507fda3f 10/15/2009 05:00 pm Iustin Pop

Allow displaying the instance map in hbal

This is similar to --print-nodes, but with much fewer fields.

181d4e04 10/15/2009 11:49 am Iustin Pop

Add an explicit export list to Instance.hs

This exports all functions, but it's still good to have.

3a3c1eb4 10/15/2009 11:33 am Iustin Pop

More hlint fixes

This makes (for now) the code hlint-clean. This is per se not a huge
gain, but it allows easier tracking of regressions in style later
(one-two new violations are easier to diagnose when not hidden among 20
“known” ones).

c15f7183 10/14/2009 04:43 pm Iustin Pop

Style change: camel-casing of unittests

f5b553da 10/14/2009 04:41 pm Iustin Pop

Style change: cluster CStats camel-casing

This is again the cs_x to csX name change.

2060348b 10/14/2009 04:41 pm Iustin Pop

Style change: node and instance attributes

This changes from a_b to aB in all node and instance attributes, to
match the standard Haskell style. Also attributes that should have been
camel-cased but weren't were changed (e.g. plist → pList, pnode →

fca250e9 10/14/2009 01:45 pm Iustin Pop

Modify the internals of the detailed CV scores

Before we used a tuple; since we'll need more metrics in the future,
it's simpler to transform this into a list of doubles, whose elements
are handled homogeneously by all the code that needs them.

0df5a1b4 10/14/2009 11:56 am Iustin Pop

Add a command line option for executing jobs

Signed-off-by: Iustin Pop <>
Reviewed-by: Michael Hanselmann <>

dfbbd43a 10/14/2009 11:56 am Iustin Pop

Change iMoveToJob to properly create migrates

The current Cluster.iMoveToJob always creates failovers, which is not
what we want. This simply used the original instances status to select
between these two (this is not optimal by the way, since the status...

924f9c16 10/14/2009 11:55 am Iustin Pop

Extend the MoveJob type to hold the instance index

This will be needed in order to generate the proper instance move commands.

Signed-off-by: Iustin Pop <>

66dac8e0 10/12/2009 03:13 pm Iustin Pop

Fix haddock issues with tuple members

It seems that haddock cannot document tuple members - but arguably, once
one needs to do that, tuples should not be used anymore.

This just moves the comments to the tuple comment.

Signed-off-by: Iustin Pop <>...

e97f211e 10/08/2009 01:06 pm Guido Trotter

parseNode: don't lookup values in drained nodes

Currently parseNode skips looking for values in offline nodes, but tries
to read them for drained ones. With this patch we treat offline and
drained nodes in the same way (which is compatible with the iallocator...

a2e90275 10/02/2009 06:54 pm Iustin Pop

Store the instance move in the MoveJobs

This will automatically sort our Ganeti jobs into the independent job
sets, and then we can submit them separately.

92e32d76 10/02/2009 06:48 pm Iustin Pop

Move some more type definitions to Types.hs

6b20875c 10/02/2009 06:37 pm Iustin Pop

Add a function converting Placements into Jobs

This converts from htools-specific Placements into Ganeti standard
OpCodes, which will later allow execution via Luxi.

3173c987 10/02/2009 05:52 pm Iustin Pop

Record the move being performed in a Placement

This will allow a more descriptive output later in the solution list, as
opposed to trying to reconstruct the move from the node indices.

The patch also documents the Placement members.