| Branch: | Tag: | Revision:

root / Ganeti @ 0a8dd21d

# Date Author Comment
0a8dd21d 06/11/2009 12:17 am Iustin Pop

Add cpu ratio to cluster calculation

f1e64aba 06/10/2009 11:37 pm Iustin Pop

Update cpu counters correctly after pinst changes

The cpu counters are update on primary instance adds/removes.

1a82215d 06/10/2009 11:29 pm Iustin Pop

Add cpu-count-related attributes to nodes

This patch adds cpu-count related attributes to nodes:
- total cpus
- cpus in use
- ratio of virtual:physical cpus

We also set correctly the cpu values at load time, but we don't do
anything yet while moving instances around. The cpu ratio is shown in...

d752eb39 06/10/2009 10:42 pm Iustin Pop

Add a new vcpus attribute to instances

This patch adds reading of vcpu count for instances, in preparation for
using the vcpu ratio in cluster scoring.

734b1ff1 06/10/2009 10:31 pm Iustin Pop

Fix reading of total disk space in iallocator

IAllocator currently uses a wrong key name for reading the total disk
space (‘disk_usage’ which was copied from RAPI, but the actual
iallocator key is ‘disk_space_total’).

This patch fixes that and also makes iallocator always use this key,...

70db354e 06/04/2009 04:32 pm Iustin Pop

Fix the ReplacePrimary instance move

During a replace-primary instance move, on the real cluster the instance
is temporarily started on the secondary, and as such we must check that
the secondary node can hold it for this duration. Currently the code
does not, and depending on cluster scoring it will put instances on such...

9dcec001 06/01/2009 04:48 pm Iustin Pop

Rework the tryAlloc/tryReloc functions

Currently tryAlloc/tryReloc do not return the new instance, as this is
not needed for IAllocator alloc/reloc requests. However, for computing
the space, the new instance is useful, so we modify these functions to
return this information too....

a80bf544 06/01/2009 04:48 pm Iustin Pop

Add an utility function for triples

903a7d46 06/01/2009 02:44 pm Iustin Pop

Small doc change

And an alignment issue.

9b1e1cc9 06/01/2009 01:25 pm Iustin Pop

Ensure consistent naming of the tools

This patch makes sure that all references to the name of the software is
ganeti-htools, not simply htools.

e2fa2baf 06/01/2009 12:55 pm Iustin Pop

Add copyright/license information

This doc-patch adds copyright and license information to (hopefully) all
needed files.

7dd5ee6c 06/01/2009 12:24 pm Iustin Pop

tests: move the test declaration in QC.hs

This patch moves the test declaration into QC.hs, so that test.hs has to
be modified only when we add a new test category.

0991ed70 06/01/2009 12:18 pm Iustin Pop

Small whitespace change

dbba5246 06/01/2009 12:18 pm Iustin Pop

Move some alloc functions from hail into Cluster

These are generic enough to be used from multiple places, they belong
better in Cluster.hs than in the hail source.

19f38ee8 06/01/2009 12:18 pm Iustin Pop

Move the RqType and Request types to Loader.hs

These two will be more generic than now, and belong somewhere else -
Loader.hs is a generic module for data loading, thus we move them there.

d85a0a0f 06/01/2009 12:18 pm Iustin Pop

Cleanup an old function

Also replace a type with its synonim.

9188aeef 06/01/2009 12:18 pm Iustin Pop

Lots of documentation updates

This patch does only doc build changes, doc changes and function move
around (for more logical documentation). It should have no impact at all
on the code.

095d7ac0 06/01/2009 12:18 pm Iustin Pop

A simple test for Container.addTwo

7bc82927 06/01/2009 12:17 pm Iustin Pop

Add some very trivial Instance tests

This is more of an exercise in QuickCheck than strong testing.

9cf4267a 06/01/2009 12:14 pm Iustin Pop

Finish removal of unused params from PeerMap

This completes the removal started earlier byt removeing the need to
pass the number of nodes to Node.buildPeers, which is now unused.

15f4c8ca 06/01/2009 12:14 pm Iustin Pop

Add test infrastructure and initial tests

This patch adds a QuickCheck-based test infrastructure and initial tests
based on it. The PeerMap module has a 100% coverage ☺

Side-note: one has to read the source of QuickCheck to see how to use it
(especially the Batch submodule), the docs are not enough…

71e13e48 05/27/2009 11:50 pm Iustin Pop

Some cleanup of the PeerMap module

This patch removes some unused functions and does some cleanup of the
remaining ones.

17c59f4b 05/27/2009 11:09 pm Iustin Pop

Remove unused parameters from PeerMap creation

We remove some unused arguments (added way back for compatibility with
Arrays, which we didn't use in the end). This makes the code clearer
(and doesn't need the Ndx type to be an instance of Num).

f9fc7a63 05/27/2009 11:01 pm Iustin Pop

Remove an unused type synonim

5a1edeb6 05/27/2009 10:53 pm Iustin Pop

Add a separate function for looking up instances

Currently we (wrongly) use lookupNode to lookup instances, just because
the name assoc list has the same type. This patch adds a separate
function for it.

608efcce 05/27/2009 10:45 pm Iustin Pop

Add type synonyms for the node/instance indices

This is a first step towards full datatype renaming. That requires more
changes, so at first we only want to document clearly what is a node
index, what is an instance index, and what is a plain Int.

262a08a2 05/27/2009 02:17 am Iustin Pop

Change the module import hierarchy

This patch makes the Types module a base module, and Node/Instance ones
import it, from the previous (opposite) situation. This will allow in
the future to use newtypes for the index and name types.

1de50907 05/26/2009 12:28 pm Iustin Pop

Port offline node fixes from Rapi to IAllocator

The IAllocator source was copied from Rapi before the offline node fixes
were made. This changes such that offline nodes are accepted correctly.

bd1794b2 05/25/2009 11:47 pm Iustin Pop

Fix loading of plain instances via iallocator

Currently iallocator is broken when reading single-node instances (and
with an ugly error message). This patch fixes this case, by marking them
with secondary node “noSecondary” like the rest of the code.

a8946537 05/25/2009 10:22 pm Iustin Pop

Fix some haddock issues

Slash is a reserved char. Slash is a reserved char. Slash is a…

8c2ebac8 05/25/2009 09:50 pm Iustin Pop

hail: do not allocate on offline/drained nodes

This patch implements filtering out of the offline/drained nodes and
fixes a bug in IAllocator.hs parsing (similar to an older bug in Rapi.hs
from where the code was copied).

5e15f460 05/25/2009 09:31 pm Iustin Pop

hail: Implement non-mirrored instance allocation

This patch implements non-mirrored instance allocation, by allocating as
secondary node “noSecondary”.

4a340313 05/25/2009 02:09 am Iustin Pop

Implement hail allocate (for 2-node requests)

This patch implements allocate for two node requests. One node requests
can be done as soon as we have a valid allocateOn function for single

58709f92 05/25/2009 02:06 am Iustin Pop

Working implementation if relocate

This patch completes the implementation of hail relocate. It maps all
valid destination nodes through a ReplaceSecondary IMove, filters out
the failed relocations, computes the resulting scores and picks the
lowest one.

ed41c179 05/25/2009 01:09 am Iustin Pop

Start implementing the hail functionality

This patch implements a very stupid (and broken) version of hail

e3a684c5 05/24/2009 02:43 am Iustin Pop

Remove ktn/kti from first half of loader

This patch removes the ktn/kti lists from most parts of the first half
of the loading sequence. Some remain as the [(String, Int)] is the
nicest way to lookup names and get indices back.

8472a321 05/24/2009 02:36 am Iustin Pop

Remove the ktn/kti from second half of loading

This removes the return of ktn/kti from Loader.mergeData and associated

db1bcfe8 05/24/2009 02:29 am Iustin Pop

Remove most uses of ktn/kti

This patch removes all uses of ktn/kti from the past-loader stages.

4333a887 05/24/2009 02:22 am Iustin Pop

Add some utility functions for kt deprecation

These will be used to remove even more uses of ktn/kti in non-critical

dbd6700b 05/24/2009 02:05 am Iustin Pop

Remove some extraneous uses of ktn/kti

Since we have Node/, we can now simplify a few constructs.

26b5d395 05/24/2009 01:47 am Iustin Pop

Strip the suffix from the names in the objects

This strips the suffix from the objects themselves, not only from the
ktn/kti vars.

b513faa1 05/24/2009 01:33 am Iustin Pop

Make IAlloc.loadData return maps

This patch makes the format of IAlloc.loadData be similar the same as

fae371cc 05/24/2009 01:22 am Iustin Pop

Move common loading sequence in CLI

This patch moves the common loading sequence to CLI, such that hbal/hn1
and possible future scripts that take the input from same sources can
use it.

446d8827 05/24/2009 01:16 am Iustin Pop

Move checkData from Cluster to Loader

This moves the remaining loading function to Loader (together with its
associated support functions).

75d1edf8 05/23/2009 03:34 am Iustin Pop

Introduce a class for CLI options

This patch generalizes a little the CLI handling by not passing in a
special function for help and such, but instead requiring that the
options object supports some common functionality.

497e30a1 05/23/2009 03:24 am Iustin Pop

Add a small class for Nodes and Instances

Since both nodes and instances support some common functionality (names
and indices), we add a class so that we can access these attributes in a
generic way.

2727257a 05/23/2009 02:34 am Iustin Pop

Add back names to nodes/instances

In order to simplify the data structures, we add back the name on the
node and instance objects. We still keep the index for, well, indexing,
but we will use the name directly from the object, in order to get rid
of the ktn/kti arguments which are passed around everywhere.

e4c5beaf 05/23/2009 02:29 am Iustin Pop

More code reorganizations

This new big patch does a couple of more cleanups in the loading of data
- introduce a Types module that holds most types (except the base
Node/Instance/etc.) so that multiple other modules can use these
(instead of only Cluster and its users)...

040afc35 05/22/2009 08:03 pm Iustin Pop

Rework the loader model

This big patch changes the loader model from “string data as common
format” to actual object structures as common format.

The text loading function move from Cluster.hs to a new Text.hs module,
some common functions are moved to a new Loader.hs module, and the...

585d4420 05/22/2009 01:14 am Iustin Pop

Initial work on hail

This makes hail compile and get a request parsed via IAlloc, but nothing

7e7f6ca2 05/21/2009 03:54 am Iustin Pop

Experimental support for non-redundant instances

This patch adds experimental support to hbal for non-redundant instances
(i.e. instances with only one node). They are currently handled as
non-moveable, and as such the algorithm simply ignores them.

Supports needs to be added when reading from RAPI via hscan, and...

b33a2243 05/21/2009 03:31 am Iustin Pop

Small doc addition

1c035cb3 05/21/2009 03:26 am Iustin Pop

Introduce nice errors on invalid input fields

This patch switches from plain read to a wrapper over readsPrec that
returns better error messages than the buildin 'Prelude: no parse'.

62007053 05/21/2009 03:10 am Iustin Pop

Split node/instance parsing into functions

This allows easy checking for valid format of the input data (row-wise).

9d3fada5 05/21/2009 02:37 am Iustin Pop

Add initial validation checks in Cluster.loadData

This patch converts loadTabular and loadData to a monadic form, thus
allowing meaningful error messages from the node/instance load routines.

1297ce13 05/21/2009 02:11 am Iustin Pop

Small syntax improvement

D'oh, one can extract from a wrapped variable, not only from functions.

fd22ce8e 05/21/2009 02:09 am Iustin Pop

Convert Cluster.loadData to Result return

This patch changes Cluster.loadData to return a Result, instead of
directly the values; this will allow us to return meaningful error
values (e.g. when an instances lives on unknown node) rather than simply
abort. Currently the result is always an Ok, the actual signalling of...

e015b554 05/21/2009 01:24 am Iustin Pop

Allow overriding the RAPI port/scheme

This patch adds a very dumb way to override the port and scheme for
RAPI: if the master string doesn't contain a colon, we consider it a
simple hostname specification, and thus prepend ‘https://’ and append
‘:5080’, (the default RAPI port); otherwise, we consider it a fully...

ba00ad4d 05/21/2009 12:55 am Iustin Pop

Remove some 1.2 specific code

Now that we dropped 1.2 compatibility, we can removed the special casing
for RAPI differences.

5aa48dbe 05/21/2009 12:54 am Iustin Pop

Generalize some Result function into monad ones

We don't really needed, but is more clean like this.

942403e6 05/21/2009 12:25 am Iustin Pop

Switch from hand-written monads to a real one

This big patch converts from our home-grown monad-like constructs
(the Either stuff) to a real, Either-like-but-another-name monad.

We introduce a “Result a” monad, and this allows dropping many of the
extra constructs. Hopefully the code is also more clear....

144f190b 05/21/2009 12:25 am Iustin Pop

Small changes to the IAlloc module

Adding a small request type data structure.

234d8af0 05/20/2009 12:59 am Iustin Pop

Don't consider offline nodes as N+1 failed

This is just a cosmetic (I hope) change; the nodes shouldn't be used
anyway, and we only correct the display message.

aff363a4 05/20/2009 12:50 am Iustin Pop

Drop RAPI v1 compatiblity

Since we only support Ganeti 2.0, we drop the 1.2 compatibility.

00b15752 05/20/2009 12:45 am Iustin Pop

Add support for 'offline' nodes

This patch drops compatiblity with Ganeti 1.2 and adds support for
offline nodes in the cluster. When reading from RAPI, the drained nodes
are considered offline so that we don't allocate on them too.

3f6af65c 05/19/2009 01:19 am Iustin Pop

Update the IAlloc module

We know have a working parseData function that returns the node and
instance data. This uncovered bad support for non-drbd instances ☹

9ba5c28f 05/18/2009 12:03 pm Iustin Pop

Move the JSON utilities to Utils.hs

This patch moves the generic/reusable JSON functions to Utils.hs, so
that they're shared between RAPI/IAlloc.

43643696 05/18/2009 09:46 am Iustin Pop

Add a copy of Rapi.HS as IAlloc.hs

This will be used in two ways:
- format the response to Ganeti (easy, implemented)
- parse the input data and build the node/instance lists (hard :)

b0517d61 04/25/2009 06:40 pm Iustin Pop

hbal: Add a new min-score option

This new parameter causes the algorithm to finish (or even not start at
all) if we reach/have a score better than it.

b161386d 04/25/2009 06:39 pm Iustin Pop

hbal: Change hardcoded tests to monadic composition

In some case we manually do “if isNothing … then Nothing else …”, which
can be very easily replaced with a monadic construct in the Maybe monad.

8930eef2 04/20/2009 03:12 pm Iustin Pop

Increase allowed missing memory to 512MB

Since Xen seems to “steal” some amounts of memory (depending on total
node memory), we increase the maximum allowed missing memory to 512MB,
based on gathered data from multiple machines.

8032b3b5 04/20/2009 02:07 pm Iustin Pop

Add reading the file names from env vars

This patch adds support for selecting the instance/node file names via
two environment variables (HTOOLS_NODES, HTOOLS_INSTANCES).

Unfortunately we still have lots of duplicated code, since the options
are not unified.

e0eb63f0 04/19/2009 01:17 am Iustin Pop

Implement writing the command list to a script

This patch adds support in hbal for writing the command list to a shell
script, with error checking and allowing for early exit.

3d7cd10b 04/16/2009 12:02 pm Iustin Pop

hbal: Abort for invalid offline node names

Since it's easy to pass a wrong node name as offline, we should abort
instead of silently ignoring it.

6ef35e3c 03/23/2009 09:12 am Iustin Pop

More documentation updates

This removes most of the content of the README file (obsoleted by new
algorithm and man pages), modifies the Makefile to include the built
documentation in the source archive (so that haddock/hscolour are not
needed) and updates the haddock-prologue with current information.

9cded5d3 03/23/2009 12:32 am Iustin Pop

Add checks for missing disk space

This small patch adds disk space checks to the Cluster.checkData
function, and simplifies a little the warning messages.

190ce47c 03/23/2009 12:32 am Iustin Pop

Include DRBD overhead in sda/sdb size

For Ganeti 1.2 which doesn't have the ‘disk_usage’ instance query field,
we need to manually include the DRBD overhead (per disk). This patch
modifies the RAPI collection to do this, but loading from disk does not
as it's unknown if the query came from hscan or RAPI 1.2 or RAPI 2.0...

0ee8fd76 03/22/2009 11:33 pm Iustin Pop

Update all needed node fields on f_mem change

This fixes the setFmem function which didn't compute other related
fields after free memory change. Ideally, this should be abstracted so
that add/remove Pri and similar functions could reuse it instead of
duplicating code.

53f00b20 03/22/2009 12:40 pm Iustin Pop

Fix interaction between down instances and nodes

If an instance is down, it's memory is not reflected in the node used
memory, and thus the node free memory is higher than the actual value.
This patch deducts the memory for such instances from the node free...

f82f1f39 03/22/2009 12:24 pm Iustin Pop

Add a new instance field denoting run status

This patch modifies Rapi, the Cluster.loadData and hscan serialization to load
and save the instance run status. At instance level, we add both a boolean
field denoting the true/false run status, and a string field which holds the...

a1c6212e 03/22/2009 12:07 pm Iustin Pop

Show the x_mem/i_mem in node list

This patch adds checking of cluster data in the binaries and display of
node's x_mem/i_mem in the node list.

5d1baf63 03/22/2009 12:02 pm Iustin Pop

Add functions to check and fix cluster data

This patch adds a checkData function which goes over the node list and computes
the unaccounted memory, returning a list of warning messages (if any) and the
update nodes.

8c5b0a0d 03/22/2009 11:55 am Iustin Pop

Add a new node filed x_mem

Nodes can have some memory unaccounted for, due to (e.g.) hypervisor
overhead, rounding errors in reporting, etc.

It is better if we model this memory explicitly instead of hiding it,
and actually since the n_mem addition it is actually required to do so....

209b3711 03/22/2009 11:52 am Iustin Pop

Split common CLI functionality into a module

This patch moves the common CLI functionality (as much as currently
possible) into a separate module. This means we only have one parseOpts
and that Utils.hs doesn't keep this kind of functions anymore.

6e75a445 03/22/2009 02:18 am Iustin Pop

Remove unused and obsolete function

The Node.str function is very old and is not useful since the node
objects have much more fields today. This patch removes it, and if
needed a full node display can be done via ‘show’.

04be800a 03/22/2009 01:25 am Iustin Pop

Add node memory field to Node objects

This patch adds a new n_mem field to the node objects, and implements
read/save/show support for it. The field is not currently used (except
in the node list) but will be used for checking data consistency and
instance up/down status.

47a8bade 03/22/2009 01:12 am Iustin Pop

Pass actual types to node/instance constructors

This patch changes the parameters passed to the node and instance
constructors from generic Strings (which are then parsed via “read”) to
the actual used types, by converting them earlier in Cluster.loadData.

7847a037 03/22/2009 12:48 am Iustin Pop

Some small changes in preparation for hscan

This patch does some small changes:
- fixes a comment
- export more node functions (unneeded now, but hscan will use them)
- fixes Makefile rule for building the programs

740ec004 03/21/2009 11:00 pm Iustin Pop

Add a separate type for the [(Int, String)] list

This is added for better readability, since this is very often used in

19777638 03/21/2009 04:48 pm Iustin Pop

Handle correctly offline nodes in cluster scoring

This patch changes two things with regard to offline nodes:
- first, it only calculates the various coefficients across online
- second, it adds a new score denoting the percentage of instances...

352806f7 03/21/2009 01:20 pm Iustin Pop

Show offline nodes in the node status list

This patch adds a new ‘-’ flag for the node status which denotes offline

40d4eba0 03/21/2009 01:26 am Iustin Pop

Restrict move list based on offline node status

This patch changes the Cluster.checkInstanceMove function to restrict
the target move list based on which nodes are online.

ec18dca9 03/21/2009 12:48 am Iustin Pop

Add command line support for offlining nodes

This patch modifies hbal (only, hn1 not yet) for setting nodes offline.

c2c1ef0c 03/21/2009 12:45 am Iustin Pop

Add a new 'offline' Node attribute

This patch adds a new node attribute - offline - which will serve to
skip nodes from the target candidate list.

80d0d2f1 03/20/2009 11:48 pm Iustin Pop

Small doc update in Node.hs

7ae514ba 03/20/2009 08:58 pm Iustin Pop

Some updates to the apidoc rules

669d7e3d 03/20/2009 07:16 pm Iustin Pop

Introduce a namespace for the modules

The modules are moved from the ‘top’ namespace to ‘Ganeti.HTools’, in
compliance with standard practices.