Iustin Pop [Sat, 23 May 2009 23:43:38 +0000 (00:43 +0100)]
Remove ktn/kti from first half of loader
This patch removes the ktn/kti lists from most parts of the first half
of the loading sequence. Some remain as the [(String, Int)] is the
nicest way to lookup names and get indices back.
Iustin Pop [Sat, 23 May 2009 23:36:36 +0000 (00:36 +0100)]
Remove the ktn/kti from second half of loading
This removes the return of ktn/kti from Loader.mergeData and associated
functions.
Iustin Pop [Sat, 23 May 2009 23:29:52 +0000 (00:29 +0100)]
Remove most uses of ktn/kti
This patch removes all uses of ktn/kti from the past-loader stages.
Iustin Pop [Sat, 23 May 2009 23:23:15 +0000 (00:23 +0100)]
Remove a function in hail
This is not yet used.
Iustin Pop [Sat, 23 May 2009 23:11:19 +0000 (00:11 +0100)]
Add some utility functions for kt deprecation
These will be used to remove even more uses of ktn/kti in non-critical
paths.
Iustin Pop [Sat, 23 May 2009 22:50:37 +0000 (23:50 +0100)]
Remove some extraneous uses of ktn/kti
Since we have Node/Instance.name, we can now simplify a few constructs.
Iustin Pop [Sat, 23 May 2009 22:47:56 +0000 (23:47 +0100)]
Strip the suffix from the names in the objects
This strips the suffix from the objects themselves, not only from the
ktn/kti vars.
Iustin Pop [Sat, 23 May 2009 22:33:21 +0000 (23:33 +0100)]
Make IAlloc.loadData return maps
This patch makes the format of IAlloc.loadData be similar the same as
Loader.mergeData.
Iustin Pop [Sat, 23 May 2009 22:32:52 +0000 (23:32 +0100)]
Some small layout changes
Iustin Pop [Sat, 23 May 2009 22:22:01 +0000 (23:22 +0100)]
Move common loading sequence in CLI
This patch moves the common loading sequence to CLI, such that hbal/hn1
and possible future scripts that take the input from same sources can
use it.
Iustin Pop [Sat, 23 May 2009 22:06:45 +0000 (23:06 +0100)]
Move checkData from Cluster to Loader
This moves the remaining loading function to Loader (together with its
associated support functions).
Iustin Pop [Sat, 23 May 2009 00:34:54 +0000 (01:34 +0100)]
Introduce a class for CLI options
This patch generalizes a little the CLI handling by not passing in a
special function for help and such, but instead requiring that the
options object supports some common functionality.
Iustin Pop [Sat, 23 May 2009 00:24:06 +0000 (01:24 +0100)]
Add a small class for Nodes and Instances
Since both nodes and instances support some common functionality (names
and indices), we add a class so that we can access these attributes in a
generic way.
Iustin Pop [Fri, 22 May 2009 23:34:51 +0000 (00:34 +0100)]
Add back names to nodes/instances
In order to simplify the data structures, we add back the name on the
node and instance objects. We still keep the index for, well, indexing,
but we will use the name directly from the object, in order to get rid
of the ktn/kti arguments which are passed around everywhere.
Iustin Pop [Fri, 22 May 2009 23:18:50 +0000 (00:18 +0100)]
More code reorganizations
This new big patch does a couple of more cleanups in the loading of data
chapter:
- introduce a Types module that holds most types (except the base
Node/Instance/etc.) so that multiple other modules can use these
(instead of only Cluster and its users)
- bring IAlloc.hs in line with the recent changes of providing data
types and not strings
- removal of obsolete Utils.hs json-related functions
Iustin Pop [Fri, 22 May 2009 17:03:53 +0000 (18:03 +0100)]
Rework the loader model
This big patch changes the loader model from “string data as common
format” to actual object structures as common format.
The text loading function move from Cluster.hs to a new Text.hs module,
some common functions are moved to a new Loader.hs module, and the
return values from both Rapi.hs and Text.hs are uniformized.
Iustin Pop [Thu, 21 May 2009 22:13:45 +0000 (23:13 +0100)]
Initial work on hail
This makes hail compile and get a request parsed via IAlloc, but nothing
more.
Iustin Pop [Thu, 21 May 2009 00:54:27 +0000 (01:54 +0100)]
Experimental support for non-redundant instances
This patch adds experimental support to hbal for non-redundant instances
(i.e. instances with only one node). They are currently handled as
non-moveable, and as such the algorithm simply ignores them.
Supports needs to be added when reading from RAPI via hscan, and
probably in other corner cases.
Iustin Pop [Thu, 21 May 2009 00:30:49 +0000 (01:30 +0100)]
Small doc addition
Iustin Pop [Thu, 21 May 2009 00:26:51 +0000 (01:26 +0100)]
Introduce nice errors on invalid input fields
This patch switches from plain read to a wrapper over readsPrec that
returns better error messages than the buildin 'Prelude: no parse'.
Iustin Pop [Thu, 21 May 2009 00:10:56 +0000 (01:10 +0100)]
Split node/instance parsing into functions
This allows easy checking for valid format of the input data (row-wise).
Iustin Pop [Wed, 20 May 2009 23:37:03 +0000 (00:37 +0100)]
Add initial validation checks in Cluster.loadData
This patch converts loadTabular and loadData to a monadic form, thus
allowing meaningful error messages from the node/instance load routines.
Iustin Pop [Wed, 20 May 2009 23:11:48 +0000 (00:11 +0100)]
Small syntax improvement
D'oh, one can extract from a wrapped variable, not only from functions.
Iustin Pop [Wed, 20 May 2009 23:07:24 +0000 (00:07 +0100)]
Convert Cluster.loadData to Result return
This patch changes Cluster.loadData to return a Result, instead of
directly the values; this will allow us to return meaningful error
values (e.g. when an instances lives on unknown node) rather than simply
abort. Currently the result is always an Ok, the actual signalling of
errors will come later.
Its callers are changed to accommodate for the new return type and to
display errors as needed.
Iustin Pop [Wed, 20 May 2009 22:31:10 +0000 (23:31 +0100)]
Allow hscan to save data for fully-spec'd URLs
In case we pass a fully-specified URL to hscan, currently it will use as
is and that means the directory in which we want to save the status
files changes from what we mean. This patch changes hscan to replace all
slashes in the cluster name with underscores when generating the output
filenames.
Iustin Pop [Wed, 20 May 2009 22:02:45 +0000 (23:02 +0100)]
Allow overriding the RAPI port/scheme
This patch adds a very dumb way to override the port and scheme for
RAPI: if the master string doesn't contain a colon, we consider it a
simple hostname specification, and thus prepend ‘https://’ and append
‘:5080’, (the default RAPI port); otherwise, we consider it a fully
specified URL, and don't do any mangling to it.
This allows to use saved RAPI responses in testing, with the “file://”
scheme, or to use another port or non https mode with an actual RAPI
instance.
Iustin Pop [Wed, 20 May 2009 21:55:52 +0000 (22:55 +0100)]
Remove some 1.2 specific code
Now that we dropped 1.2 compatibility, we can removed the special casing
for RAPI differences.
Iustin Pop [Wed, 20 May 2009 21:23:31 +0000 (22:23 +0100)]
Generalize some Result function into monad ones
We don't really needed, but is more clean like this.
Iustin Pop [Wed, 20 May 2009 21:13:02 +0000 (22:13 +0100)]
Switch from hand-written monads to a real one
This big patch converts from our home-grown monad-like constructs
(the Either stuff) to a real, Either-like-but-another-name monad.
We introduce a “Result a” monad, and this allows dropping many of the
extra constructs. Hopefully the code is also more clear.
Many of the functions could now be written in a generic-monad style,
instead of Result specifically, but that will come in future patches.
IAlloc.hs also has some unrelated patches.
Iustin Pop [Tue, 19 May 2009 19:53:01 +0000 (20:53 +0100)]
Small changes to the IAlloc module
Adding a small request type data structure.
Iustin Pop [Tue, 19 May 2009 22:04:53 +0000 (23:04 +0100)]
Update NEWS file for the 0.1.0 release
Iustin Pop [Tue, 19 May 2009 21:58:02 +0000 (22:58 +0100)]
Don't consider offline nodes as N+1 failed
This is just a cosmetic (I hope) change; the nodes shouldn't be used
anyway, and we only correct the display message.
Iustin Pop [Tue, 19 May 2009 21:50:12 +0000 (22:50 +0100)]
Drop RAPI v1 compatiblity
Since we only support Ganeti 2.0, we drop the 1.2 compatibility.
Iustin Pop [Tue, 19 May 2009 21:45:31 +0000 (22:45 +0100)]
Add support for 'offline' nodes
This patch drops compatiblity with Ganeti 1.2 and adds support for
offline nodes in the cluster. When reading from RAPI, the drained nodes
are considered offline so that we don't allocate on them too.
Iustin Pop [Mon, 18 May 2009 22:19:26 +0000 (23:19 +0100)]
Update the IAlloc module
We know have a working parseData function that returns the node and
instance data. This uncovered bad support for non-drbd instances ☹
Iustin Pop [Mon, 18 May 2009 08:24:24 +0000 (10:24 +0200)]
Move the JSON utilities to Utils.hs
This patch moves the generic/reusable JSON functions to Utils.hs, so
that they're shared between RAPI/IAlloc.
Iustin Pop [Mon, 18 May 2009 06:46:12 +0000 (08:46 +0200)]
Add a copy of Rapi.HS as IAlloc.hs
This will be used in two ways:
- format the response to Ganeti (easy, implemented)
- parse the input data and build the node/instance lists (hard :)
Iustin Pop [Mon, 18 May 2009 06:18:21 +0000 (08:18 +0200)]
Remove the apidoc dir on clean
Iustin Pop [Sun, 26 Apr 2009 14:15:33 +0000 (16:15 +0200)]
hbal: add a --quiet option
This option is the opposite of the --verbose option, and it allows
decreasing the verbosity level from the default of one to zero (which
currently doesn't show the warning messages for missing disk/memory).
Iustin Pop [Sat, 25 Apr 2009 15:38:22 +0000 (17:38 +0200)]
hbal: Simplify the oneline formatting
This patch moves the oneline format into a separate function for easier
usage.
Iustin Pop [Sat, 25 Apr 2009 15:32:46 +0000 (17:32 +0200)]
hbal: Early exit when we don't have any instances
For clusters with no instances, there is no point in computing either
the score or in running the algorithm. In this case, we exit prematurely
and when running in one-line mode we show dummy information.
Iustin Pop [Sat, 25 Apr 2009 15:27:27 +0000 (17:27 +0200)]
hbal: Add a new min-score option
This new parameter causes the algorithm to finish (or even not start at
all) if we reach/have a score better than it.
Iustin Pop [Sat, 25 Apr 2009 14:31:25 +0000 (16:31 +0200)]
hbal: Change hardcoded tests to monadic composition
In some case we manually do “if isNothing … then Nothing else …”, which
can be very easily replaced with a monadic construct in the Maybe monad.
Iustin Pop [Tue, 21 Apr 2009 07:05:52 +0000 (09:05 +0200)]
Fix the makefile dist rule
It was missing a dependency on the Version.hs file, so right after “make
clean”, a “make dist” used to fail.
Iustin Pop [Tue, 21 Apr 2009 06:50:04 +0000 (08:50 +0200)]
Update NEWS file for the 0.0.8 release
Iustin Pop [Mon, 20 Apr 2009 12:12:13 +0000 (14:12 +0200)]
Increase allowed missing memory to 512MB
Since Xen seems to “steal” some amounts of memory (depending on total
node memory), we increase the maximum allowed missing memory to 512MB,
based on gathered data from multiple machines.
Iustin Pop [Mon, 20 Apr 2009 11:13:17 +0000 (13:13 +0200)]
Update man pages with the env variables
This patch documents the environment variables in the man pages of hbal
and hn1.
Iustin Pop [Mon, 20 Apr 2009 11:07:09 +0000 (13:07 +0200)]
Add reading the file names from env vars
This patch adds support for selecting the instance/node file names via
two environment variables (HTOOLS_NODES, HTOOLS_INSTANCES).
Unfortunately we still have lots of duplicated code, since the options
are not unified.
Iustin Pop [Mon, 20 Apr 2009 11:00:10 +0000 (13:00 +0200)]
Implement error checks for extra arguments
Neither hbal nor hn1 take any arguments beside the options, so if any
are passed is most likely an unintended error. This patch aborts in such
cases.
Iustin Pop [Mon, 20 Apr 2009 10:00:51 +0000 (12:00 +0200)]
Fix the makefile clean rule
Use the $HPROGS variable instead of hardcoding the program names.
Iustin Pop [Mon, 20 Apr 2009 10:00:29 +0000 (12:00 +0200)]
Add a gitignore file
Iustin Pop [Sat, 18 Apr 2009 22:17:05 +0000 (00:17 +0200)]
Implement writing the command list to a script
This patch adds support in hbal for writing the command list to a shell
script, with error checking and allowing for early exit.
Iustin Pop [Thu, 16 Apr 2009 09:02:48 +0000 (11:02 +0200)]
hbal: Abort for invalid offline node names
Since it's easy to pass a wrong node name as offline, we should abort
instead of silently ignoring it.
Iustin Pop [Mon, 23 Mar 2009 07:26:08 +0000 (08:26 +0100)]
Updated new in preparation for the 0.0.7 release
Iustin Pop [Mon, 23 Mar 2009 07:12:40 +0000 (08:12 +0100)]
More documentation updates
This removes most of the content of the README file (obsoleted by new
algorithm and man pages), modifies the Makefile to include the built
documentation in the source archive (so that haddock/hscolour are not
needed) and updates the haddock-prologue with current information.
Iustin Pop [Mon, 23 Mar 2009 06:40:49 +0000 (07:40 +0100)]
More man page updates
This moves some data from README to the man pages and has other general
improvements.
Iustin Pop [Sun, 22 Mar 2009 22:32:50 +0000 (23:32 +0100)]
Add checks for missing disk space
This small patch adds disk space checks to the Cluster.checkData
function, and simplifies a little the warning messages.
Iustin Pop [Sun, 22 Mar 2009 22:30:42 +0000 (23:30 +0100)]
Include DRBD overhead in sda/sdb size
For Ganeti 1.2 which doesn't have the ‘disk_usage’ instance query field,
we need to manually include the DRBD overhead (per disk). This patch
modifies the RAPI collection to do this, but loading from disk does not
as it's unknown if the query came from hscan or RAPI 1.2 or RAPI 2.0...
Iustin Pop [Sun, 22 Mar 2009 22:12:16 +0000 (23:12 +0100)]
Documentation updates
This patch adds a man page for hscan and updates the README and other
man pages with the latest changes.
Iustin Pop [Sun, 22 Mar 2009 21:33:55 +0000 (22:33 +0100)]
Update all needed node fields on f_mem change
This fixes the setFmem function which didn't compute other related
fields after free memory change. Ideally, this should be abstracted so
that add/remove Pri and similar functions could reuse it instead of
duplicating code.
Iustin Pop [Sun, 22 Mar 2009 10:40:58 +0000 (11:40 +0100)]
Fix interaction between down instances and nodes
If an instance is down, it's memory is not reflected in the node used
memory, and thus the node free memory is higher than the actual value.
This patch deducts the memory for such instances from the node free
memory, allowing a correct calculation for such cases.
Iustin Pop [Sun, 22 Mar 2009 10:24:23 +0000 (11:24 +0100)]
Add a new instance field denoting run status
This patch modifies Rapi, the Cluster.loadData and hscan serialization to load
and save the instance run status. At instance level, we add both a boolean
field denoting the true/false run status, and a string field which holds the
original value (since we don't have a 1-to-1 mapping) for use in hscan
serialization.
The run status is not yet used.
Iustin Pop [Sun, 22 Mar 2009 10:03:52 +0000 (11:03 +0100)]
Show the x_mem/i_mem in node list
This patch adds checking of cluster data in the binaries and display of
node's x_mem/i_mem in the node list.
Iustin Pop [Sun, 22 Mar 2009 10:02:59 +0000 (11:02 +0100)]
Add functions to check and fix cluster data
This patch adds a checkData function which goes over the node list and computes
the unaccounted memory, returning a list of warning messages (if any) and the
update nodes.
Iustin Pop [Sun, 22 Mar 2009 09:55:12 +0000 (10:55 +0100)]
Add a new node filed x_mem
Nodes can have some memory unaccounted for, due to (e.g.) hypervisor
overhead, rounding errors in reporting, etc.
It is better if we model this memory explicitly instead of hiding it,
and actually since the n_mem addition it is actually required to do so.
The new attribute is not yet used.
Iustin Pop [Sun, 22 Mar 2009 09:52:50 +0000 (10:52 +0100)]
Split common CLI functionality into a module
This patch moves the common CLI functionality (as much as currently
possible) into a separate module. This means we only have one parseOpts
and that Utils.hs doesn't keep this kind of functions anymore.
Iustin Pop [Sun, 22 Mar 2009 00:18:22 +0000 (01:18 +0100)]
Remove unused and obsolete function
The Node.str function is very old and is not useful since the node
objects have much more fields today. This patch removes it, and if
needed a full node display can be done via ‘show’.
Iustin Pop [Sat, 21 Mar 2009 23:25:55 +0000 (00:25 +0100)]
Add node memory field to Node objects
This patch adds a new n_mem field to the node objects, and implements
read/save/show support for it. The field is not currently used (except
in the node list) but will be used for checking data consistency and
instance up/down status.
Iustin Pop [Sat, 21 Mar 2009 23:12:29 +0000 (00:12 +0100)]
Pass actual types to node/instance constructors
This patch changes the parameters passed to the node and instance
constructors from generic Strings (which are then parsed via “read”) to
the actual used types, by converting them earlier in Cluster.loadData.
Iustin Pop [Sat, 21 Mar 2009 23:06:51 +0000 (00:06 +0100)]
Small change in hscan
This fixes a mistake between Int/Integer. Should be more careful :)
Iustin Pop [Sat, 21 Mar 2009 22:51:54 +0000 (23:51 +0100)]
Add hscan to Makefile
Iustin Pop [Sat, 21 Mar 2009 22:50:47 +0000 (23:50 +0100)]
Add the hscan tool
This patch adds an hscan tool that loads data from clusters via RAPI and
writes it to files that can be later used offline.
Iustin Pop [Sat, 21 Mar 2009 22:48:49 +0000 (23:48 +0100)]
Some small changes in preparation for hscan
This patch does some small changes:
- fixes a comment
- export more node functions (unneeded now, but hscan will use them)
- fixes Makefile rule for building the programs
Iustin Pop [Sat, 21 Mar 2009 21:00:22 +0000 (22:00 +0100)]
Add a separate type for the [(Int, String)] list
This is added for better readability, since this is very often used in
declarations.
Iustin Pop [Sat, 21 Mar 2009 14:48:12 +0000 (15:48 +0100)]
Handle correctly offline nodes in cluster scoring
This patch changes two things with regard to offline nodes:
- first, it only calculates the various coefficients across online
nodes
- second, it adds a new score denoting the percentage of instances
which live on such nodes
The first change allows correct score computation in presence of offline
nodes (whose properties we don't need to take into account), while the
second change actively evacuates offline nodes.
Iustin Pop [Sat, 21 Mar 2009 11:20:15 +0000 (12:20 +0100)]
Show offline nodes in the node status list
This patch adds a new ‘-’ flag for the node status which denotes offline
nodes.
Iustin Pop [Fri, 20 Mar 2009 23:26:00 +0000 (00:26 +0100)]
Restrict move list based on offline node status
This patch changes the Cluster.checkInstanceMove function to restrict
the target move list based on which nodes are online.
Iustin Pop [Fri, 20 Mar 2009 22:48:07 +0000 (23:48 +0100)]
Add command line support for offlining nodes
This patch modifies hbal (only, hn1 not yet) for setting nodes offline.
Iustin Pop [Fri, 20 Mar 2009 22:45:15 +0000 (23:45 +0100)]
Add a new 'offline' Node attribute
This patch adds a new node attribute - offline - which will serve to
skip nodes from the target candidate list.
Iustin Pop [Fri, 20 Mar 2009 22:43:59 +0000 (23:43 +0100)]
More fixes to the Makefile
Iustin Pop [Fri, 20 Mar 2009 21:46:09 +0000 (22:46 +0100)]
Small doc update in Node.hs
Iustin Pop [Fri, 20 Mar 2009 18:58:35 +0000 (19:58 +0100)]
Some updates to the apidoc rules
Iustin Pop [Fri, 20 Mar 2009 18:26:10 +0000 (19:26 +0100)]
Fix/enhance makefile rules after the rename
Iustin Pop [Fri, 20 Mar 2009 17:17:27 +0000 (18:17 +0100)]
Add a .gitattributes file
This will enhance the ‘dist’ rule by skipping unneeded files.
Iustin Pop [Fri, 20 Mar 2009 17:16:21 +0000 (18:16 +0100)]
Introduce a namespace for the modules
The modules are moved from the ‘top’ namespace to ‘Ganeti.HTools’, in
compliance with standard practices.
Iustin Pop [Mon, 16 Mar 2009 07:35:48 +0000 (08:35 +0100)]
Update NEWS for version 0.0.6
Iustin Pop [Sat, 14 Mar 2009 19:25:41 +0000 (20:25 +0100)]
Abstract the version format into a function
This patch moves the version string creation into a function in Utils
which shows some more information.
Iustin Pop [Sat, 14 Mar 2009 19:12:02 +0000 (20:12 +0100)]
Add a man page for hn1 and update the hbal one
A new man page and typos fixed in hbal.1.
Iustin Pop [Sat, 14 Mar 2009 11:53:43 +0000 (12:53 +0100)]
Add a manpage for hbal
Iustin Pop [Sat, 14 Mar 2009 11:49:43 +0000 (12:49 +0100)]
Add a --version option
This patch adds a -V, --version command line option that shows the
program version and also updates the hn1 usage string (similar to hbal).
Iustin Pop [Sat, 14 Mar 2009 11:41:13 +0000 (12:41 +0100)]
Move a function around in hbal.hs
This just reorders some functions for a more logical ordering.
Iustin Pop [Sat, 14 Mar 2009 11:35:04 +0000 (12:35 +0100)]
Show the step counter in the solution list
This patch changes the solution list to include a step counter so that
it's more clear these are successive steps (in a definite order), and
not just an unordered list of changes.
Iustin Pop [Sat, 14 Mar 2009 11:16:52 +0000 (12:16 +0100)]
Use gnt-instance migrate instead of failover
This patch changes the gnt-instance failover to migrate, and fixes a bug
in the formatting of commands.
Iustin Pop [Sat, 14 Mar 2009 11:14:17 +0000 (12:14 +0100)]
hbal: added a verbose setting and changed output
This patch added a verbose output and changed the output so that by
default it is less verbose and more clear.
Iustin Pop [Sat, 14 Mar 2009 09:11:36 +0000 (10:11 +0100)]
Add a new move FailoverAndReplace
This patch adds a new instance move, FailoverAndReplace, which promotes
the old secondary to primary and then uses a new secondary node.
This is the last move that we can do within the limitations of one node
changed per move.
Iustin Pop [Fri, 13 Mar 2009 19:42:56 +0000 (20:42 +0100)]
Some more docstring updates
Iustin Pop [Fri, 13 Mar 2009 19:11:12 +0000 (20:11 +0100)]
Enhance the command list for the solution
This patch moves the formatting of the command list to Cluster.hs and
enhances it with separator messages between the steps.
Iustin Pop [Fri, 13 Mar 2009 18:53:41 +0000 (19:53 +0100)]
Add a new ReplaceAndFailover move
This patch adds a new replace secondary and failover move (equals to
“r:x f”), which can improve the solution (since we are testing more
options at each step).
Iustin Pop [Fri, 13 Mar 2009 18:52:44 +0000 (19:52 +0100)]
Some whitespace changes
Aligned the comments in Instance.hs
Iustin Pop [Fri, 13 Mar 2009 07:49:12 +0000 (08:49 +0100)]
Convert hbal from multiple rounds to a step-method
Currently hbal does multiple rounds, stopping when a rounds doesn't
bring improvements. With the recent changes to not remove instances from
the candidate list, this is obsolete as the first round will always run
to the end of the improvements.
This patch changes this so that the Cluster.checkMove function doesn't
recurse, but just computes the next best move (as its docstring says).
This means we can actually incrementally compute and print the solution,
and this is needed as otherwise an instance could move twice and the
second time it needs the current placement to compute the exact command
line and operation needed for the move.