Iustin Pop [Tue, 7 Jul 2009 08:01:30 +0000 (10:01 +0200)]
Fix score calculation to work with empty clusters
Currently the cluster score calculation includes an offline instance
percentage, expressed as “offline inst / (offline + online inst)”, which
results in NaN for empty clusters. This patch changes the calculation
such that clusters with no offline instances (this criteria includes
completely empty clusters) get a score of zero on this component of the
total score.
Iustin Pop [Mon, 6 Jul 2009 15:58:14 +0000 (17:58 +0200)]
hspace: convert N1 error exit into FailN1 result
Currently hspace exits with an error if the cluster is not N+1 compliant
at the beginning of the run. This patch changes hspace such that this
condition is instead treated as a zero-allocation-possible, FailN1 mode.
This allows the same stats to be reported in this case as in the normal
case.
Iustin Pop [Mon, 6 Jul 2009 14:46:51 +0000 (16:46 +0200)]
Some docstring updates
Iustin Pop [Mon, 6 Jul 2009 14:40:01 +0000 (16:40 +0200)]
hspace: add display of instance spec
This is mostly for user-friendliness in the default mode, when we don't
specify the instance parameters.
Iustin Pop [Mon, 6 Jul 2009 22:20:41 +0000 (00:20 +0200)]
Optimize the Utils.stdDev function
This patch optimizes the stdDev function in two respects:
- first, we don't do sum . map which builds an intermediate list, but
instead use a fold over the list to build incrementally the sum;
this should reduce both the time and space characteristics, as we
have fewer objects created
- second, we move from “a ^ 2” to “a * a” as the latter has a much
simpler implementation and thus a higher performance
Since the ‘square’ function is obsoleted by the above the patch also
removes it.
Iustin Pop [Mon, 6 Jul 2009 21:24:14 +0000 (23:24 +0200)]
Take the foldl out of Loader.fixNodes
Currently Loader.fixNodes is foldl' with a complicated function. It
makes more sense to take foldl' out of this function (and put it into
the caller) and let fixNodes be only this internal function.
Iustin Pop [Mon, 6 Jul 2009 21:13:15 +0000 (23:13 +0200)]
Simplify Cluster.computeMoves
This patch changes the function Cluster.computeMoves to use guards and a
couple of subexpressions in order to greatly simplify it.
Iustin Pop [Mon, 6 Jul 2009 20:50:18 +0000 (22:50 +0200)]
Fix hlint-generated warnings
This big patch cleans up the code per hlint indications. Many removals
of extra parentheses, replacements of concat . map with concabtMap,
extra dollar signs, eta reductions, etc. were performed.
The code still compiles and passes a couple of manual tests on sample
files. The individual changes are also small enough as to be visually
easy to confirm.
Iustin Pop [Sun, 5 Jul 2009 15:56:10 +0000 (18:56 +0300)]
Add computation of the failure reason in hspace
This patch enhances hspace to report why the allocation sequence
stopped, both in absolute error count and for the top reason.
Iustin Pop [Sun, 5 Jul 2009 13:09:43 +0000 (16:09 +0300)]
Return correct failure data from Node.add*
This patch alters the Node.addPri/addSec to return correct failure data.
It removes the computeFailN1 function from the module as that used to
combine both mem and disk checks in the same function and thus the real
error reason cannot be deduced when using it.
Iustin Pop [Sun, 5 Jul 2009 12:53:40 +0000 (15:53 +0300)]
Introduce a new type for allocation results
Currently the allocation/move operations workflow return ‘Maybe a’,
which is very convenient but loses all details about the failure mode.
This patch introduces a new data type which encodes the specific failure
mode. It is not yet used correctly (e.g. all node operations result in
FailN1), but the workflow is updated. Most of the changes are in the
hail/hspace tools, the library code required only trivial adjustments.
Iustin Pop [Sun, 5 Jul 2009 12:21:56 +0000 (15:21 +0300)]
Remove hn1 and related code
hn1 was deprecated for a while and this patch removes it altogether. The
support code in Cluster.hs is also removed.
Iustin Pop [Fri, 3 Jul 2009 19:40:35 +0000 (22:40 +0300)]
Display two more stats in hspace
This adds two new stats - sum of reserved ram and disk.
Iustin Pop [Fri, 3 Jul 2009 18:37:51 +0000 (21:37 +0300)]
Fix totalResources avail disk computation
This uses the newly-added Node.availDisk to compute the actual available
disk correctl, and display the total allocatable disk in hspace.
Iustin Pop [Fri, 3 Jul 2009 17:22:06 +0000 (20:22 +0300)]
Add an availDisk node function
This function returns the amount of available disk, which depends on
whether a low disk limit has been configured or not and on the free disk
space of the node.
Iustin Pop [Fri, 3 Jul 2009 17:02:48 +0000 (20:02 +0300)]
Add two new autocomputed vars to Nodes
Currently we track the max disk usage/max vcpus as percentages, however
sometimes it's easier to check against minimum free disk or maximum
number of cpus, as units instead of percentages.
This patch adds two new variables, lo_dsk, hi_cpu, which are recomputed
whenever we change the m_dsk/m_cpu 'master' values.
Iustin Pop [Thu, 2 Jul 2009 21:50:50 +0000 (23:50 +0200)]
Add a new type for cluster statistics
Currently totalResources returns a 5-tuple of integers. This is not easy
to handle, as each change on the return type means that each caller must
be updated.
This patch adds a new type for cluster stats and uses that instead as
its return type. This simplifies its callers while complicating only a
little the computation of the stats.
Iustin Pop [Thu, 2 Jul 2009 10:43:33 +0000 (12:43 +0200)]
Enhance hspace resource display
The display of cluster resources is extracted into a separate function
and enhanced to display more stats.
Iustin Pop [Thu, 2 Jul 2009 10:33:06 +0000 (12:33 +0200)]
Add display of more stats in hspace
This patch changes Cluster.totalResources to compute more details about
the cluster status, and enhances hspace to display more of these.
Iustin Pop [Tue, 16 Jun 2009 09:50:30 +0000 (11:50 +0200)]
Update NEWS file for the 0.1.4 release
Iustin Pop [Tue, 16 Jun 2009 09:52:09 +0000 (11:52 +0200)]
Fix a haddock/docstring issue
Iustin Pop [Fri, 12 Jun 2009 22:58:33 +0000 (00:58 +0200)]
Fix some hscan bugs
Currently hscan has a number of bugs:
- doesn't add the common suffix (csf) to the instance's nodes
- doesn't export the cpus for neither nodes nor instances
- doesn't support single-node instances
This patch fixes these issues.
Iustin Pop [Fri, 12 Jun 2009 21:50:07 +0000 (23:50 +0200)]
Some documentation updates for the new parameters
Iustin Pop [Fri, 12 Jun 2009 21:33:02 +0000 (23:33 +0200)]
Add cpu/disk limits in hbal
Iustin Pop [Fri, 12 Jun 2009 00:17:09 +0000 (02:17 +0200)]
Add setting of node limits in hspace
Iustin Pop [Fri, 12 Jun 2009 00:12:27 +0000 (02:12 +0200)]
Implement cpu/disk limits in instance moves
We modify Node.addPri/addSec to take into account the limits on instance
adds.
Iustin Pop [Thu, 11 Jun 2009 23:47:27 +0000 (01:47 +0200)]
Add two new node attributes
Two new min disk free ratio and max cpu usage attributes are added to the
nodes. These will be used in the future to restrict allocation.
Iustin Pop [Thu, 11 Jun 2009 23:29:31 +0000 (01:29 +0200)]
Fix 'unused X' warnings
This removes some unused functions and imports to cleanup the warnings.
Iustin Pop [Thu, 11 Jun 2009 23:22:20 +0000 (01:22 +0200)]
Fix the various monomorphism warning
In a few places (e.g. tryRead or any printf call) it's a little bit hard
to add the correct type signatures, but in the it is possible to fix
these warnings (which can bite one in subtle cases).
Iustin Pop [Thu, 11 Jun 2009 22:12:40 +0000 (00:12 +0200)]
Small changes to the node list output
This is just some cleanup of the node list output, adding pcpu/vcpu
counters, and making the display slightly nicer.
Iustin Pop [Wed, 10 Jun 2009 21:17:40 +0000 (23:17 +0200)]
Add cpu ratio to cluster calculation
Iustin Pop [Wed, 10 Jun 2009 20:37:34 +0000 (22:37 +0200)]
Update cpu counters correctly after pinst changes
The cpu counters are update on primary instance adds/removes.
Iustin Pop [Wed, 10 Jun 2009 20:29:22 +0000 (22:29 +0200)]
Add cpu-count-related attributes to nodes
This patch adds cpu-count related attributes to nodes:
- total cpus
- cpus in use
- ratio of virtual:physical cpus
We also set correctly the cpu values at load time, but we don't do
anything yet while moving instances around. The cpu ratio is shown in
the cluster list.
Iustin Pop [Wed, 10 Jun 2009 19:42:45 +0000 (21:42 +0200)]
Add a new vcpus attribute to instances
This patch adds reading of vcpu count for instances, in preparation for
using the vcpu ratio in cluster scoring.
Iustin Pop [Wed, 10 Jun 2009 19:31:05 +0000 (21:31 +0200)]
Fix reading of total disk space in iallocator
IAllocator currently uses a wrong key name for reading the total disk
space (‘disk_usage’ which was copied from RAPI, but the actual
iallocator key is ‘disk_space_total’).
This patch fixes that and also makes iallocator always use this key,
instead of computing the size by itself—with Ganeti 2.0 this is
key is always available.
Iustin Pop [Fri, 5 Jun 2009 10:13:20 +0000 (12:13 +0200)]
Update NEWS and README for the 0.1.3 release
Small updates to the documentation and make a new small release.
Iustin Pop [Thu, 4 Jun 2009 13:32:59 +0000 (15:32 +0200)]
Fix the ReplacePrimary instance move
During a replace-primary instance move, on the real cluster the instance
is temporarily started on the secondary, and as such we must check that
the secondary node can hold it for this duration. Currently the code
does not, and depending on cluster scoring it will put instances on such
'bad' secondaries (usually without enough memory).
This patch fixes this by adding/removing the instance as a primary on
the secondary node; and then using the result node as the new secondary.
(Since we're in a monad, we could have just tried the allocation and
dropped the result, but this is more clear).
Iustin Pop [Tue, 2 Jun 2009 06:44:59 +0000 (08:44 +0200)]
Update NEWS file for the 0.1.2 release
Iustin Pop [Mon, 1 Jun 2009 20:15:34 +0000 (22:15 +0200)]
Update the README file with hspace informations
Iustin Pop [Mon, 1 Jun 2009 14:21:51 +0000 (16:21 +0200)]
Fix hspace with plain type instances
This also fixes other required node numbers.
Iustin Pop [Mon, 1 Jun 2009 14:05:58 +0000 (16:05 +0200)]
Add a man page for hspace
Iustin Pop [Mon, 1 Jun 2009 13:48:44 +0000 (15:48 +0200)]
Rework the tryAlloc/tryReloc functions
Currently tryAlloc/tryReloc do not return the new instance, as this is
not needed for IAllocator alloc/reloc requests. However, for computing
the space, the new instance is useful, so we modify these functions to
return this information too.
The patch also improves hspace to have (with default parameters) a
parseable output.
Iustin Pop [Mon, 1 Jun 2009 13:48:18 +0000 (15:48 +0200)]
Add an utility function for triples
Iustin Pop [Mon, 1 Jun 2009 11:59:02 +0000 (13:59 +0200)]
Initial add of the hspace tool
This is a tool that checks how many instances (of same size, specified
by command line arguments) can be added to a cluster while remaining N+1
compliant.
Iustin Pop [Mon, 1 Jun 2009 11:44:53 +0000 (13:44 +0200)]
Small doc change
And an alignment issue.
Iustin Pop [Mon, 1 Jun 2009 10:25:46 +0000 (12:25 +0200)]
Ensure consistent naming of the tools
This patch makes sure that all references to the name of the software is
ganeti-htools, not simply htools.
Iustin Pop [Mon, 1 Jun 2009 10:16:21 +0000 (12:16 +0200)]
Small documentation update
Iustin Pop [Mon, 1 Jun 2009 09:55:48 +0000 (11:55 +0200)]
Add copyright/license information
This doc-patch adds copyright and license information to (hopefully) all
needed files.
Iustin Pop [Mon, 1 Jun 2009 09:24:27 +0000 (11:24 +0200)]
tests: move the test declaration in QC.hs
This patch moves the test declaration into QC.hs, so that test.hs has to
be modified only when we add a new test category.
Iustin Pop [Sun, 31 May 2009 20:28:12 +0000 (22:28 +0200)]
Small whitespace change
Iustin Pop [Sun, 31 May 2009 20:22:53 +0000 (22:22 +0200)]
Move some alloc functions from hail into Cluster
These are generic enough to be used from multiple places, they belong
better in Cluster.hs than in the hail source.
Iustin Pop [Fri, 29 May 2009 18:02:19 +0000 (20:02 +0200)]
Move the RqType and Request types to Loader.hs
These two will be more generic than now, and belong somewhere else -
Loader.hs is a generic module for data loading, thus we move them there.
Iustin Pop [Fri, 29 May 2009 17:53:04 +0000 (19:53 +0200)]
Cleanup an old function
Also replace a type with its synonim.
Iustin Pop [Fri, 29 May 2009 17:25:34 +0000 (19:25 +0200)]
Lots of documentation updates
This patch does only doc build changes, doc changes and function move
around (for more logical documentation). It should have no impact at all
on the code.
Iustin Pop [Fri, 29 May 2009 08:42:01 +0000 (09:42 +0100)]
Change the check rule in Makefile
Since ghc won't trigger recompilation due to the -fhpc flag, it's not
useful to rm && make test, as this will only relink the binary.
Therefore we simplify this rule.
Iustin Pop [Wed, 27 May 2009 22:36:40 +0000 (23:36 +0100)]
A simple test for Container.addTwo
Iustin Pop [Wed, 27 May 2009 22:10:14 +0000 (23:10 +0100)]
Add some very trivial Instance tests
This is more of an exercise in QuickCheck than strong testing.
Iustin Pop [Wed, 27 May 2009 21:50:22 +0000 (22:50 +0100)]
Finish removal of unused params from PeerMap
This completes the removal started earlier byt removeing the need to
pass the number of nodes to Node.buildPeers, which is now unused.
Iustin Pop [Wed, 27 May 2009 21:22:38 +0000 (22:22 +0100)]
Add test infrastructure and initial tests
This patch adds a QuickCheck-based test infrastructure and initial tests
based on it. The PeerMap module has a 100% coverage ☺
Side-note: one has to read the source of QuickCheck to see how to use it
(especially the Batch submodule), the docs are not enough…
Iustin Pop [Wed, 27 May 2009 20:50:02 +0000 (21:50 +0100)]
Some cleanup of the PeerMap module
This patch removes some unused functions and does some cleanup of the
remaining ones.
Iustin Pop [Wed, 27 May 2009 20:09:47 +0000 (21:09 +0100)]
Remove unused parameters from PeerMap creation
We remove some unused arguments (added way back for compatibility with
Arrays, which we didn't use in the end). This makes the code clearer
(and doesn't need the Ndx type to be an instance of Num).
Iustin Pop [Wed, 27 May 2009 20:01:20 +0000 (21:01 +0100)]
Remove an unused type synonim
Iustin Pop [Wed, 27 May 2009 19:53:51 +0000 (20:53 +0100)]
Add a separate function for looking up instances
Currently we (wrongly) use lookupNode to lookup instances, just because
the name assoc list has the same type. This patch adds a separate
function for it.
Iustin Pop [Wed, 27 May 2009 19:25:09 +0000 (20:25 +0100)]
Add type synonyms for the node/instance indices
This is a first step towards full datatype renaming. That requires more
changes, so at first we only want to document clearly what is a node
index, what is an instance index, and what is a plain Int.
Iustin Pop [Tue, 26 May 2009 23:17:36 +0000 (00:17 +0100)]
Change the module import hierarchy
This patch makes the Types module a base module, and Node/Instance ones
import it, from the previous (opposite) situation. This will allow in
the future to use newtypes for the index and name types.
Iustin Pop [Tue, 26 May 2009 09:56:53 +0000 (10:56 +0100)]
Update NEWS file for the second attempt at 0.1.1
Iustin Pop [Tue, 26 May 2009 09:54:09 +0000 (10:54 +0100)]
Add a maintainer-clean makefile rule
This splits the current “clean” rule into proper clean (cleaning of
build artifacts) and maintainer-clean (cleaning of distributed files).
This should make it better for Debian packaging.
Iustin Pop [Tue, 26 May 2009 09:28:25 +0000 (10:28 +0100)]
Port offline node fixes from Rapi to IAllocator
The IAllocator source was copied from Rapi before the offline node fixes
were made. This changes such that offline nodes are accepted correctly.
Iustin Pop [Mon, 25 May 2009 20:47:51 +0000 (21:47 +0100)]
Update NEWS file for the 0.1.1 release
Also replace tabs in an older entry with spaces.
Iustin Pop [Mon, 25 May 2009 20:35:54 +0000 (21:35 +0100)]
Fix loading of plain instances via iallocator
Currently iallocator is broken when reading single-node instances (and
with an ugly error message). This patch fixes this case, by marking them
with secondary node “noSecondary” like the rest of the code.
Iustin Pop [Mon, 25 May 2009 19:54:15 +0000 (20:54 +0100)]
hail: don't relocate to current primary node
This patch fixes the relocate-to-primary-node problem, and doesn't mark
the excluded nodes as offline anymore (we don't use them anyway, so
let's not use a different/fake cluster state).
Iustin Pop [Mon, 25 May 2009 19:22:56 +0000 (20:22 +0100)]
Fix some haddock issues
Slash is a reserved char. Slash is a reserved char. Slash is a…
Iustin Pop [Mon, 25 May 2009 19:11:52 +0000 (20:11 +0100)]
Update README for hail
Iustin Pop [Mon, 25 May 2009 19:04:03 +0000 (20:04 +0100)]
Add man page and .gitignore entry for hail
Iustin Pop [Mon, 25 May 2009 18:50:54 +0000 (19:50 +0100)]
hail: do not allocate on offline/drained nodes
This patch implements filtering out of the offline/drained nodes and
fixes a bug in IAllocator.hs parsing (similar to an older bug in Rapi.hs
from where the code was copied).
Iustin Pop [Mon, 25 May 2009 18:35:03 +0000 (19:35 +0100)]
hail: cleanup unused options
Iustin Pop [Mon, 25 May 2009 18:31:03 +0000 (19:31 +0100)]
hail: Implement non-mirrored instance allocation
This patch implements non-mirrored instance allocation, by allocating as
secondary node “noSecondary”.
Iustin Pop [Mon, 25 May 2009 17:45:46 +0000 (18:45 +0100)]
hail: unify the post-processing of results
Both allocate and relocate compute new node lists, whose score must be
ranked and the best option chosen. This means we can convert the code to
a generic function.
Iustin Pop [Sun, 24 May 2009 23:09:55 +0000 (00:09 +0100)]
Implement hail allocate (for 2-node requests)
This patch implements allocate for two node requests. One node requests
can be done as soon as we have a valid allocateOn function for single
nodes.
Iustin Pop [Sun, 24 May 2009 22:48:07 +0000 (23:48 +0100)]
Working implementation if relocate
This patch completes the implementation of hail relocate. It maps all
valid destination nodes through a ReplaceSecondary IMove, filters out
the failed relocations, computes the resulting scores and picks the
lowest one.
Iustin Pop [Sun, 24 May 2009 22:07:08 +0000 (23:07 +0100)]
Start implementing the hail functionality
This patch implements a very stupid (and broken) version of hail
‘allocate’.
Iustin Pop [Sat, 23 May 2009 23:43:38 +0000 (00:43 +0100)]
Remove ktn/kti from first half of loader
This patch removes the ktn/kti lists from most parts of the first half
of the loading sequence. Some remain as the [(String, Int)] is the
nicest way to lookup names and get indices back.
Iustin Pop [Sat, 23 May 2009 23:36:36 +0000 (00:36 +0100)]
Remove the ktn/kti from second half of loading
This removes the return of ktn/kti from Loader.mergeData and associated
functions.
Iustin Pop [Sat, 23 May 2009 23:29:52 +0000 (00:29 +0100)]
Remove most uses of ktn/kti
This patch removes all uses of ktn/kti from the past-loader stages.
Iustin Pop [Sat, 23 May 2009 23:23:15 +0000 (00:23 +0100)]
Remove a function in hail
This is not yet used.
Iustin Pop [Sat, 23 May 2009 23:11:19 +0000 (00:11 +0100)]
Add some utility functions for kt deprecation
These will be used to remove even more uses of ktn/kti in non-critical
paths.
Iustin Pop [Sat, 23 May 2009 22:50:37 +0000 (23:50 +0100)]
Remove some extraneous uses of ktn/kti
Since we have Node/Instance.name, we can now simplify a few constructs.
Iustin Pop [Sat, 23 May 2009 22:47:56 +0000 (23:47 +0100)]
Strip the suffix from the names in the objects
This strips the suffix from the objects themselves, not only from the
ktn/kti vars.
Iustin Pop [Sat, 23 May 2009 22:33:21 +0000 (23:33 +0100)]
Make IAlloc.loadData return maps
This patch makes the format of IAlloc.loadData be similar the same as
Loader.mergeData.
Iustin Pop [Sat, 23 May 2009 22:32:52 +0000 (23:32 +0100)]
Some small layout changes
Iustin Pop [Sat, 23 May 2009 22:22:01 +0000 (23:22 +0100)]
Move common loading sequence in CLI
This patch moves the common loading sequence to CLI, such that hbal/hn1
and possible future scripts that take the input from same sources can
use it.
Iustin Pop [Sat, 23 May 2009 22:06:45 +0000 (23:06 +0100)]
Move checkData from Cluster to Loader
This moves the remaining loading function to Loader (together with its
associated support functions).
Iustin Pop [Sat, 23 May 2009 00:34:54 +0000 (01:34 +0100)]
Introduce a class for CLI options
This patch generalizes a little the CLI handling by not passing in a
special function for help and such, but instead requiring that the
options object supports some common functionality.
Iustin Pop [Sat, 23 May 2009 00:24:06 +0000 (01:24 +0100)]
Add a small class for Nodes and Instances
Since both nodes and instances support some common functionality (names
and indices), we add a class so that we can access these attributes in a
generic way.
Iustin Pop [Fri, 22 May 2009 23:34:51 +0000 (00:34 +0100)]
Add back names to nodes/instances
In order to simplify the data structures, we add back the name on the
node and instance objects. We still keep the index for, well, indexing,
but we will use the name directly from the object, in order to get rid
of the ktn/kti arguments which are passed around everywhere.
Iustin Pop [Fri, 22 May 2009 23:18:50 +0000 (00:18 +0100)]
More code reorganizations
This new big patch does a couple of more cleanups in the loading of data
chapter:
- introduce a Types module that holds most types (except the base
Node/Instance/etc.) so that multiple other modules can use these
(instead of only Cluster and its users)
- bring IAlloc.hs in line with the recent changes of providing data
types and not strings
- removal of obsolete Utils.hs json-related functions
Iustin Pop [Fri, 22 May 2009 17:03:53 +0000 (18:03 +0100)]
Rework the loader model
This big patch changes the loader model from “string data as common
format” to actual object structures as common format.
The text loading function move from Cluster.hs to a new Text.hs module,
some common functions are moved to a new Loader.hs module, and the
return values from both Rapi.hs and Text.hs are uniformized.
Iustin Pop [Thu, 21 May 2009 22:13:45 +0000 (23:13 +0100)]
Initial work on hail
This makes hail compile and get a request parsed via IAlloc, but nothing
more.
Iustin Pop [Thu, 21 May 2009 00:54:27 +0000 (01:54 +0100)]
Experimental support for non-redundant instances
This patch adds experimental support to hbal for non-redundant instances
(i.e. instances with only one node). They are currently handled as
non-moveable, and as such the algorithm simply ignores them.
Supports needs to be added when reading from RAPI via hscan, and
probably in other corner cases.
Iustin Pop [Thu, 21 May 2009 00:30:49 +0000 (01:30 +0100)]
Small doc addition