Synnefo » snf-ganeti

Rework the types used during data loading

This improves on the previous change. Currently, the node and instance
lists shipped around during data loading are (again) association lists.
For instances it's not a big issue, but the node list is rewritten
continuously while we assign instances to nodes, and that is very slow....

Convert some leftovers to NameAssoc

The type alias NameAssoc has been introduced a long time ago, but there
are some few not-yet-converted cases. In preparation for changes to that
type, let's make sure we use it consistently.

Signed-off-by: Iustin Pop <iustin@google.com>...

Add Cluster.splitCluster for node groups

This splits a top-level cluster information into the component node
groups. Instance go to the group of their primary node, but otherwise we
don't disallow split instances.

Signed-off-by: Iustin Pop <iustin@google.com>...

Rework Container.hs and improve test coverage

Since some of the functions we export from Container.hs are 1:1
identical to IntMap, we can just export the originals and remove the
wrappers. This reduces the code we need to unittest.

Furthermore, we add two simple unittest for the two non-trivial...

Add two functions for checking cluster consistency

For now, we don't support instances allocated across two groups, and we
will reject such clusters. The isClusterConsistent function will return
a list of inconsistent instances, potentially allowing operation without...

Add a type alias for UUIDs

This is to pottentially allow easier changes later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

RAPI: read the group UUID from the server

This depends on future support from Ganeti (2.4+).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Text: read/save the node group UUID

Compatibility with old text files is kept by using the default UUID if
the file (or even some records) don't have a UUID.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Node: add the node group's UUID

This is not used anywhere yet, and the backend are all just adding the
default UUID, not the real one.

The patch also allows displaying the group UUID in the node list.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Merge branch 'devel-0.2' into master

hbal: change handling of signal

Currently, hbal does a one-two signal handling, where the first signal
causes graceful termination, and the second one an immediate on (either
SIGINT or SIGTERM can be used, interchangeably). However, this poses a
timing problem: if two programs want to send a graceful termination...

Luxi loader: split parsing from loading

Text loader: split parsing from loadData

This change, which will be followed by similar changes in the other
loaders, splits the parsing of the data from the actual loading from
disk. Since the parsing doesn't usually involve IO actions, we will be
able to better test the parsing. The loading becomes a smaller part of...

Merge branch 'devel-0.2'

• devel-0.2:
  Fix tag exclusion weight

Force UTF-8 locale for pandoc invocation

Pandoc 1.5.x uses the locale information to parse its input files (only
1.5, pre and post version use always UTF-8). Hence we need to enforce a
UTF-8 locale for proper parsing of input files.

Add design for htools/Ganeti 2.3 sync

This is a work in progress, will be modified along with the progress
of Ganeti 2.3.

Fix some warnings in unittests

Improve the error message for tiered alloc option

Document the gain options in hbal's manpage

Add new CLI options for min gain during balancing

Recent hbal seems to run many steps for small improvements (< 1e-3), so
we should stop early in this case.

We add a new option (-g), that will be used for the minimum gain during
balancing. This check will only become active when the cluster score is...

Add some more debugging functions

These are just variations of the standard debug, but are provided for
simpler code, since lazyness is something causing non-computation of
debug statements.

Update NEWS file

hspace: mark new instances as running

Otherwise the saved cluster state and the in-memory one are wrong.

Change iterateAlloc to return the instance list

The Cluster.iterateAlloc and tieredAlloc functions are changed to also
return the updated instance list, since it is needed to have a “full”
cluster view.

Abstract the cluster serialization from hscan.hs

This is currently hardcoded in an internal function in hscan.hs, and we
move it to Text.hs for later use.

Add unittest for Node text serialization

This checks that the Node text serialization and deserialization
operations are idempotent when combined other.

Move text serialization functions to Text.hs

Currently these are in hscan, and cannot be reused easily.

hail: fix error message for failed multi-evac

Currently we show the instance index, but this makes no sense outside
the current running program. Instead, we show the instance name.

NEWS: Add double blank lines before headers

This looks better for text-only viewing…

More enhancements to live-test.sh

Remove an obsolete function and add Utils tests

Use --union for hpc sum

… which fixes the issue noted in the previous commit (almost a brown
paper bag change).

Add some more imports to QC.hs

This is needed so that in the coverage report we list all modules, even
the ones we don't test at all, such that we get the complete results.

Introduce per-metric weights

Currently all metrics have the same weight (we just sum them together).
However, for the hard constraints (N+1 failures, offline nodes, etc.)
we should handle the metrics differently based on their meaning. For
example, an instance living on a primary offline node is worse than an...

Introduce a relaxed add instance mode

In case an instance is living on an offline node, it doesn't make sense
to refuse moving it because that would create N+1 failures; failing N+1
is still much better than not running at all. Similarly, if the
secondary node of an instance is offline, meaning the instance doesn't...

hbal: print short names in steps list

This was a regression from the name handling changes, as we started
using the original names for the solution list (which is not designed
for parsing/feeding back into ganeti).

Allow '+' in node list fields

When the field list is prefixed with a plus sign, this will extend the
default field list, instead of replacing it entirely.

Cleanup a node's peer map when possible

If the last secondary instance of a peer is deleted (detected by the new
peer memory value being equal to zero), then the pair (pdx, 0) should be
deleted completely. This is not optimization per se, but rather cleanup...

Fix another haddock special-char issue

Add opcode status constants/type

This mirrors, again, the Ganeti constats, and are added for future use.

Optimise the Luxi.recvMsg function

Since the current buffer cannot contain (during network reads) an EOM,
we should look for the EOM only in the newly-received string. While
this shouldn't make much difference, in some tests it cuts the recvMsg
total time by around half....

Add support for more LUXI calls

While not are directly useful, having them will open some possibilities
(e.g. polling for job changes in hbal's -X mode, and auto-archiving the
jobs once they are successful).

Change the Luxi operations structure

Currently, we define the LuxiOp type as a simple enumeration, and leave
the arguments structure to the users of the Ganeti.Luxi module. This is
suboptimal for a couple of reasons: first, we decouple the operation
type from operation arguments, and that means we don't use the type...

Add a few Loader tests

These are not comprehensive, but at least we have a start.

Reduce the warnings during the unittests

Since the unittests are not 'clean' from the p.o.v. of type
declarations, and cannot be made clean in all respects (e.g. orphan
instances), we silence some warnings for the test target, to have a
cleaner output.

Introduce OpCode unittests

Replace fromJResult with annotateJResult

This patch removes all old uses of fromJResult with the annotated
version, and removes the non-annotated version. All JSON parsing points
should now have annotated errors.

Change fromObj error messages

Currently fromObj doesn't detail what we're trying to read, which can
lead to cryptic messages: "Cannot read Int". The patch changes this
function to annotate the error messages with the key/value we're trying
to convert, by using a new version of fromJResult....

Add more unittests

Instance, Node and Text modules have improved coverage.

Move two functions from hspace to Cluster.hs

This is done so we can test a longer pipeline.