Synnefo » snf-ganeti

IAlloc: read group uuid from the input message

This makes the code incompatible with JSON files from Ganeti pre-2.4.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Luxi: read the node uuid from the cluster

This makes the code incompatible with Ganeti pre-2.4.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Utils: add a default UUID

This will be used as a placeholder for the cases when we need a UUID
(any UUID), but we don't have one handy.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

Improve the standard deviation computation

This does just two passes, instead of three, over the list. This reduces
the overall runtime well enough (~25%) in some tests, but it's not
reproducible using profiling, so I don't know how much the function
itself is being sped-up....

Simu loader: move the loading to non-IO code

While we don't actually have IO code in the Simu loader, we do have the
same interface. So we move the code again to a separate parseData
function which is exported.

Rapi loader: split parsing from loading

The change is similar to the text loader change.

Ignore nodes which are not vm_capable

This break compatibility with Ganeti pre-2.3.

Fix tag exclusion weight

Currently, the tag exclusion metric has a weight of one, which means
there might be cases where we won't move instances around because it
upsets the cluster metrics. However, we do want to make a higher effort
for cleaning up tag collisions, so we increase the weight to an...

Move from hand-written man pages to RST/pandoc

This simplifies the maintenance of the man pages, and unifies the rst-to-*
converter to pandoc.

Update NEWS file for 0.2.7 release

Add a hack for normalized CPU values in hspace

Currently, the key metrics/tiered spec computations show the virtual cpu
count. However, since we do have a maximum ration Vcpu/Pcpu, we can also
show the “normalized” cpu count, i.e. the equivalent physical cpu count...

hbal: implement user-friendly termination requests

Currently, hbal will abort immediately when requested (^C, or SIGINT,
etc.). This is not nice, since then the already started jobs need to be
tracked manually.

This patch adds a signal handler for SIGINT and SIGTERM, which will, the...

Use the mingain options in the balancing algorithm

Also adds them in hbal.

Makefile: make the rst2html converter more strict

This will make the automated builds flag any problems.

Fix ReplaceSecondary moves for offline nodes

The addition of a new secondary on a node is doing two memory tests:
- in strict mode, reject if we get into N+1 failure
- reject if the new instance memory is greater than the free memory (not
available memory) on the node...

Update man pages for the new -S option

Implement cluster state saving in hspace

This also uncovered a few issues with the allocation model (instances
not being marked up, etc.).

Compared to hbal, hspace will generate either one or two files (for both
the standard and the tiered allocation mode), depending on the input...

Implement cluster state saving in hbal

Also move the LUXI execution (-X) to the end, after all the output
messages are printed. No good in waiting for the messages for a long
while, especially as they are not up-to-date stats after the job
execution, just an estimation of what the state will be.

Add a new option --save-cluster

This option will in the future be used to serialize the cluster state in
hbal and hspace after the rebalance/allocation steps.

Switch unittest to custom hostnames

Currently, the hostnames are almost fully arbitrary chars, which breaks
the assumption that nodes/instances will be normal DNS hostnames.

This patch adds some custom generators for these hostnames, that will
allow better testing of text loader serialization/deserialization.

Fix a couple of typos in the manpages

Again, thanks to lintian.

Update NEWS file for the 0.2.6 release

hscan: return exit code 2 for RAPI failures

If some clusters failed during RAPI collection, exit with exit code 2 so
that tests can detect this failure.

Fix another haddock issue

Extend the live-test

The (recently-enabled) live test coverage stats found a few low-hanging
fruits in the tests we do…

Preliminary support for coverage during live-test

While this doesn't work correctly yet (hpc sum seems to only take common
modules, not the sum of modules?), it prepares for gathering coverage
data during live-test (as an alternative to unittest coverage data).

Change the meaning of the N+1 fail metric

Currently, this metric tracks the nodes failing the N+1 check. While
this helps (in some cases) to evacuate such nodes, it's not a good
metric since rarely it will change during a step (only at the last
instance moving away). Therefore we replace it with the count of...

Allow balancing moves to introduce N+1 errors

This patch switches the applyMove function to the extended versions of
Node.addPri and addSec, and passes the override flag based on the state
of the node that we're moving away from.

Remove obsolete Container.maxNameLen

This was only used in one place (hbal), and is obsolete by the change to
the dual name/alias structure.

Remove an obsolete function

printSolution is no longer used, as we print the solution iteratively
now.

Update the node list fields

This patch renames the pri/sec to pcnt/scnt, and adds the real primary
and secondary instance lists, the peermap and the index of a node as
selectable options.

Fix handling of offline options and short names

This needs to be abstracted in a separate function, but in the meantime
we fix the issue in both places.

Signed-off-by: Iustin Pop <iustin@google.com>

Remove JOB_STATUS_GONE and add unittests

… for the serialization/deserialization of the job and opcode status.

Job status 'gone' was not actually used. It can be reintroduced if
needed.

Rename the job status constants

The rename is done such that we match Ganeti's own constants.

Complete the client Luxi implementation

All current Luxi calls are supported after this patch. A bug in
ArchiveJob is also fixed (Ganeti's job IDs are strings).

Fix some lint errors in the unit tests

Fix a warning in Loader tests

Incomplete pattern match…

Modify the test runner to show test exceptions

QuickCheck's batch driver (at least v1) doesn't show the test aborts,
but simply discards the specific exception and increases the abort
count. This makes it hard to debug the tests, so we modify our own test...

Improve the test driver

The tests are moved to a separate data structure, and we can select a
subset of tests to run.

Introduce suport for optional keys in JObjects

Some keys are optional in the Ganeti opcodes (e.g. ‘node’ in the
OpReplaceDisks), and as such we need to transform them in a Maybe value,
instead of failing.

The patch reworks a bit fromObj and adds maybeFromObj which parses such...

Add annotations to loadJSArray

This allows, for example, the RAPI backend to detail which information
(instance or node data) fails to parse.

A few more small Node unit-tests

Add more unit tests for allocation/balance

The patch adds some simple unit-tests for both the allocation function
(we can allocate small instances on an empty cluster, we can allocate in
tiered more starting from any size) and the balancing functions (one...

Make CStats instance of show

This helps debugging via ghci.

Another haddoc fix…

Stop modifying names for internal computations

Currently the name used internally is modified and holds the shortened
name of the nodes/instances. This has caused issues before, since we
always have to strip the suffix from input data and reapply it if we...

Change some test constants

First, we reduce the max size of the disks, since Int on 32bits will
overflow for big simulated clusters. This is a real issue, that will
need fixing in real life, but for now we just "silence" this test.

Second, we increase the amount of time a test is allowed to run,...

Add more unit tests

This increases the overall coverage by 5%-10% (depending on coverage
type). Some modules are still not unittested at all, as HUnit is a
better choice for them.

Remove the noLimit values and always use limits

This patch moves from allowing no-limits for disk/cpu ratios, and always
use a real limit. For disk, it's simple since we use 0, which means no
reservations for disks. For CPU, we set an (arbitrary) limit of 64 v/p,...

Fix hspace's KM metrics

We returned the KM_POOL_* metrics as the final state, not as the delta
between the final and the initial state.