Synnefo » snf-ganeti » ganeti-local

Fix IAllocator multi-evacuate message

Since Ganeti passes full host names (not common-suffix-stripped), we
need to remove the suffix from the evac_nodes keys too. In case one node
is not part of the cluster, it will lead to a wrong error message, but
for now it fixes the problem.

Implement IAllocator node evacuate request

This patch adds the new request loading/execution (trivial), but the
actual response formatting becomes more difficult as now the response
type differs by request.

Signed-off-by: Iustin Pop <iustin@google.com>

Change an internal type from Maybe to list

In preparation for multiple responses, we change from Maybe to List
(both used in the container sense).

This allows us to keep the same workflow for all kind of requests.

Signed-off-by: Iustin Pop <iustin@google.com>

Implement evacuation mode in hbal

This mode restricts the list of instances to be moved to the instances
living on the offline (and drained) nodes.

Signed-off-by: Iustin Pop <iustin@google.com>

Reorder options in CLI.hs

This should be no code change, just reordering of the options.

Signed-off-by: Iustin Pop <iustin@google.com>

Rewrite the node add checks for simpler layout

This will make it clearer than many if…then choices.

Split the balancing function in two parts

Currently in the balancing function we do two thing:

- take the decision where to do a new balancing round or not
- and actually computing the balancing round

This is not nice, as the two parts are conceptually separate, so this...

Switch the text file format to single-file

This patch changes from the two separate files to a single file, with
sections separated by a blank line. Currently only the node and instance
data is accepted, later the cluster tags will be read too via this
format....

Convert n1_score metric from % to count

This increases the priority of fixing N+1 failures compared to balancing
metrics.

Metric: count of primary instances/offline nodes

This helps with evacuation/failover of instances on 2-node clusters with
one one offline.

Use the oper_ram field if available

For the RAPI and LUXI backends, we can get the actual memory usage (if
instances are running) via the oper_ram, whereas backend/memory only
tell what the instance will use at the next boot.

Not using oper_ram means that the node model is flawed and we consider...

Fix typo breaking LUXI backend

This really shows the need for actual dist-time full testing (not
unittests).

Configure exclusion tags via the cluster tags

This patch adds reading of the exclusion tags from the cluster tags: any
tags starting with htools:iextags: will convert their suffix into an
exclusion tags prefix. In other words, "htools:iextags:service" will...

Read cluster tags in the LUXI backend

Introduce support for reading the cluster tags

While these are not actually populated from the backends, and all the
programs ignore them, this patch contains the changes in the function
types required.

Add a new node list field

This patch adds a new node list field (ptags), showing the primary
instance tags.

Use conflicting primaries count in cluster score

This small patch adds the number of conflicting primaries in the cluster
score. This is different from the other non-CV metrics where we usually
compute the percentage of failing instances (for that metric); but for a...

Collapse the statistical functions into one

This allows us to get rid of two duplicate list length computations,
with a minor speedup.

Add a tags attribute to instances

… and read it in all the loaders. hscan is modified to save it to the
files it generates.

The attribute is not yet used in any place.

Allow overriding the field list in -p

The print nodes option can now accept an optional field list to
customise the output. This is ugly, since the field names do not match
the header names, but it is at least barely customisable (at runtime).

Change the default dynamic usage to baseUtil

This fixed the unbalanced secondary instances on partially empty
clusters, and helps in general for the cases where real utilisation data
is not available.

Enhance the error reporting for Rapi and Luxi

Currently the JSON conversion in Rapi and Luxi are giving something
like:
Error: failed to load data. Details:
Unable to read Double

This doesn't tell one where the error is (in a node specification? and...

Make some CLI options more consistent

Both the simulate and the tiered allocation mode take a machine spec on
input via a comma-separated list. This patch makes this a little bit
more consistent (always use disk,ram,cpu in this order).

A small style change in Node.hs

This imports PeerMap as P and reindents some lines.

Convert option parsing to a monadic flow

This allows us to do verification of option arguments in the assignment
functions themselves.

Some cleanup of Loader.mergeData

This doesn't need to be a monadic function, let's make it a simpler one.

Expand the --print-instances output

This adds run status, resource parameters and load parameters for
instances.

Some small style fixes

Simplify Cluster.computeMoves

Since we now have an actual type for describing the instance moves
(IMove), it's simpler to convert this into the move description/move
commands, rather than re-computing the move based on initial and final
nodes. This makes the shell commands computation and over-Luxi command...

Generalise the node/instance listing

This patch introduces a generic formatTable function (based on, and
similar to the Ganeti one, but different and more FP in style) and
changes the node and instance listing to it.

The node list (due to the many variables) is still a little bit hackish...

Fix two haddoc/happy docstring issues

Add loading and processing of utilisation data

This patch adds loading and processing the utilisation data during
instance moves. While the data is not yet used, it is correctly modified
by instance changes between nodes.

hbal has the new ‘-U’ command line argument for this. The format of the...

Merge the Node.setPri and Node.addCpus functions

The latter is only used right after the former in the Loader module, and
we'll need more of this 'update not with the data of this instance'
functionality (which is different than addPri where all information must...

Show the load on nodes in node lists

The strange printf usage is due to some limitation (it seems) in ghc for
very long argument lists. The whole printout should be rewritten later.

Allow displaying the instance map in hbal

This is similar to --print-nodes, but with much fewer fields.

More hlint fixes

This makes (for now) the code hlint-clean. This is per se not a huge
gain, but it allows easier tracking of regressions in style later
(one-two new violations are easier to diagnose when not hidden among 20
“known” ones).

Style change: cluster CStats camel-casing

This is again the cs_x to csX name change.

Modify the internals of the detailed CV scores

Before we used a tuple; since we'll need more metrics in the future,
it's simpler to transform this into a list of doubles, whose elements
are handled homogeneously by all the code that needs them.

Change iMoveToJob to properly create migrates

The current Cluster.iMoveToJob always creates failovers, which is not
what we want. This simply used the original instances status to select
between these two (this is not optimal by the way, since the status...

Fix haddock issues with tuple members

It seems that haddock cannot document tuple members - but arguably, once
one needs to do that, tuples should not be used anymore.

This just moves the comments to the tuple comment.

Signed-off-by: Iustin Pop <iustin@google.com>...

Store the instance move in the MoveJobs

This will automatically sort our Ganeti jobs into the independent job
sets, and then we can submit them separately.

Add a function converting Placements into Jobs

This converts from htools-specific Placements into Ganeti standard
OpCodes, which will later allow execution via Luxi.

Split the Luxi generic parts from the loader

The Luxi loader implements both a generic Ganeti Luxi client and the
loader; it is better if these two are separated. The patch adds a
Ganeti/Luxi.hs (not under HTools!) since that is generic for Ganeti, and
not related necessarily to htools.

Change ExtLoader to only handle I/O errors

Due to the Control.Exception changes between 6.8 and 6.10, using it
portably is difficult. Since we're only interested in handling I/O
errors, we can use prelude's catch and not have to deal with
Control.Exception at all....

Brown bag fix: invert a test

During testing I used the test inversely to see it triggers correctly,
and committed by mistake the inverted test. Fixing it.

Add support for building without curl

Since curl is not always needed (e.g. when only using luxi or less
likely file backends only) and is also not always available, it is
useful for building without it. This of course disabled the RAPI
backend.

This patch changes ExtLoader to build with the ‘-cpp’ option which makes...

Fix luxi recvMsg for messages bigger than 4K

This patch fixes a logic bug in luxi that breaks receive of messages
bigger than 4096 bytes. The send message is not impacted as it uses a
different algorithm.

Test some cases for the cluster score computation

Implement support for 'cheap' moves only

This patch adds support for cheap (failover/migrate) operations only in
the balancing algorithm and in the hbal command line options.

This allows a very quick balancing (compared to allowing replace-disks)
which can be useful as a scheduled operation.

Use migrate or failover based on instance state

While we can't guarantee that the instance will be in the same state by
the time the migrate/failover command will be run, we can at least try
to do the right thing assuming no other changes to the cluster state....