Synnefo » snf-ganeti

Add loading and processing of utilisation data

This patch adds loading and processing the utilisation data during
instance moves. While the data is not yet used, it is correctly modified
by instance changes between nodes.

hbal has the new ‘-U’ command line argument for this. The format of the...

Merge the Node.setPri and Node.addCpus functions

The latter is only used right after the former in the Loader module, and
we'll need more of this 'update not with the data of this instance'
functionality (which is different than addPri where all information must...

Show the load on nodes in node lists

The strange printf usage is due to some limitation (it seems) in ghc for
very long argument lists. The whole printout should be rewritten later.

Allow displaying the instance map in hbal

This is similar to --print-nodes, but with much fewer fields.

More hlint fixes

This makes (for now) the code hlint-clean. This is per se not a huge
gain, but it allows easier tracking of regressions in style later
(one-two new violations are easier to diagnose when not hidden among 20
“known” ones).

Style change: cluster CStats camel-casing

This is again the cs_x to csX name change.

Modify the internals of the detailed CV scores

Before we used a tuple; since we'll need more metrics in the future,
it's simpler to transform this into a list of doubles, whose elements
are handled homogeneously by all the code that needs them.

Change iMoveToJob to properly create migrates

The current Cluster.iMoveToJob always creates failovers, which is not
what we want. This simply used the original instances status to select
between these two (this is not optimal by the way, since the status...

Fix haddock issues with tuple members

It seems that haddock cannot document tuple members - but arguably, once
one needs to do that, tuples should not be used anymore.

This just moves the comments to the tuple comment.

Signed-off-by: Iustin Pop <iustin@google.com>...

Store the instance move in the MoveJobs

This will automatically sort our Ganeti jobs into the independent job
sets, and then we can submit them separately.

Add a function converting Placements into Jobs

This converts from htools-specific Placements into Ganeti standard
OpCodes, which will later allow execution via Luxi.

Split the Luxi generic parts from the loader

The Luxi loader implements both a generic Ganeti Luxi client and the
loader; it is better if these two are separated. The patch adds a
Ganeti/Luxi.hs (not under HTools!) since that is generic for Ganeti, and
not related necessarily to htools.

Change ExtLoader to only handle I/O errors

Due to the Control.Exception changes between 6.8 and 6.10, using it
portably is difficult. Since we're only interested in handling I/O
errors, we can use prelude's catch and not have to deal with
Control.Exception at all....

Turn on, and fix, more warnings

The Makefile was intented to be -Wall and not simply -W, but I missed
that. This enables more warnings and also enables -Werror (except for
the tests).

Add support for building without curl

Since curl is not always needed (e.g. when only using luxi or less
likely file backends only) and is also not always available, it is
useful for building without it. This of course disabled the RAPI
backend.

This patch changes ExtLoader to build with the ‘-cpp’ option which makes...

Fix luxi recvMsg for messages bigger than 4K

This patch fixes a logic bug in luxi that breaks receive of messages
bigger than 4096 bytes. The send message is not impacted as it uses a
different algorithm.

Add some more instance tests

This include instance text load tests.

Implement support for 'cheap' moves only

This patch adds support for cheap (failover/migrate) operations only in
the balancing algorithm and in the hbal command line options.

This allows a very quick balancing (compared to allowing replace-disks)
which can be useful as a scheduled operation.

Use migrate or failover based on instance state

While we can't guarantee that the instance will be in the same state by
the time the migrate/failover command will be run, we can at least try
to do the right thing assuming no other changes to the cluster state....

Add a simulated cluster data loader

This is useful especially for hspace, where we might want to simulate a
hypothetical cluster to check allocation beforehand.

Unify the command line options and structures

This patch moves all the command line options and their internal
representation into CLI.hs. This means that duplicated options between
any two binaries are no longer declared twice, and that we no longer
need the two *Option classes.

CLI: Prevent incompatible options to be selected

This patch makes CLI abort if more than one backend is selected.

Initial commit of the luxi backend

This patch adds a luxi backend that allows direct query of the master
daemon on the local node. This patch doesn't enable the backend to be
used.

There are a couple of things still missing in the implementation:
- we don't have a master timeout in reads and writes, only a...

Fix a haddoc issue

Change the tryAlloc/tryReloc workflow

Currently, the tryAlloc and tryReloc function return a list with all the
results, both failures and successes. This is fine for hail, which does
one round of allocations, but is not so good for hspace, which does
iterative rounds; since at each (successful) step we only take the best...

Slight change to the internal allocation results

Currently the Cluster.AllocSolution type is defined as a list of
‘(OpResult Node.list, …)’ and the results for applyMove are defined as
‘(OpResult Node.List, …)’. Both these means that the failure/success
indication is hidden in the first elements of this tuple, which makes is...

hspace: move instance count and score into CStats

Currently the instance count and cluster score are separated from the
other initial/final phase stats, even though they are very similar. This
patch moves computation of these two into totalResources/CStats and...

Export more stats in hspace

This patch changes Cluster.totalResources to compute more resources and
prints them in hspace.

Fix score calculation to work with empty clusters

Currently the cluster score calculation includes an offline instance
percentage, expressed as “offline inst / (offline + online inst)”, which
results in NaN for empty clusters. This patch changes the calculation...

Take the foldl out of Loader.fixNodes

Currently Loader.fixNodes is foldl' with a complicated function. It
makes more sense to take foldl' out of this function (and put it into
the caller) and let fixNodes be only this internal function.

Fix hlint-generated warnings

This big patch cleans up the code per hlint indications. Many removals
of extra parentheses, replacements of concat . map with concabtMap,
extra dollar signs, eta reductions, etc. were performed.

The code still compiles and passes a couple of manual tests on sample...

Return correct failure data from Node.add*

This patch alters the Node.addPri/addSec to return correct failure data.
It removes the computeFailN1 function from the module as that used to
combine both mem and disk checks in the same function and thus the real...

Remove hn1 and related code

hn1 was deprecated for a while and this patch removes it altogether. The
support code in Cluster.hs is also removed.

Add an availDisk node function

This function returns the amount of available disk, which depends on
whether a low disk limit has been configured or not and on the free disk
space of the node.

Add a new type for cluster statistics

Currently totalResources returns a 5-tuple of integers. This is not easy
to handle, as each change on the return type means that each caller must
be updated.

This patch adds a new type for cluster stats and uses that instead as...

Fix a haddock/docstring issue

Add two new node attributes

Two new min disk free ratio and max cpu usage attributes are added to the
nodes. These will be used in the future to restrict allocation.

Fix the various monomorphism warning

In a few places (e.g. tryRead or any printf call) it's a little bit hard
to add the correct type signatures, but in the it is possible to fix
these warnings (which can bite one in subtle cases).

Add cpu ratio to cluster calculation

Add cpu-count-related attributes to nodes

This patch adds cpu-count related attributes to nodes:
- total cpus
- cpus in use
- ratio of virtual:physical cpus

We also set correctly the cpu values at load time, but we don't do
anything yet while moving instances around. The cpu ratio is shown in...

Fix reading of total disk space in iallocator

IAllocator currently uses a wrong key name for reading the total disk
space (‘disk_usage’ which was copied from RAPI, but the actual
iallocator key is ‘disk_space_total’).

This patch fixes that and also makes iallocator always use this key,...

Rework the tryAlloc/tryReloc functions

Currently tryAlloc/tryReloc do not return the new instance, as this is
not needed for IAllocator alloc/reloc requests. However, for computing
the space, the new instance is useful, so we modify these functions to
return this information too....

Small doc change

And an alignment issue.

Add copyright/license information

This doc-patch adds copyright and license information to (hopefully) all
needed files.

Small whitespace change

Move the RqType and Request types to Loader.hs

These two will be more generic than now, and belong somewhere else -
Loader.hs is a generic module for data loading, thus we move them there.

Lots of documentation updates

This patch does only doc build changes, doc changes and function move
around (for more logical documentation). It should have no impact at all
on the code.

Add some very trivial Instance tests

This is more of an exercise in QuickCheck than strong testing.

Add test infrastructure and initial tests

This patch adds a QuickCheck-based test infrastructure and initial tests
based on it. The PeerMap module has a 100% coverage ☺

Side-note: one has to read the source of QuickCheck to see how to use it
(especially the Batch submodule), the docs are not enough…

Remove unused parameters from PeerMap creation

We remove some unused arguments (added way back for compatibility with
Arrays, which we didn't use in the end). This makes the code clearer
(and doesn't need the Ndx type to be an instance of Num).