Synnefo » snf-ganeti » ganeti-local

A few unittests improvements

Small simplifications of other unit tests using the (==?)
operator when possible, and typo fixes.

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Remove custom OpResult type/monad

Since we now have the GeneralResult as a multi-purpose monad, we can
remove the custom OpResult monad, and just use 'GeneralResult
FailMode' as our type. This allows removal of a few bits of
specialised infrastructure, relying instead on the generic one....

Generalise the Result type

Currently, our error monad—Result—has a plain string error type. This
is not good, as we don't have structured errors, we can't pass back
proper error information to Python code, etc.

To solve this, we generalise this type as 'GenericResult a', and make...

Group.hs: add 'allTags'; adjust loaders and test data for it

This commit adds a Group.allTags field to store the tags of node groups,
and teaches each loader backend in HTools to populate it (additionally, the
IAllocator class in lib/cmdlib.py now includes tags for groups too). Test...

Merge branch 'devel-2.6'

• devel-2.6:
  ensure-dirs: Fix permissions on master socket
  Update security document for version 2.6
  Update NEWS and bump version to 2.6.1
  Text.hs: update field lists in parseData comments

Conflicts:
NEWS: Trivial
lib/tools/ensure_dirs.py: constant moved to pathutils...

confd: fix RpcVersion call

By accident, we sent the node object as call data in this rpc call,
instead the version request (i.e. nothing). This is due to the fact
that the 'call' data comes second, not first in the function argument.

Signed-off-by: Iustin Pop <iustin@google.com>...

Text.hs: update field lists in parseData comments

The comments in parseData had become out of date with the implementations
of load{Group,Node,Inst}. This commit updates the field list in comments to
match the implementations.

Signed-off-by: Dato Simó <dato@google.com>...

Cleanup network timeouts and htools imports

This patch removes the last HTools module imports from non-htools code
(the HTools.Types module), but it requires an associated cleanup:
using luxi-specific constants for luxi timeouts (the only effect is
that one timeout decreases from 15 to 10, the default value in the...

Add functionality for checking validity of names

This replicates in the Haskell Query2 implementation the behaviour of
the Python code: if a "simple" filter is passed (one that contains
only Or aggregators and EQ binary ops on the name field), then an
failure is flagged if the given values are not known....

Switch ordering of names on query to niceSort

This makes the "all" names queries consistent with the Python
results. The change requires updating the unittests, at which point a
duplicate error message is simplified.

Signed-off-by: Iustin Pop <iustin@google.com>...

Simplify a bit queryFields

We don't add a type class for fully-generic handling, but we do
abstract the duplicate part.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Add a makeSimpleFilter function

And associated unittests. This will be needed for classic-style
queries.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Fixup node disk free/total queries

The disk free/total values are optional ones, wrapped in a Maybe, so
we shouldn't directly serialise them. In order to simplify the
embedded extraction, we add a small helper function.

Signed-off-by: Iustin Pop <iustin@google.com>...

Make regex-pcre an optional dependency

This patch makes regex-pcre optional, allowing its disable via a
preprocessor define NO_REGEX_PCRE. This define will be added to
config.ac/Makefile.am in the next patch.

The patch also changes multi-line strings into string concatenation,...

Add an Utils.NiceSort() equivalent

This patch adds a NiceSort equivalent and the corresponding unittest
(partially copied from Python unittest). The difference between the
Python version and this one is that this implementation doesn't use
regular expressions, and as such it doesn't have the 8-groups...

Define the actual arguments that are supported by programs

This defines the arguments supported and then modifies the
--help-completion output to include them too.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add support for arguments in --help-completion

This extends the current option-only support with support for
arguments too. They are serialised without a without the prefix (so
just with the type) and the min/max ranges (max="none" for unlimited).

A few new option types have been added as well....

Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs

This is, I believe, the last non-htools specific file that still lived
in the htools directory; it's already widely used in non-htools code,
so let's move it before we add more functionality to this module....

Remove explicit recursion from sanitizeDictResults

Since ERpcError is still not a monad, a simple foldr will have to
do for now.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Merge parseHttpResponse with rpcResultParse

parseHttpResponse is such a simple function that we can actually
do the entire unpacking in rpcResultParse.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Remove monadic results wherever possible

There is no need (at least right now) for some of the resulting
types to be in additional monad, so let us just have more
"normal" functions.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Rename RpcError -> String function

As per Iustin's suggestion, show is not a good solution - so instead
the name is now explainRpcError.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Change RpcResult typeclass

For storage_list call, the result type depends on the call parameters.
Therefore, we have to add call as an argument for rpcResultFill - and
by extension, to the typeclass.

Signed-off-by: Agata Murawska <agatamurawska@google.com>...

Node query now uses live data

Added support for NodeInfo RPC call as source of data for node query.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add live parameter to query

The tests we currently have assume, that all the data required for
running the query is available - once we add live data, this will no
longer be the case.

This patch adds boolean parameter to query function, which tells it
whether to ignore live parameters gathering....

Json helper for node query

This is a helper function that gets the JSON encoding of the first
element of the list (if possible).

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

NodeGroup query in Haskell

Implementation of nodegroup queries in Haskell. This is not yet
complete as we are missing merged disk parameters and option
want_diskparams is not implemented.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fix RPC call and result (de)serialization

Previous version of RPC calls implementation in Haskell did not take
into account that the actual result type for queries is a list, not
a dictionary.

This patch aims at fixing the problem "for now" - it is not a pretty...

Cleanup for RPC errors

Curl error messages are self contained and don't need "error" prefix;
added RpcResult Error which will be used once proper deserialization
of rpc response is in place.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Correct top comment in Query/Server

I assumed this was a copy-paste+forgetting to change the header comment.

Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Remove generic options from individual programs

Currently, we test and require that each individual program (hbal,
etc.) defines/supports the generic options (currently --help and
--version). Even with the test, this is not optimal, since it requires
changes in many places whenever we modify the list of generic options,...

Annotate options with completion information

This is a quite boring patch, just adding annotation information to
all existing options. Some of the annotations are not very good; but
we don't have support for more precise completion in
build-bash-completion, so this is good enough....

Add printing of group score in hinfo

The global cluster score is less interesting than individual group
scores, for multi-group allocation purposes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Use autoconf-based paths from Haskell instead of constants

Future changes will change Path.hs to use an environment variable.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Merge branch 'devel-2.6' into submit

• devel-2.6:
  Fix bug in non-mirrored instance allocation
  Fix gnt-debug iallocator

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Fix deserialisation bug in ResultEntry

Found via the newly added unit-tests, which test most of the
serialisation code in Query/Language (except for QueryResult, for
which we already tests both sub-components separately).

Signed-off-by: Iustin Pop <iustin@google.com>...

Further hlint fixes

Commit 2cdaf22, “Re-enable standard hlint warnings”, got it almost
right. The only problem is that (confusingly) the default set of hints
is not in HLint.Default, but in HLint.HLint (it includes Default and
some built-ins).

After changing the lint file to correctly include the defaults, we had...

Add some unittests for node queries

These new tests check that:

- no known fields return unknown
- any unknown field returns unknown
- the type of the fields is consistent between the getters and the
field definition
- the length of each result row corresponds with the number of fields...

Move Version.hs up from under HTools/

This is another module that is generic, and not htools-specific.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Split last two remaining tests from QC.hs

This makes QC.hs obsolete, but we will remove it in a later patch,
once we automatically build a file with all the imports (to ensure
correct coverage results).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Split most HTools test code into separate files

Except for Ganeti.HTools.JSON, which needs rename, we split all the
other test suites into separate files.

We have to add another common test helper, due to import dependencies
(sigh), but otherwise this split is pretty straightforward....

Split Rpc tests from QC

This required lots of other code moves, so I created it as a
standalone patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Split out Objects.hs from QC.hs

This is the first file split out from QC.hs - an easy one, since it
has just one test.

The patch changes the way we build hpc excludes, since now we'll have
many modules that need to be excluded, and hpc doesn't seem to be able...

Implement compilation of regexes at creation time

This means that the verification of the correctness of the regex is
done once, at the deserialisation/creation time, as in the Python
code. To do this, we have to change the FilterRegex type from an alias...

Add filtering support in Query

This adds basic infrastructure for filtering (fully functional except,
as usual, for runtime data), and then uses it for node queries.

Since the filtering exports regex matching as an external
functionality, we have to use a regex library. There are many flavours...

Parameterize the Filter type

In preparation for introducing filtering functionality, we convert the
'Filter' type from a '*' kind to a '* -> *' kind.

This allows us to define some general properties for the filter, and
for example introduce later an easy filter compilation, etc....

Putting the multiallocate pieces together

This is the final part:

• Parsing the new request type
• Feed it to allocList
• Format the result

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add missing luxi query 'QueryFields'

This was missed; we add the definition and the de-serialisation
support.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

One more SADT type - VType

This adds the VTYPE_* conversion to VType, used for QFT equivalence
later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Add more node-related data types and functions

This is a simple type declaration for NodeRole, a NdParamObject type
class, and a few related helper functions for nodes and node groups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>