Synnefo » snf-ganeti » ganeti-local

Design for remote command execution via RPC

This is a first design for executing commands via RPC.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add new constant for pending job status

This constant contains the job status' “queued”, “waiting” and
“cancelled”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

ensure-dirs: Fix permissions on master socket

A socket shouldn't have its executable bit set.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add a very simple test rpc program

This only supports test delay for now, is not built by default (only
on demand), and is also not installed anywhere.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add htools program code to hlint call

Currently only the libraries are added, but not the actual main binary
(usually short) code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

confd: fix RpcVersion call

By accident, we sent the node object as call data in this rpc call,
instead the version request (i.e. nothing). This is due to the fact
that the 'call' data comes second, not first in the function argument.

Signed-off-by: Iustin Pop <iustin@google.com>...

Merge branch 'stable-2.6' into devel-2.6

• stable-2.6:
  Update NEWS and bump version to 2.6.1

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

errors: Document arguments to QueryFilterParseError

Also fix one small mistake in the docstring for QuitGanetiException.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Makefile/check-local: Ensure design document exists

This way a missing design document is detected.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

QA: TestAssignNodesIncludingSplit can be disabled

The test can now be enabled/disabled with the "group-rwops" flag.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add option to disable iallocator use in QA

By default some QA tests use iallocators. It is now possible to disable
iallocators in QA, so that QA doesn't force having iallocators set up.
The affected tests have now alternatives pathways.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...

Text.hs: update field lists in parseData comments

The comments in parseData had become out of date with the implementations
of load{Group,Node,Inst}. This commit updates the field list in comments to
match the implementations.

Signed-off-by: Dato Simó <dato@google.com>...

LUClusterVerifyGroup: Localize virtual file paths

The check for file consistency didn't properly handle virtual paths
in case of a virtual cluster. This didn't cause any breakage as in
a standard virtual cluster setup with only one node all files are
visible for every node....

Cleanup/expand the filter/query tests

This patch cleans up duplicate code in Test.Ganeti.Query.Filter and
then adds a test for names consistency with Python's code behaviour
(stable ordering for simple filters and otherwise niceSort'ed
ordering).

Signed-off-by: Iustin Pop <iustin@google.com>...

Cleanup haddock documentation a bit

While grepping for htools imports in the non-htools subdirectory, I
saw that our haddock prologue and title are very very old and refer to
the old htools-only state. Let's cleanup a bit…

Signed-off-by: Iustin Pop <iustin@google.com>...

Use the new name filtering behaviour in query

We do this not quite generically, which means we have to add
another layer in the call chain, and rename the current query
function, plus add special-case code for each query type. Hopefully we
will be able to improve on this in the future....

Add function for getting a group's merged disk params

… and use it in the Query implementation, removing the last
non-correct query field for Groups.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Fix the node powered field

When initially implementing the node query, I thought the 'powered'
field is a representation of the run-time powered status, which would
make its query complex.

In reality, it's a simple config query, which we can support
easily. We also add a small helper, so that we don't hardcode the...

Add support for classic queries

This patch adds support for classic-style queries (before query2) to
the query socket server. The patch is rather trivial, since as in
Python we just piggy-back on the query2 implementation.

Signed-off-by: Iustin Pop <iustin@google.com>...

vcluster: Don't virtualize /etc/hosts path

/etc/hosts is a bit special as it's a system-wide file and the virtual
cluster/node root doesn't apply. The modification of /etc/hosts should
be disabled in virtual clusters. If it isn't, however, the vcluster
functions would raise an exception complaining about a path outside of...

qa_utils: Stop hardcoding path to /etc/hosts

- Simplified shell command for adding hosts, too
- The exception handling is also improved by re-raising any exceptions
instead of swallowing qa_error.Error

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Move constant for /etc/hosts to pathutils

Needed for coming patches.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Merge branch 'devel-2.6'

• devel-2.6:
  verify-disks: Explicitely state nothing has to be done
  Add list of design documents implemented in version 2.6
  Better list of replace-disks arguments + typos fixed
  jqueue: Look at archived jobs when watching
  Show old primary/secondary node on disk replacement...

gnt-job: List archived jobs if requested

If requested via a filter or by including the “archived” output,
archived jobs will be loaded and shown. This is significantly slower
than just listing normal jobs, therefore by default they are not loaded
at all....

jqueue: Correct docstring

The description was not accurate.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

query: Report data type for unary operators

All data kinds (used to restrict the data collected) referenced in a
filter can be requested once it's been “compiled”. However, the kinds
of fields used in boolean expressions (e.g. ["?", "xyz"]) were not
recorded. This patch changes the code accordingly and provides a unit...

verify-disks: Explicitely state nothing has to be done

Example output:
$ gnt-cluster verify-disks
Submitted jobs 4327
Waiting for job 4327 ...
No disks need to be activated.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fixup node disk free/total queries

The disk free/total values are optional ones, wrapped in a Maybe, so
we shouldn't directly serialise them. In order to simplify the
embedded extraction, we add a small helper function.

Signed-off-by: Iustin Pop <iustin@google.com>...

Fix pylint breakage due to unused var in gnt_cluster

The usage of that variable was removed in 45a36f36, but accidentally
the enumerate() was left in.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Make regex-pcre an optional dependency

This patch makes regex-pcre optional, allowing its disable via a
preprocessor define NO_REGEX_PCRE. This define will be added to
config.ac/Makefile.am in the next patch.

The patch also changes multi-line strings into string concatenation,...

Fixup node live field names

The String parameter to 'nodeLiveFieldExtract' is the query2 field
name, not the RPC-layer field name. Grrr for not having a real data
type for this.

Furthermore, we add some safety check that we don't return JSNull via
rsNormal…...

Add list of design documents implemented in version 2.6

Each version should have its dedicated list.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fix usage of errors.ResolverError

This exception is documented to have three arguments, but in one case
we raise it with a simple string argument. Let's fix that.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Typos and grammar fixes

Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

backend: Use utils.IsBelowDir instead of local code

utils.IsBelowDir is actually tested and doesn't allow writes to
“…/queue*”, like the old code here did.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fixed QA for replace-disks

Bug introduced in commit cc4b14f07: Part of a statement was missing. This
makes pylint-qa fail.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Improve the confd man page

This was a very very thin man page; this patch expands it a bit, but
it could use more help still.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Enable bash completion for Haskell daemons too

This requires a few tiny changes to build-bash-completion and to
Makefile rules, but is straightforward.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Support the new option types in build-bash-completion

Beside parsing the arguments as well, we also change the serialisation
format for choices/suggests, to make the Python code simpler.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

A few cleanups in Makefile.am

A lot of the lists in Makefile.am were not sorted properly (or at
all); let's sort them for more sanity.

Additionally, check-local used to spew this big shell block, even
though it does emit nice messages when failing, so we don't need to...

Fixup TAGS generation with newer GHC

Newer GHC refuses to allow "-O" with interactive mode, so let's filter
that out. Furthermore, sometimes you don't have a clean tree exactly
when you need to look up something/update the tags, so let's filter
out the "-Werror" too....

Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs

This is, I believe, the last non-htools specific file that still lived
in the htools directory; it's already widely used in non-htools code,
so let's move it before we add more functionality to this module....

Better list of replace-disks arguments + typos fixed

The man page and the bultin-in help for gnt-instance replace-disks were
inconsistent. Also fixed some typos in man pages.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Explicitly ask for the default iallocator in commands

Now "gnt-instance recreate-disks" uses the default iallocator when "." is
specified as the iallocator. For uniformity, the same behavior applies to
these commands:
gnt-node evacuate
gnt-instance migrate...

Instance QA uses default iallocator

QA for recreate-disk and replace-disks now run also with the default
iallocator.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

bdev: Add functions to verify file storage paths

- LoadAllowedFileStoragePaths: Loads a list of allowed file storage
paths from a file
- CheckFileStoragePath: Checks a path against the list of allowed paths

The unit test for “utils.IsBelowDir” is updated with cases which weren't...

jqueue: Look at archived jobs when watching

First: This enables the use of “gnt-job watch $id” for archived jobs.

Now, the reason for actually making this work is that during
sufficiently large group or node evacuations jobs are archived before
the client gets to poll for their output. This led to situations where...

backend: Check for shared storage also

If normal file storage was disabled but shared storage enabled,
“_TransformFileStorageDir” would still throw an exception.

in “opcodes._CheckStorageType” there's also a check, but I wasn't quite
sure what the correct way of handling it was, so I added a TODO comment....

Improve disk wipe unit test

- Don't hardcode node name in some places
- Don't define functions inside functions
- Simplify code for testing with and without offset, this is now in two
separate tests

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Add unit tests for cmdlib._WipeDisks

This is in preparation for adding disk wipe on growing disks.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>