Synnefo » snf-ganeti » ganeti-local

gnt-network: Fix CLI issues with node groups

The original implementation reserved the name “all” to specify all node
groups. That would prevent the use of a node group actually named “all”.

This patch changes the order and definition of the command line...

Improve network query algorithm

Similar to commit 572a277. Directly iterate over “self.wanted” instead
of iterating through all networks and checking if they're wanted.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Format network field "group_list" in client

The field is now a list of tuples containing the group name, mode and
link.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add function for generating UUIDs in the Haskell codebase

Its first use will be to generate the salt for ConfD requests of the Haskell
client, as in the Python client.

Unit test added as well.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add utility to format dictionary as key=value strings

This will be used in QA to format network interface parameters.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Adjust LUInstanceSetParams wrt the `ext' template

Initial support for the `ext' template in gnt-instance modify.
Take care of all needed checks in CheckArguments and CheckPrereqs
wrt ExtStorage Providers and ext-params.

ExtStorage disk addition and specification of arbitrary ext-params...

Stop all daemons on cluster initialization

Addresses issue 145. When daemons are still running from a previous
cluster instance, starting the daemons again will fail.
“ConfigWriter.IsCluster” only looks at the configuration file, which
could already have been deleted....

More style updates for network-related code

Error message formatting, else/if merging to elif, docstrings.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

Detect wrong Python module structure

If, for one reason or another, a module named "ganeti.ganeti" can be
succesfully imported (e.g. through a symlink in the wrong place), error
which are diffcult to debug can occur. This piece of code tests for the
unlikely case of that happening....

Fix network LUs use of error code in OpPrereqError

Currently, most network LUs simply use EINVAL instead of correct error
codes; this patch tries to adjust the obvious cases.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

cmdlib: Factorize network conflict detection

LUNetworkConnect and LUNetworkDisconnect had very similar code to detect
conflicts between instance's network interfaces and networks. This code
factorizes the common part and does some cleanup:

- Remove single-letter variable for lambda (“l”)...

gnt-network add: "--network" is required

Also do some minor code re-formatting.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

query: Factorize code for getting statistics value

This was not only copied for the networking fields in commit 306bed0e,
but commit cfcea7ef fixed wrongly ordered parameters and didn't fix the
original. Either way, this patch merges the two cases again. The newly...

OpNetworkConnect: Check for network mode

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Replace frozenset with compat.UniqueFrozenset

This is not a trivial s/frozenset/compat.UniqueFrozenset/, but rather
only replaces “frozenset” where appropriate. Most of the places are
“static” information that doesn't change after the module has been
loaded....

Remove some unused Python code

This patch removes code which is no longer used due to refactoring:

- http.InitSsl, last usage removed in commit 33231500 (“Convert RPC
client to PycURL”)
- rapi.baserlib.MakeParamsDict, last usage remove in commit 4e5a68f8...

Allow shutting down offline instance

If an instance is offline we definitely shouldn't start it up.
But shutting it down, should it be up by mistake is not "that" bad.
Still, we only allow it with --force, as it still performs an action on
an instance we shouldn't touch. This should make everybody happy....

Fixes and improvements to comments

Some fixes, added more information in a few points, removed a stale (5+
year old) TODO comment.

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Make LUNetworkAdd pass _VerifyLocks()

LEVEL_NODE_ALLOC should be aquired too if LEVEL_NODE is ALL_SET.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

hv_kvm: List of SPICE parameters should be module-global

This list is static at runtime and doesn't need to be recreated every
time.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

LUOobCommand: Always declare NAL in shared mode

Doing so avoids triggering an assertion in mcpu which cross-checks the
node lock and node allocation lock acquisition mode.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add optional formatting for OP_DSC_FIELD

For some opcodes, the output is not "stable", and depends on the exact
input values; this makes it harder to check consistency against
Haskell code.

To compensate for this, we add a way to override the formatting of the...

Very very very basic openvswitch support

This is a "better than nothing" support, just for kvm and just joining
the machine to the opevswitch bridge with the right command.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>...

Read watcher pause using RPC, not directly

The master daemon should not directly read files written by the node
daemon. This patch adds a new RPC to read the watcher pause file and
changes the master code to use it.

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Add RPC for setting watcher pause

The watcher pause file should be set/unset on all nodes at once, not
only the master node. For that a new RPC is needed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Improve test for tools.ensure_dirs

- Add more checks, some of them are deliberately redundant
- Descriptive error messages
- Add comment describing order to “tools.ensure_dirs”
- Avoid copying a list in an assertion in “tools.ensure_dirs”

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Remove checks wrt IDISK_PARAMS from OpCode level

Change the "--disks" option validation, to just check the format
of the dict and do not check whether the keys are included in the
IDISK_PARAMS constant at OpCode level. This allows the passing of
arbitrary parameters at the CLI, which will then be logically...

Fix ordering of entries in tools.ensure_dirs

Commit ebd437a added two new entries to tools.ensure_dirs, but did so in
the wrong order. Patch forthcoming to improve the unittest's error
message.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

lib/tools/ensure_dirs.py: Code formatting

Wrap lines in a consistent manner (uid/gid on the second line) if
wrapping is necessary at all. “git diff --color-words” shows no
difference at all as only whitespace changed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Add utility function to create frozenset with unique values

When used instead of a plain call to “frozenset”, this would have
avoided the issue fixed in commit e2dd6ec. The new function is located
in the “compat” module as it will be used at module load time in most...

Change value for ECODE_TEMP_NORES

Unfortunately there was a bug in commit 518a45e whereby ECODE_TEMP_NORES
had the same value as ECODE_NORES, leading to failures in a Haskell
test. Of course this would also have affected other users of the constant.

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Improve error message for when adding inotify watcher fails

Explicitely mention the fs.inotify.max_user_watches sysctl value.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Export error codes from RAPI client module

Until now the error codes were not available from the RAPI client
module. A newly added unit test ensures all error codes are contained in
“ECODE_ALL”, as well as ensuring consistency between the RAPI client and...

cmdlib: Opportunistic locking on instance creation

Adds a new parameter to “OpInstanceCreate” and “OpInstanceMultiAlloc” to
use opportunistic locks.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

mcpu: Verify node allocation lock mode

Add verification code to mcpu to check an LU's locks. Two whitelists are
provided to exclude LUs from the two tests.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

LUBackupExport: Use node allocation lock

Block instance allocations when all node locks will be acquired.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

LUInstanceMultiAlloc: Use node allocation lock

Avoid conflicts between instance allocations.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

LUNodeSetParams: Use node allocation lock

Block instance allocations when all node locks will be acquired.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

LUOobCommand: Use node allocation lock

If no node names are given, all node locks are acquired.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add tool to configure node daemon

The design for this is in “doc/design-node-add.rst”. The tool receives a
JSON data structure on stdin and configures the node's daemon after
verifying the received values.

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Locking related fixes for networks

Use GetNetwork() only when having already aquired the lock,
i.e. in CheckPrereq().

In LUNetworkConnect/Disconnect do not include Network info in Hooks
environment, so that network locking can be avoided if conflicts
are not checked....

Use new util function for mac_prefix validation

Use new NormalizeAndValidateThreeOctetMacPrefix() util function in
LUNetworkAdd/LUNetworkSetParams to validate network's MAC prefix.
Additionally, move the check in CheckArguments() in the case of
LUNetworkAdd....

LUClusterRepairDiskSizes: Use node allocation lock

This opcode acquires all node resource locks, which conflicts with
instance allocations.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

LUClusterVerifyGroup: Use node allocation lock

See comment in code.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

LUInstanceChangeGroup: Acquire node allocation lock

Changing instances' groups shouldn't conflict with instance allocations.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>

iallocator: Add node whitelist

In the future instance creations might have a lock on all nodes as was
the case until the implementation of opportunistic locking. Nodes for
which the lock is not held will be shown to the iallocator plugin as if
they were marked offline....

errors: Show error descriptions in API documentation

Comments with a colon after the hash sign (“#:”) show up in the
epydoc output.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add safety check on job dependency/TIsLength

If TIsLength is applied to a non-container item, it will fail (type
error) due to invalid application of len(). Since this can happen on
user-supplied data, we add an explicit TList/TTuple check (the TTuple
test is a new one)....

LUInstanceCreate: Acquire node allocation lock

Opportunistic locks are not yet used. This patch changes
LUInstanceCreate to acquire the node allocation lock to avoid conflicts
with other opcodes acquiring many node locks.

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Use GetMultiInstanceInfo in LUNetwork* opcodes

LUNetworkConnect/Disconnect looks up a nodegroup's instances for
conflicting IPs. To do so, use GetNodeGroupInstances() and
GetMulitInstanceInfo().

Additionally, check if the correct locks were acquired.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...