Synnefo » snf-ganeti » ganeti-local

burnin: implement basic confd testing

Just a few queries are checked, but this should give us confidence that
at least the basic confd framework is working properly.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Rightname confd's HMAC key

Currently, the ganeti-confd's HMAC key is called “cluster HMAC key” or
simply “HMAC key” everywhere. With the implementation of inter-cluster
instance moves, another HMAC key will be introduced for signing critical
data. They can not be the same, so this patch clarifies the purpose of the...

Switch burnin to cli.JobExecutor

Burnin has a custom job executor, because of its need to retry some job
series.

While we cannot replace all of it, at least the execution we can switch
to cli.JobExecutor, to take advantage of the recently-introduced
out-of-order waiting....

Switch from os.path.join to utils.PathJoin

This passes a full burnin with lots of instances, and should be safe as
we mostly to join a known root (various constants) to a run-time
variable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Adding tool for automated cluster-merger

This is the implementation of docs/design-cluster-merger.rst. It allows
the automatic merging of one or more clusters into the invoking cluster.

While this version is tested and working it still needs some tweaking...

Fix dumpers/loaders after slots cleanup

Commit 154b958 changed (correctly) the slots usage, but this broke
dumpers/loaders since we relied directly on the own class slots
field.

To compensate, we introduce a simple function for computing the slots...

Fix unused imports or add silences where needed

In some cases pylint doesn't parse the import correctly, so we add
silences; but there are also many cases of unused imports, which we
simply remove.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Convert to static methods (where appropriate)

Many methods are simple pure functions, and not depending on the object
state. We convert these to staticmethods.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Add targeted pylint disables

This patch should have only:

- pylint disables
- docstring changes
- whitespace changes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Fix use of the logging functions

The logging functions expand the arguments themselves, thus it's safer
to let them do it rather than manual string formatting.

Also re-wraps one comment.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Remove quotes from CommaJoin and convert to it

This patch removes the quotes from CommaJoin and converts most of the
callers (that I could find) to it. Since CommaJoin does str(i) for i in
param, we can remove these, thus simplifying slightly a few calls....

Make bootstrap._GenerateSelfSignedSslCert public

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

burnin: skip instance moves on single node

If we have only one node, instance moves fail, because it tries to move
the instance to itself. Skipping the operation, because in that case it
doesn't make sense.

Signed-off-by: Guido Trotter <ultrotter@google.com>...

Merge remote branch 'origin/master' into mogu

• origin/master:
  Fix burnin's verbose mode
  Final NEWS update and version increase for 2.0.4
  Encode the actual exception raised by LU execution
  Move the luxi error handling into errors.py
  Fix the confusing ssh/hostname message in node add...

Unify the multiple 'verbose' options

Currently multiple files define a 'verbose' option. This patch moves all
these definitions to a single VERBOSE_OPT option which is exported from
cli.py.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add burnin support for instance moves

This patch adds support for instance moves in burnin. This means that
non-drbd instances finally get support for being moved.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add more auto-completion metadata

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Merge branch 'next' into branch-2.1

• next:
  lvmstrap: Change diskinfo to use GenerateTable
  Get rid of constants.RAPI_ENABLE
  Remove references to utils.debug
  ganeti-rapi, replace hardcoded exit value
  Add the bind-address option to ganeti-rapi
  noded: Abstract hard-coded sys.exit value...

burnin: move batch init/commit into a decorator

Many burnin steps initialize the batch queue at the beginning and commit
it at the end of their operation. This patch moves this code to a
decorator, in order to reduce redundant code.

Signed-off-by: Iustin Pop <iustin@google.com>...

burnin: Implement retryable operations

Some burnin steps are idempotent: e.g. reinstalling an instance (from
burning p.o.v.) can be done multiple times without any side-effects that
would affect later burnin steps. As such, failing the whole burnin
process due a reinstall failure is undesirable....

Merge branch 'next' into branch-2.1

• next:
  Create a new --no-voting option for masterfailover
  ganeti-masterd: allow non-interactive --no-voting
  Fix pylint warnings
  Add custom pylintrc
  bootstrap: Don't leak file descriptor when generating SSL certificate...

Use ReadFile.splitlines() rather than readlines

A few places in the code open a file "manually" rather than using our
wrapper function, because they need an array with the lines. Combining
the result of utils.ReadFile with splitlines() we get rid of the
exceptions....

Fix serial_no field on instances

The instance objects did not get a serial_no field. This patch adds a
new constants for the field name and uses it for all three cases
(cluster, nodes, instances).

Reviewed-by: imsnah

First run at cfgupgrade for 2.0 upgrades

This patch makes cfgupgrade work on empty cluster (i.e. no instances),
up to a point that the config file can be converted from 1.2 to 2.0.
This is not yet complete, though.

Reviewed-by: ultrotter

burnin: do not use drained nodes

This patch updates burnin not to use drained nodes (similar to the
handling of offline nodes).

Reviewed-by: imsnah

lvmstrap: allow removable devices too

For testing or just in case a device is exported by a bad driver with
the 'removable' flag set, this patch adds a flag to lvmstrap that allows
it to use these devices too.

Reviewed-by: ultrotter

Rework the execution model in burnin

This patch changes (significantly) the execution model in burnin:
- for all runs, (almost) all instance mods in a single Burn* procedure
are done as part of a job; so for example add disk, stop, remove
disk, start are no longer done as separate jobs but as a single job...

burnin: only call self.GrowDisks() if needed

In case we pass --disk-grow 0[,0..] then we should not call GrowDisks as it
prints confusing log lines.

Reviewed-by: imsnah

burnin: update migration to latest log formatting

Reviewed-by: ultrotter

burnin: redo the output formatting

Since we added many more tests in burnin, the output became almost
unreadable. This patch changes the output to an indented one, so that
the different phases and operations of burnin are more easily
understood.

Reviwed-by: ultrotter

burnin: introduce instance alive checks

This patch adds instance alive checks after most start operations. The
check is done in a custom way:
- the instance is expected to have an http server up and running
- and it should server the '/hostname.txt' resource containing the...

burnin: Add activate/deactivate disks

Reviewed-by: imsnah

burnin: do not use offline nodes

This patch makes burnin skip the offline nodes in it's builtin node
selection. It also removes an extra line.

Reviewed-by: imsnah

burnin: don't do export/import for file storage

This is currently not supported, so don't try to do export/import in
this case.

Reviewed-by: imsnah

Burner: fix ImportExport

Update the call to CreateInstance used in ImportExport to the new API for
multidisk/nic.

Reviewed-by: iustinp

burnin: fix default growth parameter

Reviewed-by: ultrotter

Remove the logger.py module

Since now we use only one function from the logger module
(SetupLogging), we move it to utils.py (which is already imported by all
users of this function), and we remove the module.

Reviewed-by: imsnah

Burnin OpGrowDisk

With this patch both the os and the swap disk are grown during
burnin. You can pass an increase size of 0 to skip this operation.

and:

burnin: don't try to grow diskless instances

When burnin is run on a diskless instance, it fails when trying...

Disable re-reading of config file

Since the objects read from the config file are passed to the various
threads, it's unsafe to re-read the config file (and throw away
ConfigWriter._config_data). As such, we disable the re-reading of the
file (since now the master is the owner the file, it makes not sense to...

Remove auto_balance from burnin/cmdlib

There is no such feature in trunk yet.

Reviewed-by: iustinp

Another burnin fix

This is a result of the log timestamp changes.

Reviewed-by: imsnah

Improve cli.SubmitOpCode

Currently, the feedback_fn argument to SubmitOpCode is no longer used.
We still need it in burnin, so we re-enable it by making the code call
that function with the msg argument in case feedback_fn is callable. The
patch also modifies burnin to accept the new argument format (msg is not...

cfgupgrade: Implement upgrading to Ganeti 2.0 configuration

Reviewed-by: iustinp

cfgupgrade: Add logging module

Reviewed-by: iustinp

Burnin: Use iallocator in import/export

Currently the iallocator option is ignored by burnin at import/export
time even if passed in. With this patch it becomes used. The log message
used by the importer is also changed to reflect this.

This patch also improves import/export on the non-iallocator case:...

LURemoveInstance: fix op.ignore_failures usage

Currently: the LURemoveInstance.Exec() method uses the ignore_failures
attribute of the OpRemoveInstance opcode, but it doesn't check for its
existence. The patch adds this attribute to _OP_REQP and to all the...

Convert cli.SubmitOpCode to use the master

This patch converts the cli.py SubmitOpCode method to use the unix
protocol and thus execute the opcodes via the master.

The patch allows a partial burnin to work with the master. Currently the
query opcodes, since they are executed via the SubmitOpCode, are...

Implement memory size option for burnin

This simple patch adds memory size for the burnin instances, which helps
testing allocator algorithm response based on the available node memory
and instance memory size values.

Reviewed-by: ultrotter

burnin: improve instance create message

When the allocator or a non-mirrored template are used the message shown by
burnin at instance creation time was wrong. Fixing it.

Reviewed-by: iustinp

Add instance rename to burnin

This patch adds the possibility to execute instance renaming when
doing a cluster burnin. The optional parameter --rename takes
one 'spare' instance name.
Given this option it renames all given instances to the spare one
and back to their original name....