Synnefo » snf-ganeti

Modify cli.JobExecutor to use SubmitManyJobs

This patch changes the generic "multiple job executor" to use the many
jobs submit model, which automatically makes all its users use the new
model.

This makes, for example, startup/shutdown of a full cluster much more...

Fix some epydoc style issues

99% of the epydoc return tags are "@return:", but each of the modified files
had one "@returns:" line. We fix this for consistency.

Reviewed-by: imsnah

Instance parameters: force typing

We want all the hv/be parameters to have a known type, rather than a
random mix of empty string, boolean values, and None, so we declare the
type of each variable and we enforce/convert it.

- Add some new constants for enforceable value types...

Implement lockless query operations

This patch adds the framework for, and enables lockless OpQueryInstances. This
means that instances will be shown in ERROR_up or ERROR_down state, even though
this is not an error (but just an in-progress job).

The framework is implemented as follows:...

Rework the multi-instance gnt commands

This patch changes the multi-instance gnt-* commands (gnt-instance
start/stop, gnt-node evacuate/failover) such that the individual
operations are submitted in parallel, ideally improving the speed of the
execution....

Add job queue size limit

A job queue with too many jobs can increase memory usage and/or make
the master daemon slow. The current limit is just an arbitrary number.
A "soft" limit for automatic job archival is prepared.

Reviewed-by: iustinp

jqueue: Allow jobs waiting for locks to be canceled

- Add new "canceling" status
- Notify clients when job is canceled
- Give a return value from CancelJob
- Handle it in the client library

Reviewed-by: iustinp

Enable auto-unit formatting in script output

This patch enables by default the old 'human-readable' option, but in a
slightly different model.

The option is now called "units" and takes either:
- 'h' for automatic formatting
- 'm', 'g' or 't' for mebi/gibi/tebibytes...

Reuse HTTP client pool for RPC

ganeti-masterd: Add initialization and shutdown of RPC pool. It needs
to be shutdown before forking.

ganeti.cli: Add decorator function to initialize and shutdown RPC pool.

ganeti.rpc: Add functions to initialize and shutdown RPC pool. Throw...

Fix gnt-* command with --submit

Currently we don't catch the JobSubmittedException (although the code to
format it nicely is there). The patch adds the exception name to the
catch list and makes --submit work.

Reviewed-by: imsnah

Convert cli.py to logging

We also add two function for printing messages, so that scripts won't
have to import logger to get these. They are a simple extension over the
logger ones, as they accept the call style from logging:
ToStdout("Message: %s", msg)...

Add cli function for beparams validation

This is used to validate the parameters in both “gnt-instance add” and
“gnt-backup import”.

Reviewed-by: ultrotter

Add two new options types for CLI usage

For the new 2.0-style command line options, we need to parse strings of
the type:
ident:key=val[,...]
and
key=val[,...]

This patch adds two new option builders for these two, which return
(ident, {key=val,}) and {key=val,} for the above two formats. It also...

Add a simple timespec parsing function

This function will be used for auto-archiving jobs via the command line.
The function is pretty simple, we only support up to weeks since months
and higher are not 'precise' entities, and dealing with them would
require us to start using calendar functions....

Change the results from cli.PollJob

Curently PollJob accepts a generic job, but will return (history
artifact) only the first opcode result. This is wrong, as it doesn't
allow polling of a job with multiple results.

Its only caller (for now) is also changed, so no functional changes...

Rewrite the 'only submit job' handling in scripts

The "sys.exit(0)" was not nice as you couldn't differentiate it from
other exit codes. We change this to a specially defined exception for
this, so that multi-opcode commands can handle this nicely.

Reviewed-by: imsnah

Make sure that client programs get all messages

This is a large patch, but I can't figure out how to split it without
breaking stuff. The old way of getting messages by always getting the
last one didn't bring all messages to the client if they were added...

jqueue: Change log message time format

See the comment in the patch.

Reviewed-by: ultrotter

Implement job submission for scripts

This patch adds the infrastructure for executing a job in background,
instead of foreground, via a new “--submit” option. The behaviour is
that the job ID is printed and the script will immediately exit.

The patch also converts gnt-node list to this model (yes, this will be a...

Make gnt-* commands fail nicely on non-masters

This patch adds a check that we are on the master after failing to
connect to the socket, and log nicely the master name.

Reviewed-by: ultrotter

Unify SetupDaemon/SetupLogging

The 'old-style' info, error, debug logs do not make much sense. This
patch unifies the SetupLogging and SetupDaemon functions. As a result,
all the commands logs to a 'commands.log' file.

The patch also changes the log setup to keep going if there's an error...

First version of user feedback fixes

This patch contains a raw version for fixing feedback_fn.

The new mechanism works as follows:
- instead of a per-Processor feedback_fn, there's one for each
ExecOpCode, so that feedback for different opcodes go via possibly...

Change masterd/client RPC protocol

- Introduce abstraction class on client side
- Use constants for method names
- Adopt legacy function SubmitOpCode to use it

Reviewed-by: iustinp

Implement handling of luxi errors in cli.py

Currently the generic handling of ganeti errors in cli.py (GenericMain
and FormatError) only handles the core ganeti errors, and not the client
protocol errors (which live in a separate hierarchy).

This patch adds handling of luxi errors too, and also adds another luxi...

Remove unreachable code from cli.SubmitOpCode

Reviewed-by: iustinp

Revert "CLI: remove command opts/args in "gnt-X""

This reverts commit 976.

Reviewed-by: ultrotter

Move the 'cmd' lock from cli.py to ganeti-masterd

This patch removes the lock and the lock options from cli.py and moves
them to the master.

Later during development we can remove it completely, but for now it's
good to protect any other tool that uses the lock directly....

Move some checks from cli.py to luxi.py

The idea of cli.py and luxi.py is that all protocol checks should be in
luxi, and cli.py should just offer some helpful shortcuts for the
command line scripts.

This patch removes the result checks from cli and adds some other checks...

Improve the documentation of query output fields

The gnt-node and gnt-instance list commands have a customizable list of
output fields, but the list is not up to date (in the man page) and not
easily understandable from the ‘--help’ output.

This patch updates the man pages and adds the available fields and...

Add support for command aliases

Passing a new aliases dict to generic main we can easily support aliases for
compatibility reasons or simply useability.

Reviewed-by: iustinp

Make utils.RunCmd log failures when using debug

This patch adds logging of command failures to the debug log in case the
user either started the command (gnt-*) or the node daemon with the
debug flag.

Reviewed-by: imsnah

Small cleanup for error formatting

Reviewed-by: ultrotter

Change the signature of some methods of mcpu.Processor

This patch moves the passing of the feedback_fn argument from the
(Exec|Chain)OpCode to the initialization of the Processor instance.

Reviewed-by: imsnah

Fix AskUser to not die on extra input

Currently, AskUser dies with -ESPIPE if the user gives more than one
character plus newline. This is because the python library, while
returning only two chars from the readline(2) call, will cache the rest
of the input, and when we do a write, it will try to seek back to just...

Fix a wrong exception name

This was introduced in rev 208.

Reviewed-by: imsnah

Enhance cli.SubmitOpcode to use custom parameters

This patch allows the SubmitOpcode function to take a pre-created
Processor instance and a different feedback function. This is helpful
when submitting more than one opcode from a shell script.

Reviewed-by: imsnah

Add source file option for {add,remove}-tags

This patch adds a '--from' option to the {add,remove}-tags commands that
allows the command to read from a file or from stdin (when passing a
dash for the filename).

Reviewed-by: imsnah

Extend GenericMain with a way to override options

This allows scripts to pass options to generic functions (options that
are not visible in the command line).

Reviewed-by: imsnah

Improve the cli.AskUser() function

This patch improves the AskUser function by allowing it to:
- choose from multiple choices (instead of only y/n)
- give help to the user
- preserve line breaks and whitespace formatting in the message

With this patch, an instance removal looks like this:...

Style changes for pep-8 and python-3000 compliance.

This changes the raising of exceptions from:
raise Exception, value
to
raise Exception(value)

as the first form will be removed in python-3000 and the second form is
preferred now.

The changes also involve a few cases of changing from raising standard...

Comment formatting updates.

Reviewed-by: iustinp

- Implement “gnt-node volumes”
- Create all --output options using a constant
- Put node checking code from opcodes into a single function
- Do the same for output fields

Reviewed-by: iustinp