Synnefo » snf-ganeti

Change gnt-instance list to the hvparams model

This is just a change of the various hvm_ and pvm parameters to the hv
model. Parameters are queried via hv/$name or via the whole dict as
returned by hvparams.

Reviewed-by: ultrotter,imsnah

Temporary fix for dual hvm/pvm instances

We have a problem with the current model of combining instance lists
from multiple hypervisors: we don't allow duplicates, but "xm list"
gives the same output for both pvm and hvm. This is a lack in the actual
xen hypervisor implementation/split between pvm and hvm, but for now we...

Export the hypervisor.ValidateParameters over RPC

The newly-added node-specific ValidateParams hypervisor method is
exported over RPC, using the semi-standard (success, message) return
value. Multi-node call, so that we call on both primary and secondary at...

Get rid of httperror module

This was a leftover from the early days of the remote API in Ganeti 1.2.

Reviewed-by: ultrotter

Add the hv/be params object attributes

This patch adds the instance and cluster be/hvparams attributes, and the
Fill* methods on the cluster object. Nothing else is changes, except
that these attributes exist now on the objects.

Reviewed-by: ultrotter

Fix a few rpc-related errors

This fixes:
- whitespace change, double lines between methods
- duplication of call_upload_file, introduced by mistake in rev 1795
and which went undetected because of the many changes in that ref
(only diff -b shows it clearly)...

Abstract checking own address into a function

Currently, we check if we have a given ip address (i.e. it's alive on
one of our interfaces) but manually calling TcpPing(source=localhost).
This works, but having it spread all over the code makes it hard to...

OS API: support for multiple versions in an OS

Allow multiple api versions in an OS. This is according to the OS API
changes design doc, by which an OS can support multiple versions of the
Ganeti API and if one is supported by Ganeti it will work. Since up to...

Add new HTTP server implementation

This patch adds another implementation of an HTTP server. It's
based on code of Python's BaseHTTPServer, from both version
2.4 and 3k. In the future we can write code to decide whether
we should fork for a request or not. Keep-alive is not supported....

Convert rpc module to RpcRunner

This big patch changes the call model used in internode-rpc from
standalong function calls in the rpc module to via a RpcRunner class,
that holds all the methods. This can be used in the future to enable
smarter processing in the RPC layer itself (some quick examples are not...

Small random fixes

Indentation in bootstrap was wrong and some names in cmdlib.py were not
right.

Reviewed-by: imsnah

Update scripts and qa config for changed hypervisor names.

Reviewed-by: ultrotter

Fix for gnt-cluster init.

Reviewed-by: iustinp

rpc.call_instance_migrate: pass the whole instance

Currently the call_instance_migrate call only passes the instance name;
we need to pass the whole object for the hypervisor_type changes (all
the other individual instance rpc calls already pass the instance...

Implement job auto-archiving

This patch adds a new luxi call that implements auto-archiving of jobs
older than a certain age (or -1 for all completed jobs), and the gnt-job
command that makes use of this (with 'all' for -1).

Reviewed-by: imsnah

backend.py change to get cluster name from master

Currently there are three function in backend that need the cluster name
in order to instantiate an SshRunner. The patch changes these to get the
cluster name from the master in the rpc call; once the multi-hypervisor...

Fix gnt-job list with empty timestamps

In case the job object doesn't have a timestamp (which is a separate
issue), the listing should not break. We fix this by changing the
FormatTimstamp function itself to return '?' in case the timestamp
doesn't look good (note that it still can break if non-integers are...

Fix SshRunner breakage from the changed API

More places actually use the SshRunner than just the gnt-cluster
commands.

Reviewed-by: ultrotter

Fix ssconf.GetMasterAndMyself

The ssconf migration left this out.

Reviwed-by: imsnah,ultrotter

Don't pass sstore to LUs anymore

sstore is no longer used in LUs.

Reviewed-by: iustinp

Convert cmdlib.py

Replacing ssconf with configuration. Cluster rename is broken and stays
that way.

Reviewed-by: iustinp

Convert rpc.py

Replacing ssconf with utility functions.

Reviewed-by: iustinp

Convert mcpu.py

Replacing ssconf with configuration.

Reviewed-by: iustinp

Convert backend.py

Replacing ssconf with simpleconfig.

Reviewed-by: iustinp

Move functions from ssconf.py elsewhere

These functions will be used to access config values instead of using
ssconf.

Reviewed-by: iustinp

Remove last use of utils.RunCmd from the watcher

The watcher has one last use of ganeti commands as opposed to sending
requests via luxi. The patch changes this to use the cli functions.

The patch also has two other changes:
- fix the docstring for OpVerifyDisks (found out while converting...

Move instantiation of config into bootstrap.py

Future patches will add even more variables to the cluster config.
Adding more parameters wouldn't make the function easier to use and
it doesn't make sense to pass them to another function, as it's
only done once in bootstrap.py on cluster initialization....

Enhance the job-related timestamps

This patch adds start, stop, and received timestamp for jobs (and allows
querying of them), and allows querying of the opcode timestamps.

Reviewed-by: imsnah

Add opcode execution log in job info

This patch adds the job execution log in “gnt-job info” and also allows
its selection in “gnt-job list” (however here it's not very useful as
it's not easy to parse). It does this by adding a new field in the query
job call, named ‘oplog’....

Enable the use of shared secrets

This patch enables the use of the shared secrets for DRBD8 disks, using
(hardcoded in constants.py) the md5 digest algorithm.

For making this more flexible, either we implement a cluster parameter
(once the new model is in place), or we can make it ./configure-time...

Implement job summary in gnt-job list

It is not currently possibly to show a summary of the job in the output
of “gnt-job list”. The closes is listing the whole opcode(s), but that
is too verbose. Also, the default output (id, status) is not very
useful, unless one looks for (and knows about) an exact job ID....

Move the pseudo-secret generation to utils.py

The bootstrap code needs a pseudo-secret and this is currently generated
inside the InitGanetiServerSetup function. Since more users will need
this, move it to utils.py

Reviewed-by: ultrotter

Add checks for tcp/udp port collisions

In case the config file is manually modified, or in case of bugs, the
tcp/udp ports could be reused, which will create various problems
(instances not able to start, or drbd disks not able to communicate).

This patch extends the ConfigWriter.VerifyConfig() method (which is used...

Allow listing of the serial_no via gnt-* list

This patch adds listing of the serial_no attribute in gnt-instance and
gnt-node list, and updates to the manpages to reflect the change.

Reviewed-by: ultrotter

Switch the global serial_no to the top object

Currently the serial_no that is incremented every time the configuration
file is written is located on the 'cluster' object in the configuration
structure. However, this is wrong as the cluster serial_no should be...

Replace a cfg.AddInstance with UpdateInstance

This seems to be the last (deprecated) use of AddInstance in order to
update an instance.

The patch also removes a whitespace-at-eol case.

Reviewed-by: ultrotter

Fix a broken format string

This patch fixes a broken format string. It's expecting 3 parameters, but only
gets 2. This change will add the missing parameter. This is a forward-port
of the fix in Ganeti 1.2

Reviewed-by: imsnah

Switch to static minors for DRBD

With some todos remaining, this patch switches the DRBD devices to use
the passed minors, and the cmdlib code (add instance and replace disks)
to request and assign minors to the DRBD disks.

Todos:
- look at the disk RPC calls to see which can be optimized away, since...

Fix disk replace secondary with static minors

The code in 'updating instance configuration' section of the replace
disks with change secondary node was setting a wrong new logical_id for
the drbd devices (only set the new node, not the new minor). The patch...

Apply filter properly in LUQuery{Nodes, Instances}

Currently when not locking all nodes/instances are returned, regardless
if the user asked only for some of them. With this patch we return to
the previous behaviour:
- if no names are specified return info on all current ones...

Add utils.ReadFile function

It abstracts exception handling and is like a complement to
utils.WriteFile.

Reviewed-by: iustinp

Parallelize Tag operations

For now we lock the instance/node for adding/deleting tags from it, but
we could probably in the future do without, with more support from the
config for atomic operations.

Reviewed-by: iustinp

Parallelize LURemoveExport

Reviewed-by: imsnah

Parallelize LUCreateInstance

Finally, instance create on different node, without iallocator, can run
in parallel. Iallocator usage still needs all nodes to be locked,
unfortunately. As a bonus most checks which could have been moved to
ExpandNames, before any locking is done....

LockSet: forbid add() on a partially owned set

This patch bans add() on a half-acquired set. This behavior was
previously possible, but created a deadlock if someone tried to acquire
the set-lock in the meantime, and thus is now forbidden. The
testAddRemove unit test is fixed for this new behavior, and includes a...

Use is_owned to determine whether to unlock

Now that is_owned is public we don't need to play games at the end of an
LU. If we're still owning anything we just release it.

Reviewed-by: imsnah

Fix LockSet._names() to work with the set-lock

If the set-lock is acquired, currently, the _names function will fail on
a double acquire of a non-recursive lock. This patch fixes the behavior,
and some lines of code added to the testAcquireSetLock test check that...

Remove locking of instances in certain queries

This patch is similar to the node patch (rev 1650). We disable locking
of instance (and nodes) if we only query static information.

Reviewed-by: ultrotter

Add ConfigWriter._UnlockedGetInstanceList/Info()

This patch splits the GetInstanceInfo and GetInstanceList methods into
two parts, one locked one _Unlocked similar to the way nodes are
queried.

Reviewed-by: ultrotter

Optimize the OpQueryNodes for names only

Currently, OpQueryNodes is locking all nodes (in shared mode), which
will also block the special case of querying only for the node names
(this is needed for gnt-cluster command, for example). There is no
logical requirement to not give the administrator enough power if she/he...