Synnefo » snf-ganeti

Start implementation of parallel burnin

This patch introduces a simple framework for executing jobs in parallel
in burnin (the ExecJobSet function) and the "--parallel" command line
flag.

The patch also changes the instance creation to run in parallel when the...

Increase the number of threads to 25

Since our locks are not gathered nicely, we can have jobs that are
actually blocking on locks (parallel burnin shows this), so at least we
need to increase the number of threads above the usual number of jobs we
could have in a such a case....

Fix SshRunner breakage from the changed API

More places actually use the SshRunner than just the gnt-cluster
commands.

Reviewed-by: ultrotter

Update document describing cluster security

It may need further updates, but here's a start.

Reviewed-by: ultrotter

Fix a mistake in the gnt-backup man page

The actual location is /export, not /exports.

Reviewed-by: ultrotter

Remove references to Twisted framework

Reviewed-by: iustinp

Don't pass sstore to LUs anymore

sstore is no longer used in LUs.

Reviewed-by: iustinp

Convert ganeti-watcher

Use RPC calls instead of ssconf.

Reviewed-by: iustinp

Convert gnt-cluster

Replace ssconf with configuration.

Reviewed-by: iustinp

Convert cmdlib.py

Replacing ssconf with configuration. Cluster rename is broken and stays
that way.

Reviewed-by: iustinp

Convert rpc.py

Replacing ssconf with utility functions.

Reviewed-by: iustinp

Convert mcpu.py

Replacing ssconf with configuration.

Reviewed-by: iustinp

Convert backend.py

Replacing ssconf with simpleconfig.

Reviewed-by: iustinp

Move functions from ssconf.py elsewhere

These functions will be used to access config values instead of using
ssconf.

Reviewed-by: iustinp

Fix the watcher with down nodes

The watcher didn't handle the down nodes, fix this by ignoring (in
secondary node reboot checks) any node that doesn't return a boot id.

Reviewed-by: imsnah

Remove last use of utils.RunCmd from the watcher

The watcher has one last use of ganeti commands as opposed to sending
requests via luxi. The patch changes this to use the cli functions.

The patch also has two other changes:
- fix the docstring for OpVerifyDisks (found out while converting...

Add cluster options from ssconf to configuration

ssconf will become write-only from ganeti-masterd's point of view,
therefore all settings in there need to go into the main configuration
file.

Reviewed-by: iustinp

Change the results from cli.PollJob

Curently PollJob accepts a generic job, but will return (history
artifact) only the first opcode result. This is wrong, as it doesn't
allow polling of a job with multiple results.

Its only caller (for now) is also changed, so no functional changes...

Build HTML from RST input

This patch also adds the design documents to Makefile.am.

Reviewed-by: iustinp

Add output of job/opcode timestamps

This patch adds posibility of selection of job/opcode timestamps in
gnt-job list and info.

The code handling the possible cases (None or a valid timestamps) are
ugly though...

Reviwed-by: imsnah

Small fixes for master daemon design document

It said CLI/RAPI will talk to master using HTTP, which isn't true. Add
a reference to job queue design document. Small typos.

Reviewed-by: iustinp

locking design: code path and declarations

Reviewed-by: iustinp

locking design: talk about removing locks

Reviewed-by: iustinp

Abstract the timestamp formatting into cli.py

Currently we format the timestamp inside the gnt-job info function. We
will need this more times in the future, so move it to cli.py as a
separate, exported function.

Reviewed-by: imsnah

Add an 'index' of design documents

This will be an overview document, enumerating the changes without going
into details and pointing to the actual documents.

Reviewed-by: ultrotter

Move a hardcoded constant to constants.py

For now we only use the ‘C’ protocol so we can put it in constants.py
instead of hardcoding it.

Reviewed-by: imsnah

Extend DRBD disks with shared secret attribute

This patch, which is similar to r1679 (Extend DRBD disks with minors
attribute), extends the logical and physical id of the DRBD disks with a
shared secret attribute. This is generated at disk creation time and...

Implement job summary in gnt-job list

It is not currently possibly to show a summary of the job in the output
of “gnt-job list”. The closes is listing the whole opcode(s), but that
is too verbose. Also, the default output (id, status) is not very
useful, unless one looks for (and knows about) an exact job ID....

Move the pseudo-secret generation to utils.py

The bootstrap code needs a pseudo-secret and this is currently generated
inside the InitGanetiServerSetup function. Since more users will need
this, move it to utils.py

Reviewed-by: ultrotter

Add checks for tcp/udp port collisions

In case the config file is manually modified, or in case of bugs, the
tcp/udp ports could be reused, which will create various problems
(instances not able to start, or drbd disks not able to communicate).

This patch extends the ConfigWriter.VerifyConfig() method (which is used...

Allow listing of the serial_no via gnt-* list

This patch adds listing of the serial_no attribute in gnt-instance and
gnt-node list, and updates to the manpages to reflect the change.

Reviewed-by: ultrotter

Switch the global serial_no to the top object

Currently the serial_no that is incremented every time the configuration
file is written is located on the 'cluster' object in the configuration
structure. However, this is wrong as the cluster serial_no should be...

Replace a cfg.AddInstance with UpdateInstance

This seems to be the last (deprecated) use of AddInstance in order to
update an instance.

The patch also removes a whitespace-at-eol case.

Reviewed-by: ultrotter

Fix iallocator name

port forward of patch from revision 1690 with following message:

Patch on revision 1686 used the wrong field: ial.name, which is the instance
name and not the iallocator name. self.op.iallocator is the right field.

Sorry for this inconvenience....

Design doc for the master daemon

It's not complete, but I hope it's up to date. It's restructured text,
but no make rules or such for html output are needed, it's readable
as-is.

Reviewed-by: imsnah

Switch to static minors for DRBD

With some todos remaining, this patch switches the DRBD devices to use
the passed minors, and the cmdlib code (add instance and replace disks)
to request and assign minors to the DRBD disks.

Todos:
- look at the disk RPC calls to see which can be optimized away, since...

Fix disk replace secondary with static minors

The code in 'updating instance configuration' section of the replace
disks with change secondary node was setting a wrong new logical_id for
the drbd devices (only set the new node, not the new minor). The patch...

Apply filter properly in LUQuery{Nodes, Instances}

Currently when not locking all nodes/instances are returned, regardless
if the user asked only for some of them. With this patch we return to
the previous behaviour:
- if no names are specified return info on all current ones...

Add utils.ReadFile function

It abstracts exception handling and is like a complement to
utils.WriteFile.

Reviewed-by: iustinp

Make burnin aware of hvm device type flags.

Merged from branches/ganeti/ganeti-1.2 r1648
Use static values for new hvm instance flags

Reviewed-by: iustinp

gnt-instance: fix tags commands online help

It used to refer to "nodes", which was confusing.

Reviewed-by: iustinp

Parallelize LUSetClusterParams (and add a FIXME)

Reviewed-by: imsnah

Parallelize LURemoveExport

Reviewed-by: imsnah

Parallelize LUCreateInstance

Finally, instance create on different node, without iallocator, can run
in parallel. Iallocator usage still needs all nodes to be locked,
unfortunately. As a bonus most checks which could have been moved to
ExpandNames, before any locking is done....

LockSet: forbid add() on a partially owned set

This patch bans add() on a half-acquired set. This behavior was
previously possible, but created a deadlock if someone tried to acquire
the set-lock in the meantime, and thus is now forbidden. The
testAddRemove unit test is fixed for this new behavior, and includes a...

Use is_owned to determine whether to unlock

Now that is_owned is public we don't need to play games at the end of an
LU. If we're still owning anything we just release it.

Reviewed-by: imsnah

Fix LockSet._names() to work with the set-lock

If the set-lock is acquired, currently, the _names function will fail on
a double acquire of a non-recursive lock. This patch fixes the behavior,
and some lines of code added to the testAcquireSetLock test check that...

jqueue: Add common RPC error handling function

We didn't decide yet what exactly it should do with failed nodes.

Reviewed-by: ultrotter

Add an atomic ConfigWrite.GetAllInstanceInfo()

In order to be able to query instance without locking them, we need the
same atomic query of multiple instances as for nodes.

Reviewed-by: ultrotter

Do not use jobs in gnt-instance _ExpandNames()

In the gnt-instance script, _ExpandNames() uses jobs to query instance
names. This is not optimal, so we change it to use queries.

Reviewed-by: ultrotter