Synnefo » snf-ganeti » ganeti-local

Implement software release version checks too

Currently the LUVerifyCluster only reports the protocol version changes,
not software ones. This is useful to know/monitor, so we add this too as
a warning.

Reviewed-by: ultrotter

Fix some more pylint errors

Two are real errors (invalid names) and one is style error (overriding
name from outer scope).

Reviewed-by: ultrotter

Add calls in the intra-node migration protocol

Currently the hypervisor is expected to do all the migration from the
source side. With this patch we also add the option of passing some
information to the target side, and starting some operation there.

As a bonus, a function to cleanup any started operation is included....

Convert RenameInstance to (status, data)

This allows the rename failures to show the ouput of OS scripts.

Reviewed-by: ultrotter

Convert AddOSToInstance to (status, data)

This allows the install and reinstall instance to return (hopefully)
relevant log files from the OS create scripts.

Reviewed-by: ultrotter

Switch call_blockdev_create call to (status, data)

This allows errors to be visible at the user level instead of just node
daemon logs.

Reviewed-by: ultrotter

Forward port the live migration from 1.2 branch

This is forward port via copy (and not individual patches cherry-pick)
of the latest code on the 1.2 branch related to the migration.

The changes compared to 1.2 are the fact that we don't need the
IdentifyDisks step anymore (the drbd rpc calls are independent now), and...

backend: rename AttachOrAssemble to Assemble

Since now the Assemble function is different than Attach, we rename this
backend function to show that the intent is to fully assemble the device
(and it's always allowed to modify the device).

Reviewed-by: ultrotter

backend: Remove symlinks by disk name

This is a modified forward-port of commit 1184 on the 1.2 branch:

backend: Remove symlinks by disk name, not using a wildcard

Reviewed-by: ultrotter

The changes to the original patch are related to the docstring style and...

Fix the _RemoveBlockDevLinks() function

This is a forward-port of commit 1163 on the 1.2 branch:
This fixes the removal of the instance symlinks (probably breakage from
the glob changes).

Reviewed-by: imsnah

Catch BlockDeviceError when starting instance

This is a forward-port of commit 1149 on the 1.2 branch:
_GatherAndLinkBlockDevs used to raise the errors.BlockDeviceError
exception when it failed to create a block device, and with this patch
set it does so also when it fails to create a symlink to it....

Simplify hypervisor block_devices structure

This is a partial forward-port of commit 1136 on the 1.2 branch:

The hypervisor doesn't need to be passed the whole block device
  structure, so we'll just give it the block device name on the local
  node, and the name as seen by the instance. This will make it easier to...

cleanup: fix export NIC count the same way as disk

For safety, we use the same algorithm as in disk count.

Reviewed-by: amishchenko

cleanup: more unused vars

Reviewed-by: amishchenko

Fix epydoc format warnings

This patch should fix all outstanding epydoc parsing errors; as such, we
switch epydoc into verbose mode so that any new errors will be visible.

Reviewed-by: imsnah

RPC: Compress file upload data

Adding compression to larger amounts of data is more efficient than
transferring it (len(nodes) - 1) times over the network without
compression. We were able to compress a 800KB config file to about
30 KB, which is about 40 KB with Base64 encoding (required due to...

Fix gnt-cluster verify w.r.t. rpc changes

This partially reorganizes the cluster verify LU:
- introduce constants for the node verify rpc call
- move from additional rpc calls to a single rpc call, the
call_node_info, which gaters all data needed...

Revert "Get rid of ssconf"

This partially reverts the "Get rid of ssconf" patch.

It adds back a simpler version of the SimpleStore class, and drops the
WritableSimpleStore class. The new version of the class also has
node_list as a new key, and increases the size of the keys so that big...

Pass ssconf values from master to node

Instead of parsing the configuration on the node, we pass the ssconf
values from the master.

Reviewed-by: iustinp

Add RPC call to update ssconf files

Reviewed-by: iustinp

Update backend.py docstrings

This patch converts all of backend.py to epydoc formatting.

Reviewed-by: imsnah

Fix an error handling case

Found while reviewing documentation.

Reviewed-by: ultrotter

Some documentation updates

This fixes a few doc issues and converts a few docstrings to epydoc.

Reviewed-by: imsnah

Convert ImportOSIntoInstance to OS API 10

- Change ImportOSIntoInstance not to get any "os_disk" and "swap_disk"
arguments but to accept multiple target images to import, and to
return a list of booleans with the result of each import
- Change the relevant rpc call and the only caller to conform...

LUExportInstance: snapshot all disks

Rather than just snapshotting the "sda" disk, we'll snapshot all of the
instance disks. If we can't snapshot a disk for any reason we'll log an
error and proceed anyway: in this case the resulting export will miss a
disk. This also changes all the warning messages to self.LogWarning()....

Cleanup os_add/rename rpc for OS API 10

- remove now unused osdev and swapdev arguments from backend, noded,
rpc, cmdlib
- convert docstrings to epydoc

Reviewed-by: iustinp

AddOSToInstance: convert to api10

Reviewed-by: iustinp

Add new OSEnvironment function

This function calculates the basic environment for OS scripts in API
version 10.

Reviewed-by: iustinp

Change OSFromDisk's docstring to epydoc

Plus update it with the real variable name

Reviewed-by: iustinp

Change the backend to use the beparams

The backend.FinalizeExport function is changed to use the beparams
instead of the instance attributes. Future enhancements should be done
in order to export and import/reuse the whole be/hv params.

Reviewed-by: ultrotter

Export the hypervisor.ValidateParameters over RPC

The newly-added node-specific ValidateParams hypervisor method is
exported over RPC, using the semi-standard (success, message) return
value. Multi-node call, so that we call on both primary and secondary at...

OS API: support for multiple versions in an OS

Allow multiple api versions in an OS. This is according to the OS API
changes design doc, by which an OS can support multiple versions of the
Ganeti API and if one is supported by Ganeti it will work. Since up to...

rpc.call_instance_migrate: pass the whole instance

Currently the call_instance_migrate call only passes the instance name;
we need to pass the whole object for the hypervisor_type changes (all
the other individual instance rpc calls already pass the instance...

Fix SshRunner breakage from the changed API

More places actually use the SshRunner than just the gnt-cluster
commands.

Reviewed-by: ultrotter

Convert hypervisor

Replacing ssconf with configuration.

Reviewed-by: iustinp

Never remove job queue lock in node daemon

Otherwise, corruption could occur in some corner cases. E.g. when
LeaveNode is running in a child and is in the process of removing
queue files, the main process gets killed, started again and gets
a request to update the queue. This is rather extreme corner case,...

Pass hypervisor type to the OS scripts

It's handy to make the os scripts know which hypervisor the instance is
going to run under. In order not to change the os API we pass this
information in the environment, where the os scripts can access it if
they're hypervisor-aware....

backend: Add optional exclusion list to _CleanDirectory

The code cleaning the queue will make use of it.

Reviewed-by: iustinp

backend: Add function to check whether file is in queue dir

Another function will need to check whether its parameters
are job queue files.

Reviewed-by: iustinp

Add job queue RPC functions

jobqueue_update: Uploads a job queue file's content to a node. The
most common operation is to upload something that we already have
in a string. Unlike in the upload_file function, the file is not
read again when distributing changes, but content has to be passed...

Clean job queue directories when leaving cluster

Old job files shouldn't be left on nodes removed from a cluster.

Reviewed-by: iustinp

Fix pylint-detected issues

This is mostly:
- whitespace fix (space at EOL in some files, not all, broken
indentation, etc)
- variable names overriding others (one is a real bug in there)
- too-long-lines
- cleanup of most unused imports (not all)...

Rework master startup/shutdown/failover

This (big) patch reworks the master startup/shutdown and the fixes the
master failover.

What does the patch do?

For master start/stop:
- remove the old ganeti-master script and its associated man page
- moves the ip start/stop directly into the backend.(Start|Stop)Master...

Distribute the queue serial file after each update

This patch adds distribution of the queue serial file after each write
to it (but before a new job is created and written with that ID, and
before a response is returned, so we should be safe from crashes in...

Fix backend.NodeVolumes handling of LVM output

This is the same fix as for GetVolumeList.

I've checked manually and all other places that call lvm commands are
already checking the output validity in terms of correct number of
fields.

Reviewed-by: ultrotter

Allow VNC_PASSWORD_FILE to be rpc-uploaded

What could possibly go wrong?

Reviewed-by: iustinp

Add a rpc call for BlockDev.Close()

This patch adds rpc layer calls (in rpc.py and the equivalent in
ganeti-noded) to close a list of block devices, and the wrapper in
backend.py that takes a list of Disk objects, identifies them and
returns correctly formatted results....

Add migration support at the rpc layer

This patch adds the migration rpc call and its implementation in the
backend. The patch does not deal with the correct activation of disks.

Because of the new RPC, the protocol version is increased.

Reviewed-by: imsnah

Reduce chance of ssh failures in verify cluster

The cluster verify builds a sorted list of nodes and passes that to all
the nodes (in parallel) for ssh checks. This means that for a cluster
with N nodes, there will be approximately N simultaneous connections to...

Move iallocator script execution to ganeti-noded

Currently the iallocator execution takes place in the master, which is a
violation of the current architecture, and will create problems with a
threaded master daemon.

This patch moves the execution to the backend, similar to the hooks...