query: change (debug-mode) field validation errors

Currently, the single assert just checks that the entire row is
consistent (true/false), and dumps the row and field definitions as an
accompanying message. This makes it very hard to understand what failed.

This patch changes this validation to show descriptive messages, which
makes it much faster in diagnosing invalid result.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename a few instance field titles

The conversion to querylib introduced unique field titles, which however
did an inconsistent rename of the instance vcpus/memory fields.

This patch makes both BE_MEMORY and _VCPUS be named with prefix “Config”
(which before was “Configured_”), and drops the “Runtime” (note no
underscore) prefix on the oper_ ones.

Rationale: for numeric fields, the width of the title defines the width
of the columns, as the numbers are usually small. Hence these columns
(more likely used than the BE ones) would be very wide, leading to a
not-so-nice display. I happened upon this when my terminals didn't
manage to fit a gnt-instance list anymore.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

query: Change internal result computation

While looking at the query library, I realized that while we have five
field statuses, making this a 5-dimensional space, four of them are
shrunk to a single possible value (None). Hence it should be possible to
convert this into a single value space plus extra 4 special constants.

This patch implements this, making (IMHO) the return value of normal
functions much simpler: you simply return the desired value, instead of
(QRFS_NORMAL, value); for the special results, you simply return
_FS_UNAVAIL, instead of (QRFS_UNAVAIL, None). This I believe does
simplify the code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Import QRFS_* and QFT_* in query.py

Currently, the QRFT/QFT constants are used with the module (e.g.
constants.QRFS_NORMAL) in the query library. Given that there are 176
uses of these constants in a module that has roughly ~1200 lines in
total (including whitespace), it makes sense to import these directly to
improve readability.

This patch does this and (IMHO) it does indeed make the code more to the
point; the entire query library is about queries, so using these
constants directly doesn't cause any misunderstandings.

Note: the epydoc links are not changed, since we want the generated docs
to point to the right place.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Build epydoc's module list from Makefile

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

KVM: don't fail removing non-existing nic path

This shouldn't normally happen, but if it does no need to fail.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

KVM: Perform network configuration in Ganeti

This patch introduces network configuration for KVM in Ganeti.

There are three problems with having KVM perform network configuration via ifup
scripts:
  a) Ganeti never gets to know the tap interface that is associated with an
     instance's NIC
  b) Migration of routed instances will cause network problems because the
     incoming KVM side configures the network as soon as it is spawned and not
     as soon as the migration finishes. This means that all routing
     configuration will be present in both, primary and secondary, nodes at the
     same time, possibly causing network disruption during the migration.
  c) We never get to know if the network configuration succeeded or not.

This patch moves network configuration from KVM to Ganeti, using KVM's ability
to receive already open tap devices as file descriptors.

_WriteNetScript is removed from hv_kvm.py, together with its unit tests.

Minor modifications are made to _ExecKVMRuntime to handle tap device
initialization. NIC <-> tap associations are stored under a new directory,
_ROOT_DIR/nic in a file-per-nic fashion.

The end-user semantics remain the same: The user can override the network
configuration by providing _KVM_NET_SCRIPT. If this is not present or
executable, the default constants.KVM_IFUP script is run.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Ship default kvm-ifup script

In order to retain backwards compatibility, we ship the shell script previously
generated by _WriteNetScript as a stand-alone script under PKGLIBDIR.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

KVM: Add auxiliary functions to handle tap devices

Add two new functions, _OpenTap and _ProbeTapVnetHdr, to
hypervisors.hv_kvm.

_ProbeTapVnetHdr checks if the host kernel supports the virtio net header
(IFF_VNET_HDR) feature. This feature allows KVM to ask the the host kernel to
perform checksumming and segmentation offload of the data passing through a
guest's virtio NIC, using a special header[1] prepended to data sent through
the tap interface. A qemu-kvm version later than 0.12 is required for this
feature to work.

_OpenTap opens a new tap device suitable for use with qemu-kvm's "-net
tap,fd=n" option. It also enables vnet_hdr support if requested and safe to do
so.

[1] struct virtio_net_hdr in include/linux/virtio_net.h

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

epydoc: specify files in tools/

This way we can add there non-python scripts

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Remove hard-coded OP_ID definitions

The removal requires adding a few pylint ignores, as it cannot know that
OP_ID will be defined at runtime.

Also switch the asserts to check for non-declaration.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Check consistency of the class names and OP_ID

As the class names should be now consistent with the OP_IDs, we add a
check for wrongly-defined OP_IDs.

However, the future removal of the hand-coded OP_IDs will render this
obsolete, so this check is introduced just to make sure that the
previous renaming patches did the right job, and it will then be
removed.

The consistency checks require renaming the test opcodes, which were
using arbitrary names, depending on test author. They are now all
standardized on OpTest (local scope).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpTestJobqueue and LUTestJobqueue

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpAddTags and LUAddTags

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpSearchTags and LUSearchTags

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpGetTags and LUGetTags

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpDelTags and LUDelTags

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpDiagnoseOS and LUDiagnoseOS

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpSetNodeParams and LUSetNodeParams

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpRemoveNode and LURemoveNode

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpQueryNodeStorage and LUQueryNodeStorage

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpQueryNodeVolumes and LUQueryNodeVolumes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpQueryNodes and LUQueryNodes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpPowercycleNode and LUPowercycleNode

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpModifyNodeStorage and LUModifyNodeStorage

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpMigrateNode and LUMigrateNode

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpNodeEvacuationStrategy and LUNodeEvacuationStrategy

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpAddNode and LUAddNode

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpStartupInstance and LUStartupInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpShutdownInstance and LUShutdownInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpSetInstanceParams and LUSetInstanceParams

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpReplaceDisks and LUReplaceDisks

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpRenameInstance and LURenameInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpRemoveInstance and LURemoveInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpReinstallInstance and LUReinstallInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpRecreateInstanceDisks and LURecreateInstanceDisks

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Rename OpRebootInstance and LURebootInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpQueryInstanceData and LUQueryInstanceData

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpQueryInstances and LUQueryInstances

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpMoveInstance and LUMoveInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpMigrateInstance and LUMigrateInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpGrowDisk and LUGrowDisk

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpFailoverInstance and LUFailoverInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpDeactivateInstanceDisks and LUDeactivateInstanceDisks

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpCreateInstance and LUCreateInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpConnectConsole and LUConnectConsole

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpActivateInstanceDisks and LUActivateInstanceDisks

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpSetGroupParams and LUSetGroupParams

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpRenameGroup and LURenameGroup

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpRemoveGroup and LURemoveGroup

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpQueryGroups and LUQueryGroups

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpAssignGroupNodes and LUAssignGroupNodes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpAddGroup and LUAddGroup

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpVerifyDisks and LUVerifyDisks

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpVerifyCluster and LUVerifyCluster

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpSetClusterParams and LUSetClusterParams

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpRepairDiskSizes and LURepairDiskSizes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpRenameCluster and LURenameCluster

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpRedistributeConfig and LURedistributeConfig

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpQueryClusterInfo and LUQueryClusterInfo

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpPostInitCluster and LUPostInitCluster

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpDestroyCluster and LUDestroyCluster

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpQueryConfigValues and LUQueryConfigValues

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpRemoveExport and LURemoveExport

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpQueryExports and LUQueryExports

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpPrepareExport and LUPrepareExport

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Rename OpExportInstance and LUExportInstance

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Fix the OP_ID of OpAssignGroupNodes

Commit 96276ae added the above opcode, but forgot to update the OP_ID
per our previous mailing list discussions. This patch makes the OP_ID
be consistent with the other IDs (OP_$object_$action).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Fix bug introduced by f5677ab

Commit f5677ab added the _VerifyOob functionality, but there are two
issues with this function:

- a real bug, as objects.Node doesn't have a 'master' attribute
- a design bug, as we need to verify the OOB programs not only on
  current masters, but on all nodes that can become masters via the
  auto-promote functionality; otherwise, an automated cluster operation
  could cause the cluster to become inconsistent (error in verify)

The Node.master bug didn't appear in QA because it is triggered only
with non-master-candidate nodes, and our QA doesn't test this
combination (and neither other node flags combination).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Merge branch 'devel-2.3' into devel-2.4

* devel-2.3:
  Allow burnin with hidden OSes

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Allow burnin with hidden OSes

burnin is a cluster/testing feature, so it makes sense that a hidden OS
can be used for it.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Bump version for Ganeti 2.4.0~beta1

Update the version in all necessary places. Update NEWS with release
date.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Update NEWS for Ganeti 2.4

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Update RPC protocol version for Ganeti 2.4

As it's quite easy to forget updating the protocol version (and was
actually forgotten for Ganeti 2.3), it is now derived from the
configuration version (which is equivalent with the Ganeti version). The
protocol version for Ganeti 2.4 will be 2040000.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Set +x on tools/ganeti-listrunner

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Merge branch 'devel-2.3'

Conflicts:
doc/design-2.3.rst
  This was a minor conflict with some rewording of a phrase

lib/client/gnt_instance.py
  Removed _FormatParameterDict function as it was moved to cli.py in
  master branch

Reviewed-by: Michael Hanselmann <hansmi@google.com>

query: Update docstrings

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Fix growing of DRBD disks

Patch 03941a26 made disk growth VG-aware, but the code has a bug: it
only works for LD_LV disks, and not LD_DRBD. To fix it, we add a
generic function to compute the per-VG growth requirements, and switch
to using that instead of manual build.

Furthermore, since the function _CheckNodesFreeDiskPerVG always takes
its input from this new function or _ComputeDiskSizePerVG, we change
the latter to always return a dict (possibly empty), instead of either
a dict or None. This makes the types more consistent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

List node parameters in gnt-group list

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

List node parameters (if any) in gnt-node info

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

List node parameters in gnt-cluster info output

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fix typos in RAPI docstrings, add unittest

This patch fixes a number of typos and standardizes RAPI resource
docstrings. A unittest is added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Move FormatParameterDict from gnt_instance.py to cli.py

This function is useful for other cli-tools who need to display
a dict based parameter field.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>

gnt-group modify: ndparams not correctly updated

ndparams should delete keys which are reset to default

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add Ganeti 2.4 design document

It contains links to the two implemented designs. The detailed documents
are removed from the visible table of contents to not show up on their
own.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

last_resort groups prioritized by iallocator plugin

Also change language slightly for preferred groups to look better now
that it's repeated.

Signed-off-by: Stephen Shirley <diamond@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Remove --master-netdev default value

Since the --master-netdev option is now used both at cluster init and at
cluster modify time, it cannot have a default value of DEFAULT_BRIDGE
(xen-br0) anymore. As such at cluster init we make the initialization
explicit if another value hasn't been passed. This fixes gnt-cluster
modify changing the master_netdev to xen-br0 if called without that
argument.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Run pylint over QA code too

Right now, the QA code is not covered by pylint, and this shows at
least one low-impact bug.

This patch does the necessary changes to make QA pylint-clean, and the
changes the makefile to run pylint for it.

Notable changes:

- qa_utils.GenericQueryTest: randfields was not used at all, and my
  belief is that it was indented to be used in order not to modify the
  input list; so I replaced randfields with fields, so we only shuffle
  the our local copy
- qa_node.TestOutOfBand was using it's own copy of AcquireNode(), so I
  replaced it with the existing version
- qa_os: was using 'dir' in a couple of places, replaced with dirname

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

QA: use a persistent SSH connection to the master

The recent additions to QA (many more tests) make QA slow if the
machine on which the QA runs is not very close to the tested nodes —
or in general, when the SSH handhaske is costly.

We discussed before about using a persistent connection, and here is
the patch that implements it. On a very small QA (very very small), it
cuts down a lot of time (almost half), so it should be useful even for
a full QA.

I've also thought about changing from external ssh to paramiko, but I
estimated that it would be more work to correctly interleave the IO
from the remote process than just running a background SSH.

Also note that yes, the global dict is ugly, but I don't know of
another simple way to implement this.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

QA: Fix duplicated OOB tests

Patch f55312bd added the OOB tests to TestClusterVerify, which is not
actually a test for cluster verify, but a runner for cluster verify
that is called multiple times, for each instance type, etc. This led
to running the OOB commands multiple times, which is painful
especially as this is a slow test.

The patch moves this to a separate test, that is run only once.

Furthermore, the way that data files are copied around is very
inefficient: touch + mv + chmod + mv + rm for each node (5 times
number of nodes), whereas it could be simply: touch on master, chmod
on master, cluster copyfile, chmod on master, cluster copyfile,
cluster command rm, i.e. only 5 fixed ssh calls to the master. The
code is changed as such, for increased speed.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add ability to retain specified fds open in RunCmd

Passing tap devices to KVM as file descriptors requires that the respective
file decriptors remain open during utils.RunCmd execution. To this direction,
we add a “noclose_fds” keyword argument to utils.RunCmd, accepting a list of
file descriptors to keep open. The actual fd handling is implemented in
_RunCmdPipe and _RunCmdFile using subprocess.Popen's “preexec_fn”[1],
since subprocess.Popen provides no other way to selectively handle fds.

A small modification is also made to test/ganeti.utils_unittest.py to comply
with _RunCmdPipe's new API and a new unit test is added to test the selective
fd retention functionality.

[1] “If preexec_fn is set to a callable object, this object will be called in
     the child process just before the child is executed. (Unix only)”
    Subprocess documentation

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add tests for objects.Instance

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

objects: Fix off-by-one in error message

For an instance with a single disk the message should be “… disks 0 to
0”, not “… 0 to 1”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

utils: Cleanup and sort wildcard imports

Also convert the per-line pylint instructions to one per-module
instruction.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

utils: Drop RemovePidFile

It's equivalent to a simple RemoveFile now.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

utils: Move NewUUID to utils.io

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

utils: Move more text-related code

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

utils: Move process-related code into separate file

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

utils: Move ResetTempfileModule to wrappers module

It's not exactly a wrapper, but this seemed like the best place.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

utils: Move RunningTimeout to utils.algo

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>