Helga Velroyen [Thu, 6 Jun 2013 14:22:11 +0000 (16:22 +0200)]
Use hvparams in GetInstanceInfo
This patch extends the GetInstanceInfo function of the
hypervisors by an hvparams parameters. This leads to
an extension of the instance_info RPC call and the
adjustmend of all classes using this call.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Thu, 6 Jun 2013 13:08:38 +0000 (15:08 +0200)]
backend.py: remove fallback implementation of GetNodeInfo
This removes the fallback implementation of the backend's
GetNodeInfo function to the old implementation. By now,
all calling code is adjusted to the new hvparams parameter
and thus, the old implementation should not be necessary
anymore.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Thu, 6 Jun 2013 11:45:22 +0000 (13:45 +0200)]
Adjust all callers of RPC 'call_node_info'
This patch contains all adjustments necessary to code that
uses the 'call_node_info' call. The adjustment is necessary
because the call's signature changed from receiving a list
of hypervisor names to a list of tuples
(hypervisor_name, hvparams).
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Thu, 6 Jun 2013 11:40:10 +0000 (13:40 +0200)]
Add hvparams to RPC call 'node_info'
This patch adds the hvparams parameter to the RPC call
'node_info'. It also adjusts the related code in noded.py
and Query/Node.hs
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Thu, 6 Jun 2013 08:51:21 +0000 (10:51 +0200)]
backend.py: use hvparams in GetNodeInfo
This patch extends the GetNodeInfo function of the backend
module by a hvparams parameter. For now, it contains a
fallback solution for when the parameter is None. This
fallback will be removed in a later patch of this series.
It was necessary to be abled to gradually adjust all
calling code without breaking everything at once.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Thu, 6 Jun 2013 08:38:56 +0000 (10:38 +0200)]
hypervisors: add hvparams to GetNodeInfo
This patch extends the GetNodeInfo function of the
hypervisors by a hvparams parameter. The parameter
is currently only used by the xen hypervisor to determine
which xen toolstack ('xm' or 'xl') to use.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Thu, 6 Jun 2013 12:21:07 +0000 (14:21 +0200)]
hv_xen.py: renaming residual 'xm' occurrences
Some methods still refered to 'xm', which is inaccurate now
that xen can also be run with 'xl'. No functional changes
otherwise.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Wed, 5 Jun 2013 15:15:49 +0000 (17:15 +0200)]
Watcher: Use hvparams when listing instances
This patch makes the watcher use the hvparams from ssconf
to retrieve the list of instances from the hypervisors.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Wed, 5 Jun 2013 15:05:19 +0000 (17:05 +0200)]
Include hvparams in ssconf files
The watcher needs access to the cluster's hypervisor
parameters. This patch extends the ssconf infrastructure
by adding a hvparams file for each hypervisor. The format
of the hvparams file is a simple key=value format.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Wed, 5 Jun 2013 09:08:07 +0000 (11:08 +0200)]
Use hvparams in instance migration
This patch makes use of hvparams on instance migration to
determine the list of instance before the migration. This way, in
xen, the xl/xm choice is respected.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 7 Jun 2013 08:28:52 +0000 (10:28 +0200)]
Verify: node info and instance list
This patch enhances node verification by:
- Factoring out the verification of the node info
and the instance list to increase testability.
- Make both of them use the hvparams parameter.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Tue, 4 Jun 2013 15:24:49 +0000 (17:24 +0200)]
Verify: xen toolstack, hypervisor and hvparams
This patch extends the node verification by:
- Adding a check for the xen toolstack when the hypervisor
is verified.
- Factoring out the hypervisor verification in a
subfunction to increase testability.
- Factoring out the hvparams verification in a
subfunction for the same reason.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 24 May 2013 15:56:17 +0000 (17:56 +0200)]
rpc / noded: add hvparams to instance_list call
The 'instance_list' call needs to be extended to accept
hvparams. This is one of the few RPCs where it is
necessary, because there is no instance given whose
hvparams could be reused.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 24 May 2013 16:07:58 +0000 (18:07 +0200)]
backend.py: use instance's hvparams whenever possible
The backend module's function 'ListInstances' is called
within the module several times. In many cases, it is
possible to reuse the instance's hvparams for the
'ListInstances' call and thus avoiding having to extend
RPC calls.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Wed, 5 Jun 2013 09:20:02 +0000 (11:20 +0200)]
backend.py: GetInstanceListForHypervisor
In same cases, the call 'GetInstanceList' is only done for
one hypervisor. In those cases, it is more convenient to
not provide the full collection of hypervisor parameters
for all hypervisors, but only the ones for this hypervisor.
This patch factors out this particular case.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 24 May 2013 15:10:31 +0000 (17:10 +0200)]
backend.py: hvparams in 'GetInstanceList'
Propagating the use of hvparams further up the calling
hierarchy, this patch makes the backend module use the
hvparams in 'GetInstanceList'. Unit tests provided.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 24 May 2013 13:38:11 +0000 (15:38 +0200)]
hv_xen.py: _RunXen and GetInstanceList use hvparams
Propagating the use of hvparams further up in the calling
hierarchy, the functions '_RunXen' and 'GetInstanceList'
use the given hvparams. Unit tests provided. Calling function
'StopInstance' adjusted in this patch as well.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 24 May 2013 14:20:36 +0000 (16:20 +0200)]
hv_xen.py: rename RunXmList to RunInstanceList
Since one can retrieve xen's instance list with both, 'xm'
or 'xl', the function name 'RunXmList' is no longer
appropriate. Renaming it to 'RunInstanceList'. No
functional changes otherwise.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 24 May 2013 13:55:31 +0000 (15:55 +0200)]
hv_xen.py: rename GetXmList to GetInstanceList
Since one can retrieve xen's instance list via both 'xm'
or 'xl', it is no longer appropriate to call the function
'GetXmList'. This patch renames it to 'GetInstanceList'.
Other than that, there are no functional changes.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 24 May 2013 11:10:02 +0000 (13:10 +0200)]
hv_xen._GetCommand: retrieve xen command from hvparams
This patch adds a (so far optional) hvparams parameter
to the '_GetCommand' function. This dictionary is used
to retrieve the xen command (xm or xl). It is optional
for now to make the refactoring possible without breaking
everything at once.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Fri, 24 May 2013 11:03:13 +0000 (13:03 +0200)]
Unit tests for hv_xen._GetCommand
This patch adds unit tests for the current state of the
'GetCommand' method. The tests are rather trivial, but
having tests before starting to refactor seems to be
a good idea.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Thu, 16 May 2013 11:58:37 +0000 (13:58 +0200)]
Add xen command (xm, xl) to hv parameters
This patch adds the xen command (xm, xl) as hypervisor
parameter to the constants and adds validation to
the xen hypervisors.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helga Velroyen [Tue, 4 Jun 2013 13:45:26 +0000 (15:45 +0200)]
Fixing docstring of hv_xen_unittest.py
The introductory comment of hv_xen_unittest.py referred to
the wrong hypervisor.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Klaus Aehlig [Tue, 11 Jun 2013 08:24:47 +0000 (10:24 +0200)]
Make SubmitOpCode expect options without print_jobid
As opposed to SubmitOrSend, the SubmitOpCode function is
called also by commands not supporting submit options. Hence
only inspect these options, if they are actually present.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Klaus Aehlig [Thu, 6 Jun 2013 14:10:49 +0000 (16:10 +0200)]
Add an option to print job id of submitted jobs on stdout
When submitting a job, the job id, prefixed by 'JobID: ', is output to
stderr. While this is easy to understand for humans, for scripting, it
is more convenient to have (additionally) the id on stdout (and only
the id). Add an option to provide this.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Klaus Aehlig [Thu, 23 May 2013 15:38:22 +0000 (17:38 +0200)]
Add new command gnt_job wait
This command tacitly waits for a job to finish. In that
way, job dependency can also be handled in shell scripts.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Klaus Aehlig [Mon, 10 Jun 2013 14:45:17 +0000 (16:45 +0200)]
Add type annotation to avoid monomorphism restriction
Even though we need the let-bound variable showMoves only
at type [(String, String)] -> IO (), it's most general type
would be (PrintfArg a, PrintfArg b) => [(a, b)] -> IO ().
This causes the monomorphism restriction apply to that binding,
which is a warning (and, thanks to -Werror, fatal) on some
ghc versions.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Klaus Aehlig [Wed, 5 Jun 2013 14:37:32 +0000 (16:37 +0200)]
add option --print-moves to hroller
If non-redundant instances are present in the cluster, hroller will
plan for them to move to other nodes while the group is rebooted.
This adds an option to also show this plan.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Bernardo Dal Seno [Thu, 6 Jun 2013 17:55:15 +0000 (19:55 +0200)]
New tests for hspace and exclusive storage
hspace is run on clusters with exclusive storage enabled.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Thu, 6 Jun 2013 16:04:54 +0000 (18:04 +0200)]
hspace prints info about spindles
Statistics about spindles are tracked. In human-readable output, spindles
are printed only when used (i.e., exclusive storage is enabled). For
machine-oriented output, they are always there.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Thu, 6 Jun 2013 12:12:12 +0000 (14:12 +0200)]
Add support for shrinking an instance spindles-wise
This makes tiered allocation in hspace work also with respect to spindles.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Thu, 6 Jun 2013 12:50:11 +0000 (14:50 +0200)]
Spindles become part of htools resource spec
Spindles are now part of resource spec. Instances get created with spindles
specified (which are just ignored when exclusive storage is disabled).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Thu, 6 Jun 2013 10:35:28 +0000 (12:35 +0200)]
htools cluster score takes spindles into account
When exclusive storage is enabled, spindles are used instead of disk space
to compute the cluster score.
Comments and variable names in computePDsk has been changed to match the
actual code.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Wed, 5 Jun 2013 15:44:34 +0000 (17:44 +0200)]
Update spindles when moving instances in htools
Spindles get updated, and errors raised when not enough free spindles
exist. No new error is raised when exclusive storage is disabled.
Unit tests included.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Wed, 5 Jun 2013 20:53:10 +0000 (22:53 +0200)]
Unit tests for htools and exclusive storage
The existing tests are run also on nodes with exclusive storage enabled. The
values for spindles and exclusive storage are set in a consistent way, for
both nodes and instances.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Tue, 4 Jun 2013 20:08:23 +0000 (22:08 +0200)]
Load complete instance disk information through LUXI
Information about size and spindles of all the disks of an instance is loaded
by the LUXI backend, instead of faking one equivalent big disk. In this way
instance policy checks are more accurate.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Tue, 4 Jun 2013 18:49:05 +0000 (20:49 +0200)]
Load node spindles data in htools
The data structure for nodes gets a new field for free spindles, and the
existing field for total spindles gets renamed to avoid identifying it with
the node parameter that had the same name. These fields get filled with
data from the live node, when exclusive storage is enabled.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Tue, 4 Jun 2013 16:38:11 +0000 (18:38 +0200)]
Refactor reading live data in htools
This simplifies different handling of individual items.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Tue, 4 Jun 2013 00:05:59 +0000 (02:05 +0200)]
Export node spindles
Node spindles (queried live) are exported through the LUXI, RAPI, and
iallocator interfaces.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Mon, 3 Jun 2013 16:58:43 +0000 (18:58 +0200)]
node_info RPC reports info about free spindles
node_info RPC can be used to get information about free/total spindles.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Tue, 4 Jun 2013 23:13:35 +0000 (01:13 +0200)]
Update htools text-backend documentation
Both instances and nodes have new fields.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Klaus Aehlig [Tue, 4 Jun 2013 16:11:39 +0000 (18:11 +0200)]
Use RpcResult.Warn where appropriate the cmdlib
To avoid repetitive code, make use of the functionality factored
out to the Warn method of RpcResult.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Klaus Aehlig [Tue, 4 Jun 2013 15:19:31 +0000 (17:19 +0200)]
Add a Warn function to RpcResult
At various places, LU code just wants to warn about
a failed rpc, but continue nevertheless. So factor out
that common checking, message formatting, and notifying.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michele Tartara [Mon, 3 Jun 2013 16:43:13 +0000 (12:43 -0400)]
Fix shadowing of library function
The "reads" field shadows a library function from Prelude. This commit
fixes the problem.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Wed, 29 May 2013 21:40:30 +0000 (21:40 +0000)]
Add shelltests for diskstats
Check that the behavior of running the diskstats data collector is as
expected.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Wed, 29 May 2013 21:08:14 +0000 (17:08 -0400)]
Let the monitoring daemon provide the diskstats collector
Add the collector to the list of those provided my the monitoring daemon.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Wed, 29 May 2013 20:29:36 +0000 (20:29 +0000)]
Add diskstats data collector
Add a new data collector responsible for gathering disk performance
statistics.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Wed, 29 May 2013 20:11:10 +0000 (20:11 +0000)]
Add constant with the path of the diskstats
This will be used by the diskstats storage collector.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Wed, 29 May 2013 20:09:22 +0000 (20:09 +0000)]
Add a CLI parameter for input files
For many data collectors it is useful (especially for testing) to have to
possibility to specify an input file.
This commit adds a generic option for doing that.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Tue, 28 May 2013 21:16:55 +0000 (17:16 -0400)]
Add unit tests for the Diskstats parser
Add one test parsing an actual /proc/diskstats file and one QuickCheck
test.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Tue, 28 May 2013 21:14:00 +0000 (17:14 -0400)]
Add function for generating arbitrary non-negative numbers
In many tests it is useful to have the possibility to easily generate
non-negative integer numbers.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Fri, 24 May 2013 11:53:08 +0000 (13:53 +0200)]
Factor out and rename testFile function
The testFile function used in the DRBD parser test file can actually be
useful in any case where a parser has to be tested.
This patch moves it to the TestCommon file, and renames it to a more meaningful
testParser.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Fri, 24 May 2013 08:40:19 +0000 (10:40 +0200)]
Fix erroneous docstring
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Thu, 23 May 2013 18:12:45 +0000 (20:12 +0200)]
Fix erroneously ordered files in Makefile.am
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Thu, 23 May 2013 17:53:19 +0000 (19:53 +0200)]
Add /proc/diskstats parser
Add a parser for interpreting the content of the /proc/diskstats file,
providing information about the state of the disks of the system.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Thu, 23 May 2013 16:18:41 +0000 (16:18 +0000)]
Add design for diskstats data collector
This patch adds the design for the lowest level storage collector,
gathering data about disk usage statistics.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Michele Tartara [Thu, 23 May 2013 12:19:33 +0000 (12:19 +0000)]
Modify storage collector design
Specify better how the storage collectors will actually be implemented.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Thu, 30 May 2013 15:22:59 +0000 (17:22 +0200)]
Unit tests for spindles and exclusive storage in htools
Everything should work the same when exclusive storage is active, as
spindles are not yet handled as a resource. A new test has been added for
the interactions between spindles and policies.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Fri, 24 May 2013 16:43:15 +0000 (18:43 +0200)]
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are used
to check the instance policies (as outlined in design-partitioned.rst).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Wed, 29 May 2013 18:50:42 +0000 (20:50 +0200)]
Check the full instance specs in htools
Spindles and disk count are checked too. Existing functions have been
refactored, so common parts are not duplicated.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Sun, 26 May 2013 20:14:02 +0000 (22:14 +0200)]
Add spindles to instance disks in htools
A new data type is introduced for disks to store both size and spindles.
When available, spindles are filled with input data. Except for loading and
storing, spindles are ignored.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Mon, 27 May 2013 13:51:06 +0000 (15:51 +0200)]
Export instance disk spindles
Instance disk spindles can be queried via LUXI and RAPI, and are exported
through the allocator interface. This is a prerequisite for htools to
handle spindles.
The length of a RAPI query including all the instance fields now has become
~4400 characters. For this reason the server buffer has been increased.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Sun, 26 May 2013 19:23:18 +0000 (21:23 +0200)]
Load exclusive_storage in htools
The node parameter is loaded into the data structures. No behavior is yet
modified.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Tue, 28 May 2013 18:58:12 +0000 (20:58 +0200)]
New function to load JSON arrays of optional values
This will be needed to load spindles in some htools backends. Unit tests
provided.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Bernardo Dal Seno [Mon, 27 May 2013 22:12:19 +0000 (00:12 +0200)]
Add unit test for text backend + fix bug
Test serialization and deserialization of instances. Fix check of secondary
node.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Klaus Aehlig [Tue, 28 May 2013 13:42:22 +0000 (15:42 +0200)]
Test with non-redundant instances and several node-groups
The example cluster contains two node groups. The first contains three nodes,
with two non-redundant instances each. As each node has only capacity for three
instances, the nodes have to be rebooted individually. Additionally, there is
a separate node-group with additional of capacity, which, however, cannot be used,
as instances are supposed to be moved within a node-group only.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Klaus Aehlig [Fri, 31 May 2013 08:39:34 +0000 (10:39 +0200)]
Restrict instance moves in hroller to the same node group
When scheduling rolling reboots, hroller looks for nodes to evacuate
the non-redundant instances to. This is done by greedily moving
instances to other nodes that can take them, policy wise and capacity
wise. There is, however, another restriction to be taken into account:
instances should only be moved within a node group. This is achieved
by this patch.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Bernardo Dal Seno [Fri, 31 May 2013 13:36:42 +0000 (15:36 +0200)]
Merge branch 'stable-2.8' into master
* stable-2.8:
Document users-setup tool
Adjusting permissions after confd start
Ensure the queue socket has the right permissions
Update IAllocator interface documentation
Add NEWS entry for hail honoring networks
Add tests for network-aware allocation
Honor network connections in hail
Parse NIC data from allocation request in hail
Support group networks in Text backend
Parse node group networks
Update IAllocator interface documentation
Export connected networks to IAllocator
Re-activate previously active disks in watcher
Make the disks_active flag queryable
Consider disks_active in UpgradeConfig
Add disks_active to configuration
Fix release of instances in QA
Conflicts:
tools/cfgupgrade
tools/cfgupgrade was modified in stable-2.8 to remove a new parameter when
downgrading to 2.7, but this shouldn't go into the 2.9 version of
cfgupgrade.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Bernardo Dal Seno [Fri, 31 May 2013 11:42:50 +0000 (13:42 +0200)]
Move an item into the correct section of NEWS
The removal of '--no-lvm-storage' happened on the master/2.9 branch.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Bernardo Dal Seno [Fri, 31 May 2013 10:10:00 +0000 (12:10 +0200)]
Merge branch 'stable-2.7' into stable-2.8
* stable-2.7:
Document users-setup tool
Adjusting permissions after confd start
Ensure the queue socket has the right permissions
Update IAllocator interface documentation
Conflicts:
doc/iallocator.rst
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Klaus Aehlig <aehlig@google.com>
Christos Stavrakakis [Thu, 30 May 2013 10:34:50 +0000 (13:34 +0300)]
kvm: Use -uuid to set system UUID
Use kvm '-uuid' option to set guest's system UUID to instance's UUID.
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michele Tartara [Thu, 30 May 2013 20:42:25 +0000 (16:42 -0400)]
Document users-setup tool
The users-setup tool was added but had no documentation up to now. This commit
fixes this issue.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Bernardo Dal Seno [Wed, 29 May 2013 16:29:27 +0000 (18:29 +0200)]
Support spindles in move-instance
The script was failing when exclusive storage was active (bug introduced in
7c848a6a).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Helga Velroyen [Wed, 29 May 2013 09:40:35 +0000 (11:40 +0200)]
Adjusting permissions after confd start
This is a workaround for issue 477. Confd resets the
permissions of the query socket in a wrong way. This
patch fixes them after the start of confd.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Guido Trotter [Wed, 29 May 2013 05:07:30 +0000 (14:07 +0900)]
Ensure the queue socket has the right permissions
RAPI needs to be able to perform queries as well.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Thomas Thrainer [Thu, 23 May 2013 15:43:41 +0000 (17:43 +0200)]
Update IAllocator interface documentation
Update the missing documentation for the tags fields of node groups.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
(partially cherry picked from commit
2daca99b5be21ea4667637721ee2c65a14444520)
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Thomas Thrainer [Thu, 23 May 2013 07:46:42 +0000 (09:46 +0200)]
Add NEWS entry for hail honoring networks
The entry also mentions the required change in the IAllocator protocol.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Thomas Thrainer [Thu, 16 May 2013 12:56:26 +0000 (14:56 +0200)]
Add tests for network-aware allocation
hail-alloc-invalid-network defines a cluster with two nodegroups and an
allocation request which does not fit on any of the groups. Group 1 has
invalid disk-templates while Group 2 is not connected to the right
networks.
hail-alloc-restricted-network defines a cluster wih two nodegroups.
Nodegroup Group 1 is only used as last_resort group, but is chosen by
hail as only this group is connected to the right networks.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Thomas Thrainer [Thu, 16 May 2013 14:58:00 +0000 (16:58 +0200)]
Honor network connections in hail
Before trying to allocate nodes in node groups, node groups are now
filtered based on the networks they are connected to an the networks
which are required by the new instance.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Thomas Thrainer [Tue, 21 May 2013 06:08:04 +0000 (08:08 +0200)]
Parse NIC data from allocation request in hail
Add a NIC type and extend the Instance type by a list of NIC's. Parse
the NIC's in allocation requests and store them for now. Later patches
will make use of this field in order to ensure that the requested
instance is only placed in node groups wich are connected to those
networks.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Thomas Thrainer [Fri, 24 May 2013 08:03:02 +0000 (10:03 +0200)]
Support group networks in Text backend
The Text backend now parses network UUID (comma separated) and
serializes them in the same form.
The test data is adapted to the new format.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Thomas Thrainer [Fri, 17 May 2013 11:39:07 +0000 (13:39 +0200)]
Parse node group networks
Extend the Group by the network ids it is connected to. Adapt
the IAlloc backend such that the networks are parsed correctly.
This also required the adaption of test data.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Thomas Thrainer [Thu, 23 May 2013 07:39:33 +0000 (09:39 +0200)]
Update IAllocator interface documentation
Document the newly added network field of node groups, and also update
the missing documentation for the other fields of node groups.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Thomas Thrainer [Thu, 16 May 2013 11:44:10 +0000 (13:44 +0200)]
Export connected networks to IAllocator
IAllocators should not put instances on nodes in nodegroups which are
not connected to the network the instance should be connected to.
Therefore, export the networks a node group is connected to to the
IAllocator, so it can account for this restriction.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Klaus Aehlig [Mon, 27 May 2013 12:22:01 +0000 (14:22 +0200)]
Add generic daemon options support binding to interfaces
In some situations, it is desirable to bind one of our servers only to
a particular interface, like eth0, instead of an IP address. Therefore,
add an option -i to specify an interface; its address will then be bound
to.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Klaus Aehlig [Tue, 28 May 2013 12:32:15 +0000 (14:32 +0200)]
Update ganeti-rapi man page
ganeti-rapi supports a couple of options not yet mentioned in the
man page; mention them.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Klaus Aehlig [Tue, 28 May 2013 12:03:40 +0000 (14:03 +0200)]
Complete ganeti-noded synopsis line
The synopsis section of the ganeti-noded man page, certain options
were only mentioned in the description. Add then to the synopsis
line as well.
Signed-off-by: Klaus Aehlig <aehlig@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Thomas Thrainer [Wed, 15 May 2013 11:58:06 +0000 (13:58 +0200)]
Re-activate previously active disks in watcher
The watcher process previously (incorrectly) examined the admin_state
field to figure out if the disks of an instance should be activated
after a node reboot. This is changed to use the disks_active field of
instances, which always holds the correct information (even if the
instance should not be started, but only its disks activated).
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Thomas Thrainer [Wed, 15 May 2013 11:57:25 +0000 (13:57 +0200)]
Make the disks_active flag queryable
gnt-instance list now also supports the disks_active field.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Thomas Thrainer [Thu, 16 May 2013 09:13:24 +0000 (11:13 +0200)]
Consider disks_active in UpgradeConfig
The disks_active flag is updated when masterd reads the configuration.
Also, cfgupgrade now removes the disks_active flag during downgrades.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Thomas Thrainer [Wed, 15 May 2013 11:55:30 +0000 (13:55 +0200)]
Add disks_active to configuration
This flag tracks if the disks of an instace are supposed to be active.
That's the case when an instance is running or when its disks got
activated explicitly (and in a couple of other cases).
It will be used by watcher to re-activate disks after a node reboot.
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Thu, 23 May 2013 22:45:26 +0000 (00:45 +0200)]
Report missing spindles in the configuration
cfgupgrade and cluster-verify run a check for missing spindles in disk
configuration when exclusive storage is enabled. The check in cfgupgrade
may give false positives due to the complexity of getting the correct value
of exclusive_storage for each instance.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Fri, 24 May 2013 13:30:19 +0000 (15:30 +0200)]
Adapt striping to the requested number of spindles
Otherwise LVM may use a smaller number of PVs (spindles) to accommodate for
the default stripes.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Thu, 23 May 2013 21:44:02 +0000 (23:44 +0200)]
Spindles are mandatory with exclusive storage
If they are not specified in the command line an error is reported. Also,
disk creation would fail without them.
QA has been updated.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Thu, 23 May 2013 00:03:13 +0000 (02:03 +0200)]
gnt-cluster repair-disk-sizes repairs spindles too
When exclusive storage is active, any wrong or missing spindles information
in disks gets updated too.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Wed, 22 May 2013 23:21:19 +0000 (01:21 +0200)]
New RPC to get size and spindles of disks
This RPC replaces the existing one that only returned disk size.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Wed, 22 May 2013 23:19:24 +0000 (01:19 +0200)]
New BlockDev methods to get spindles
Two new methods are created to get the number of spindles, one alone, and
one together with size. For devices that don't support spindles, None is
returned.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Tue, 21 May 2013 22:33:08 +0000 (00:33 +0200)]
Attaching a logical volume builds a list of PVs
When an LV gets attached, the list of the PVs used by the LV is built. This
will be used to count spindles for exclusive_storage, but it could also be
useful to optimize disk growing and snapshotting.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Tue, 21 May 2013 20:27:58 +0000 (22:27 +0200)]
Refactor code for attaching a logical volume
The parsing of "lvs" output is moved into private methods. The code is
slightly more readable and testable. The split in two methods is useful
for the following patches. Unit tests for the new functions are
provided.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Bernardo Dal Seno [Fri, 24 May 2013 23:19:07 +0000 (01:19 +0200)]
Fix release of instances in QA
Also, fix some uses of keyword arguments for arguments with a default
value, as required by the style guide, and a typo.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>