ganeti-local
12 years agoHypervisor: allow ballooning of instance memory
Guido Trotter [Mon, 28 Nov 2011 14:17:40 +0000 (14:17 +0000)]
Hypervisor: allow ballooning of instance memory

This new hypervisor call sets the memory of an instance to a new value,
through ballooning.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAccept both PUT and POST in noded
Iustin Pop [Fri, 21 Oct 2011 21:51:39 +0000 (23:51 +0200)]
Accept both PUT and POST in noded

Currently, noded requires PUT, even though the semantics of the RPC
calls do not match a PUT. We change the code accept both PUT and POST,
with the intention to remove the PUT support in a later version.

On the client side, the RPC code is changed to send POST requests.

Additionally, we add a message to the HttpBadRequest exception to make
clear the failure mode (not seeing any error message was what made me
send this patch…). This was the only description-less use of this
exception, by the way.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoRework instance rename QA tests
Iustin Pop [Mon, 16 Jan 2012 16:27:14 +0000 (17:27 +0100)]
Rework instance rename QA tests

After looking again at the QA speeds, it turned out that instance
rename was run 12 times via the command line and 12 times via RAPI;
outside of start/stop which we need for maintenance during QA, this is
the most used test.

This patch does a few changes with regard to instance rename:

- it changes rename functions to rename the instance back to the
  original name
- which allows us not to run said functions twice, A→B and then B→A

This brings down the number of rename tests from 12 to 8 (for both
cmdline and RAPI).

Furthermore, the CLI rename test was doing too many actual instance
renames (at Ganeti level), so we rework it to be simpler (brings down
the average duration from ~27s to ~21s), and we reorganise the run of
various tests such that we do fewer instances start/stop tests (10
instead of 14).

Overall we only gain about 3 minutes (~7%), but still it's a speedup.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAnother small optimisation in test data generation
Iustin Pop [Sun, 15 Jan 2012 23:48:11 +0000 (00:48 +0100)]
Another small optimisation in test data generation

It seems that QuickCheck will happily generate huge data structures,
if that's what one requests. For this test, where we ask for [[Char]],
it could happen that we get a huge number of chars, so let's limit
them to some (arbitrary, but low) limits. This doesn't "ruin" the
test, since the implementation of sepSplit/commaJoin is not
size-based.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoGenerate tags using the helper in Node tests
Iustin Pop [Sun, 15 Jan 2012 01:02:22 +0000 (02:02 +0100)]
Generate tags using the helper in Node tests

Otherwise we test very big lists (slow) without any advantage in doing
so.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoCombine two Node unittests
Iustin Pop [Sun, 15 Jan 2012 01:00:59 +0000 (02:00 +0100)]
Combine two Node unittests

These two tests are very similar, so let's run them together instead
of generating the data twice. Also improve the test a bit.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd unit-tests for --help/--version as common options
Iustin Pop [Sun, 15 Jan 2012 00:26:36 +0000 (01:26 +0100)]
Add unit-tests for --help/--version as common options

After reorganizing the way we process options, we can finally add a
test that all program personalities accept --help and --version and
that they exit early for these two.

This trivial patch has a higher than expected impact on the coverage,
as the use of short options finally enforces evaluation of the short
option names, thus it affects more than just the help/version options.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoSplit the list of program personalities into a module
Iustin Pop [Sat, 14 Jan 2012 23:55:58 +0000 (00:55 +0100)]
Split the list of program personalities into a module

… a mini-module, actually. This is only done for unit-testing, as we
don't want to duplicate the personalities list in the tests.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoParse options in htools.hs, not in each program
Iustin Pop [Sat, 14 Jan 2012 23:44:31 +0000 (00:44 +0100)]
Parse options in htools.hs, not in each program

This patch moves the call to getArgs/parseOpts into htools.hs, as
opposed to the current situation where htools.hs only calls the main
file of each program, which in turn calls these two functions.

By doing it only once we remove some duplicate code and we also export
each program's options, which allows us to test some common behaviour
in the future.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd some very basic options test
Iustin Pop [Sat, 14 Jan 2012 23:26:37 +0000 (00:26 +0100)]
Add some very basic options test

Just tests that passing a string argument returns it correctly. We're
not interested in the flags themselves that much just that the
function itself parses some options at least correctly.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoSplit CLI.parseOpts into IO/non-IO parts
Iustin Pop [Sat, 14 Jan 2012 22:53:14 +0000 (23:53 +0100)]
Split CLI.parseOpts into IO/non-IO parts

This will allow unittesting the CLI options much more easily. The
patch also fixes an inconsistency: usage help was displayed with an
extra new line in the error case (but not in --help).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd test for CLI.parseYesNo
Iustin Pop [Sat, 14 Jan 2012 22:26:58 +0000 (23:26 +0100)]
Add test for CLI.parseYesNo

Also fix a tiny typo in the docstring.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFix bug in CLI.parseISpecString and add test
Iustin Pop [Sat, 14 Jan 2012 22:17:39 +0000 (23:17 +0100)]
Fix bug in CLI.parseISpecString and add test

'zip' finishes on the shortest list, so parseISpecString accepted any
string of 3 values or more, which is not good. Fix this and add test
for both successful and failed parsing.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a unittest for the 'Simu' backend
Iustin Pop [Sat, 14 Jan 2012 21:23:21 +0000 (22:23 +0100)]
Add a unittest for the 'Simu' backend

This only test successful parsing, but for this case it tests
reasonably well the resulting cluster properties, I think (coverage
is >70% for expressions).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFurther optimise instance test data generation
Iustin Pop [Sat, 14 Jan 2012 20:07:15 +0000 (21:07 +0100)]
Further optimise instance test data generation

Following on the lessons from this week (commit 7806125e), this patch
replaces generation of instances `suchThat` they are smaller than on
node with direct generation of instances with upper bounds on
memory/disk/cpu.

For a particular case I was investigating, this reduces the overall
duration of tests by almost 50%. I don't expect it to speed up this
nice in most cases, but it's too good an optimisation to pass up.

The patch also does a trivial simplification (if … then True else
False, d'oh).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a Makefile target for profiling targets
Iustin Pop [Sat, 14 Jan 2012 00:07:21 +0000 (01:07 +0100)]
Add a Makefile target for profiling targets

This simplifies the building of profiling-enabled htools binaries;
there are two targets, one used for the initial two-stage build, and
one that can be used for incremental builds.

The patch adds a new internal variable for the htools/test binary,
which should also simplify custom builds for the non-profiling
targets.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agognt-instance: Adapt replace-disks to the new opcode
René Nussbaumer [Fri, 13 Jan 2012 12:56:59 +0000 (13:56 +0100)]
gnt-instance: Adapt replace-disks to the new opcode

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

12 years agoTLReplaceDisk: Add ipolicy checks
René Nussbaumer [Fri, 13 Jan 2012 12:44:17 +0000 (13:44 +0100)]
TLReplaceDisk: Add ipolicy checks

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

12 years agoFix very slow unit-test data generation in some cases
Iustin Pop [Fri, 13 Jan 2012 14:44:37 +0000 (15:44 +0100)]
Fix very slow unit-test data generation in some cases

Learning, learning. It turns out that for objects with even just five
fields, generating them and filtering can be costly, especially when
the filtering is nested (as it was in this case, arbitrary1 →
arbitrary2 + filter on arbitrary1 → arbitrary plus filter on
aribitrary2).

Changing the code to generate directly the good values speeds things
up significantly. Additionally, we also generate correctly unique
lists of disk templates (again, this should be a set, not a list,
but…).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFix one bad unit-test data generation case
Iustin Pop [Fri, 13 Jan 2012 14:43:37 +0000 (15:43 +0100)]
Fix one bad unit-test data generation case

It could happen that the node is generated with 0 disk available, in
which case the operation will fail due to disk issues, not CPU. Fix
this by using the recently-introduced genOnlineNode helper.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFurther fixes to instance policy validation
Iustin Pop [Fri, 13 Jan 2012 10:57:59 +0000 (11:57 +0100)]
Further fixes to instance policy validation

As a followup from "Remove extraneous check in policy creation", there
are more places where we build an ipolicy, and then manually check for
its validity. This is very bad style, as it duplicates the
verification code across many places.

This patch removes all such explicit checks (except for one in
cmdlib.py which is correct), and instead does a bit more validation in
the builder functions or in the actual dedicated verification
functions. It also fixes cluster init which used the wrong,
non-completed ipolicy (this was not detected before as we did call
check on it, but otherwise we ignored it), and fixes a too-strong
assert (due to the call chain, we first create the ipolicy from
cmdline params, and only then we fill it).

Finally, it removes an extraneous logging.info which I forgot from
debugging.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFix another data generation issue in tests
Iustin Pop [Mon, 9 Jan 2012 10:47:11 +0000 (11:47 +0100)]
Fix another data generation issue in tests

Another "good" seed value manages to generate nodes with identical
names, which of course makes the sets be of smaller value than the
lists. Avoid this issue by generating unique node names.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoRemove some dead code
Iustin Pop [Mon, 9 Jan 2012 10:38:50 +0000 (11:38 +0100)]
Remove some dead code

While wondering why Cluster.tryReloc is not unit-tested, I realised
that this is no longer uses. Apparently, last summer's new IAllocator
modes and related changes have silently deprecated this function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFix a few issues in unittest data generation
Iustin Pop [Mon, 9 Jan 2012 10:04:25 +0000 (11:04 +0100)]
Fix a few issues in unittest data generation

This patch fixes two ugly but non-critical issues:

- the getFQDN wrongly generated only the first component as
  limited-length name, the rest could be any length, and that makes
  the generated name bad for display purposes
- the random generator had real issues in finding an offline instance
  with the right properties when starting from certain seeds (found
  accidentally); this meant the test took 1 minute, instead of ~300ms,
  for the node_addSecOffline

Furthermore, for the same evil seed, the Container_findByName test
wasn't able to generate enough good data, so we change its
implementation to directly generate an uniques list, instead of
generating random lists and filtering the unique ones.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoExtend node evacuation unittests
Iustin Pop [Mon, 9 Jan 2012 08:41:13 +0000 (09:41 +0100)]
Extend node evacuation unittests

Currently they only check for change secondary, via iallocator's
relocate method. We add two separate unittests that checks all the
node evacuate modes (ChangeAll, ChangePrimary) and also the group
change request. All of these are done only for DRBD instances.

The patch also enhances an error message in Cluster.findBestAllocGroup
for invalid input data (guess how this was detected…).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a unit-test helper function
Iustin Pop [Thu, 5 Jan 2012 16:40:46 +0000 (17:40 +0100)]
Add a unit-test helper function

… instead of always using "printTestCase … False".

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFurther JSON unit-tests for various Types.hs types
Iustin Pop [Thu, 5 Jan 2012 16:16:56 +0000 (17:16 +0100)]
Further JSON unit-tests for various Types.hs types

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFurther unittests for ipolicy/ispec text serialisation
Iustin Pop [Thu, 5 Jan 2012 16:02:07 +0000 (17:02 +0100)]
Further unittests for ipolicy/ispec text serialisation

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoImplement saving/restoring of policies in Text backend
Iustin Pop [Thu, 5 Jan 2012 15:44:11 +0000 (16:44 +0100)]
Implement saving/restoring of policies in Text backend

This adds a new optional (so we're backwards-compat) section to the
cluster data so that we can reliably save/restore the policy data.

Note that different from the other sections, we save all policies
together and separately from the group data. I thought this is cleaner
than extending adding the policy data near each group, and a new
section just for the cluster policy.

This change also allows simplification of the unittest for cluster
create/serialize, although we would need a more complete test (right
now we only dump/load the defIPolicy or nullIPolicy).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a complex allocation/serialisation/load test
Iustin Pop [Thu, 5 Jan 2012 13:23:31 +0000 (14:23 +0100)]
Add a complex allocation/serialisation/load test

This is unfortunately a complex test that checks if:

- starting from a empty cluster
- allocating a few instances on it
- serialising it via the Text backend
- loading it back into internal data structures

we end up with the same thing. More precisely, it checks that the
cluster after dump/load is the same as after allocation.

While this is in the Text unittests, as the docstring saying, this
check more stuff than just the dump/load. It increases the overall
coverage by about 3%.

There were a few changes needed for this unittest: adding some Eq
instances for some data types, and fixing a cosmetic bug in
Cluster.iterateAlloc, where the instance indices were growing by 2
instead of by 1 in each round.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoGenerate clusters with unique node names in tests
Iustin Pop [Thu, 5 Jan 2012 13:18:46 +0000 (14:18 +0100)]
Generate clusters with unique node names in tests

Nothing breaks otherwise in the cluster routines, but serialisation
to/from text gets confused otherwise.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a help function for generating test nodes
Iustin Pop [Thu, 5 Jan 2012 13:17:07 +0000 (14:17 +0100)]
Add a help function for generating test nodes

This abstracts the concept of "give an sane, online node" in a single
place, rather than spread around the tests. We also change the
signature for isNodeBig for easier usage.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFix a couple of bugs in handling offline instances
Iustin Pop [Thu, 5 Jan 2012 11:01:45 +0000 (12:01 +0100)]
Fix a couple of bugs in handling offline instances

The addition of the offline state for instances has introduced a few
bugs related to their handling. The current code adds a helper
function for making the check whether an instances uses reserved
memory on the secondary (and uses it instead of just autoBalance),
and furthermore uses consistently the state for primary node
operations.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a helper function for 'not offline' instances
Iustin Pop [Thu, 5 Jan 2012 10:59:37 +0000 (11:59 +0100)]
Add a helper function for 'not offline' instances

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoUnittests for the instance policy checks on allocation
Iustin Pop [Wed, 4 Jan 2012 14:40:57 +0000 (15:40 +0100)]
Unittests for the instance policy checks on allocation

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoShow explicit message when hspace fails to alloc
Iustin Pop [Wed, 4 Jan 2012 10:46:05 +0000 (11:46 +0100)]
Show explicit message when hspace fails to alloc

Currently, in the human-readable mode, hspace doesn't show any
messages when no instances can be allocated in tiered-alloc mode. This
is confusing, so let's add an explicit message.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoObey instance policy in allocation routines
Iustin Pop [Wed, 4 Jan 2012 10:29:21 +0000 (11:29 +0100)]
Obey instance policy in allocation routines

This currently checks the minimum/maximum sizes and the disk
template. Unittests will come later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoSet group policy on nodes
Iustin Pop [Wed, 4 Jan 2012 10:12:18 +0000 (11:12 +0100)]
Set group policy on nodes

In order to be able to use the group policy during allocation, we have
a few choices:

- pass the group policy as a separate argument to all involved
  functions, or
- add the policy as an attribute to nodes, or
- add the group (not only the group index) as an attribute to nodes

The first and last one would change more than wanted for this patch,
so I went with the second one; it will allow easily checking the
target policy at runtime (based on the instance's primary node), with
the only downside that without a node, we can't check the policy.

Note that the first option also would complicate the allocation
routines, which examine (big) lists of node/node pairs, so adding the
group to this would complicate things even more.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoA trivial optimisation in Loader.mergeData
Iustin Pop [Fri, 30 Dec 2011 16:20:28 +0000 (17:20 +0100)]
A trivial optimisation in Loader.mergeData

During loading, we can eliminate some intermediate data
structures. This only matters for big clusters, and even there it's
less than 1% less memory allocated, but still, it's a no-brainer.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoTrivial performance optimisation
Iustin Pop [Fri, 30 Dec 2011 15:49:00 +0000 (16:49 +0100)]
Trivial performance optimisation

This is not in the hot-path, but still, no sense in doing O(n) instead
of O(1).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd ipolicy attribute to group objects
Iustin Pop [Fri, 30 Dec 2011 12:18:47 +0000 (13:18 +0100)]
Add ipolicy attribute to group objects

This is straightforward; as before, so far the text and simu backends
have no way to use a non-default ipolicy.

Furthermore, we do some unit-test preparation for future changes
(these seem a no-op for now, but they are needed for later patches).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoEnable Eq instances for TH-built objects
Iustin Pop [Fri, 30 Dec 2011 12:18:11 +0000 (13:18 +0100)]
Enable Eq instances for TH-built objects

This is needed for the next patch.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoSwitch hspace to read the disk template from the cluster
Iustin Pop [Thu, 29 Dec 2011 16:39:32 +0000 (17:39 +0100)]
Switch hspace to read the disk template from the cluster

The default disk template is the first enabled template.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoRead the disk templates part of the ipolicy
Iustin Pop [Thu, 29 Dec 2011 15:48:32 +0000 (16:48 +0100)]
Read the disk templates part of the ipolicy

The default value is badly defined (hardcoded defaults)…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoMove DiskTemplate definition around
Iustin Pop [Thu, 29 Dec 2011 15:48:06 +0000 (16:48 +0100)]
Move DiskTemplate definition around

This is needed since we'll need the DiskTemplate definition in the
IPolicy one.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd new disk_templates parameter to instance policy
Iustin Pop [Thu, 29 Dec 2011 15:33:32 +0000 (16:33 +0100)]
Add new disk_templates parameter to instance policy

This is a bit more complex patch, as it requires changing the
assumption that all keys in the policy dict points to values that are
themselves dicts. Right now we introduce an assumption that any
non-dicts are lists, we'll see in the future if this holds or whether
we need more complex type checking (manual, yay Python).

The patch also does some trivial style changes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoRemove extraneous check in policy creation
Iustin Pop [Thu, 29 Dec 2011 15:24:33 +0000 (16:24 +0100)]
Remove extraneous check in policy creation

The values are already checked in CreateIPolicy, no need to manually
check them again.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoMove the instance specs options to cli.py
Iustin Pop [Thu, 29 Dec 2011 15:20:18 +0000 (16:20 +0100)]
Move the instance specs options to cli.py

Currently these are defined twice, instead of a single time in
cli.py. Also adds the new disk_templates option to the common block,
even though it's not yet used.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a new disk-template ipolicy option
Iustin Pop [Thu, 29 Dec 2011 15:19:37 +0000 (16:19 +0100)]
Add a new disk-template ipolicy option

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a new CLI option type 'list'
Iustin Pop [Thu, 29 Dec 2011 15:18:11 +0000 (16:18 +0100)]
Add a new CLI option type 'list'

This simply splits the value in the option parser, instead of needing
to do it in the client code.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFix handling of errors from InstancePolicy.Check...
Iustin Pop [Thu, 29 Dec 2011 15:15:13 +0000 (16:15 +0100)]
Fix handling of errors from InstancePolicy.Check...

This code raises a configuration error, but we need to transform it
into a prereq error (or possibly exec error), depending on when we
call this function.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoSwitch hspace defaults to the cluster policy
Iustin Pop [Thu, 29 Dec 2011 11:29:37 +0000 (12:29 +0100)]
Switch hspace defaults to the cluster policy

This changes from the current hardcoded defaults to the cluster
policy. The command line options now override the defaults from the
cluster, and the tiered spec mode is always enabled.

Also fixes a tiny typo in the man page (together with the man page
updates).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd a helper function converting ispecs to rspecs
Iustin Pop [Thu, 29 Dec 2011 10:53:42 +0000 (11:53 +0100)]
Add a helper function converting ispecs to rspecs

The rspec type is simpler than the ispec one; most likely it should be
deprecated later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAbstract creation of instance from a spec
Iustin Pop [Thu, 29 Dec 2011 10:49:24 +0000 (11:49 +0100)]
Abstract creation of instance from a spec

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoLoad cluster ipolicy via Rapi
Iustin Pop [Thu, 29 Dec 2011 10:22:13 +0000 (11:22 +0100)]
Load cluster ipolicy via Rapi

This requires changing from querying the /tags resource to the /info
resource.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoUpdate memory/maxmem reading in Rapi backend
Iustin Pop [Thu, 29 Dec 2011 09:43:42 +0000 (10:43 +0100)]
Update memory/maxmem reading in Rapi backend

Recent changes to the instance beparams have replaced memory with
maxmem in Rapi bulk queries. Until this is either reverted (for
backwards compat) or we decide to go ahead with only maxmem, we change
the backend to read both; this only affects the "instance down" code
path (where ``oper_ram`` is missing).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoLoad cluster ipolicy via Luxi
Iustin Pop [Fri, 23 Dec 2011 11:01:51 +0000 (12:01 +0100)]
Load cluster ipolicy via Luxi

Also show it in hbal's verbose output (helpful for debugging).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoExtend ClusterData with the cluster instance policy
Iustin Pop [Fri, 23 Dec 2011 10:38:38 +0000 (11:38 +0100)]
Extend ClusterData with the cluster instance policy

This attribute is always initialised to the default, and is not (yet)
read/saved in the various backends.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd default ipolicy declarations
Iustin Pop [Fri, 23 Dec 2011 10:22:06 +0000 (11:22 +0100)]
Add default ipolicy declarations

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd support for RE patterns to convert constants
Iustin Pop [Fri, 23 Dec 2011 10:13:34 +0000 (11:13 +0100)]
Add support for RE patterns to convert constants

This is a trivial conversion.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd support for lists/frozensets in convert-constants
Iustin Pop [Fri, 23 Dec 2011 09:43:50 +0000 (10:43 +0100)]
Add support for lists/frozensets in convert-constants

Unfortunately, we only support lists of simple types, and not even
lists of tuples. If we actually needed those, it would be possible to
implement them, with a bit more complexity in the converter.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd support for tuples in convert-constants
Iustin Pop [Thu, 22 Dec 2011 17:16:24 +0000 (18:16 +0100)]
Add support for tuples in convert-constants

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoMore improvements to convert-constants
Iustin Pop [Thu, 22 Dec 2011 16:39:35 +0000 (17:39 +0100)]
More improvements to convert-constants

This prepares for tuple and other conversions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoImprove convert-constants to handle dictionaries
Iustin Pop [Thu, 22 Dec 2011 16:32:15 +0000 (17:32 +0100)]
Improve convert-constants to handle dictionaries

The two main drawbacks for convert-constants are the fact that it
can't handle sets/frozensets (mainly due to the fact that I don't know
how useful this would be to the Haskell code) and that it cannot
export dictionaries.

To fix the second case, the current patch changes the code to support
flattening (potentially nested) dictionaries into single name
space. Yes, this could generate conflicts, but they will be detected
at compile time.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAdd object definitions for the ispec and ipolicy
Iustin Pop [Thu, 22 Dec 2011 15:27:05 +0000 (16:27 +0100)]
Add object definitions for the ispec and ipolicy

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoStop exporting JSON functionality from Utils.hs
Iustin Pop [Thu, 22 Dec 2011 14:58:30 +0000 (15:58 +0100)]
Stop exporting JSON functionality from Utils.hs

This completes the Utils/JSON split started in commit f047f90f. The
import graph should be cleaner now.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoMore reshuffling of code
Iustin Pop [Thu, 22 Dec 2011 14:50:23 +0000 (15:50 +0100)]
More reshuffling of code

Following the split Types/BasicTypes, we can remove the last
JSON-related stuff from Utils.hs, and do some more cleanup.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agohtools: split parts of HTools/Types.hs into BasicTypes.hs
Iustin Pop [Mon, 21 Nov 2011 11:44:30 +0000 (12:44 +0100)]
htools: split parts of HTools/Types.hs into BasicTypes.hs

The 'Result' type is common and:

- might be used outside of HTools-specific code too
- is better split as we need these basic types for building the more
  complex ones in Types.hs

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoFix acquisition of node lock in LUInstanceGrowDisk
Constantinos Venetsanopoulos [Wed, 11 Jan 2012 15:03:18 +0000 (17:03 +0200)]
Fix acquisition of node lock in LUInstanceGrowDisk

Ensure node level locks are recalculated properly
in LUInstanceGrowDisk.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

12 years agoFix cluster ispecs on upgrade
Iustin Pop [Thu, 22 Dec 2011 10:38:37 +0000 (11:38 +0100)]
Fix cluster ispecs on upgrade

Cluster-level parameters must *not* be empty. Plus two more minor
changes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoBetter max ispec defaults
Iustin Pop [Thu, 22 Dec 2011 10:38:04 +0000 (11:38 +0100)]
Better max ispec defaults

Instead of being equal to the minimum ones, these are reusing current
constants.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoRename optISpec to optStdSpec
Iustin Pop [Thu, 22 Dec 2011 10:09:50 +0000 (11:09 +0100)]
Rename optISpec to optStdSpec

More consistency with the tiered allocation mode.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoChange how hspace reads cmdline args for std specs
Iustin Pop [Thu, 22 Dec 2011 10:04:50 +0000 (11:04 +0100)]
Change how hspace reads cmdline args for std specs

For the standard specs, hspace currently takes separate options for
the memory, disk and VCPUs. For the tiered specs, which were
introduced later, it takes a single option with all three values
combined.

This patch adds a backward-incompatible change to the standard spec,
basically moving it to the same single option format, which is a bit
more simple (both in the code and on the command line). This is
"needed" for future changes, where the command line will just
overwrite what we get from cluster, instead of setting the starting
point.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agoAbstract tiered specs command line parsing
Iustin Pop [Wed, 21 Dec 2011 16:52:45 +0000 (17:52 +0100)]
Abstract tiered specs command line parsing

This will be used in the future for the standard specs too, so let's
abstract it away.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

12 years agocmdlib._ComputeMinMaxSpec: Add unittest for this function
René Nussbaumer [Tue, 10 Jan 2012 13:47:20 +0000 (14:47 +0100)]
cmdlib._ComputeMinMaxSpec: Add unittest for this function

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agocmdlib: Rename _CheckMinMaxSpecs to _ComputeMinMaxSpec
René Nussbaumer [Tue, 10 Jan 2012 13:18:16 +0000 (14:18 +0100)]
cmdlib: Rename _CheckMinMaxSpecs to _ComputeMinMaxSpec

This is to reflect the fact that _Check functions do raise Op*Error
which is not the case for _CheckMinMaxSpec.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agocmdlib: _VerifyInstancePolicy has been replaced and is dead code
René Nussbaumer [Tue, 10 Jan 2012 12:45:03 +0000 (13:45 +0100)]
cmdlib: _VerifyInstancePolicy has been replaced and is dead code

This removes this method

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agoutils.ResetTempfileModule: Improve performance
Michael Hanselmann [Tue, 10 Jan 2012 14:20:05 +0000 (15:20 +0100)]
utils.ResetTempfileModule: Improve performance

This function is called for after every fork (e.g. for handling an RPC
request). With the changes in this patch generating the next random
filename is about 30% faster.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

12 years agoLUInstanceCreate: Release node resource lock
Michael Hanselmann [Tue, 10 Jan 2012 12:38:03 +0000 (13:38 +0100)]
LUInstanceCreate: Release node resource lock

… after running iallocator. Otherwise an assertion later in the code
would fail (for a good reason).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

12 years ago_CheckTargetNodeIPolicy: instance.primary_node is not objects.Node
René Nussbaumer [Mon, 9 Jan 2012 14:38:50 +0000 (15:38 +0100)]
_CheckTargetNodeIPolicy: instance.primary_node is not objects.Node

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agocmdlib: node.group is not a nodegroup object
René Nussbaumer [Mon, 9 Jan 2012 09:43:43 +0000 (10:43 +0100)]
cmdlib: node.group is not a nodegroup object

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agoLUInstanceCreate: self.disks is a dict not a disk object
René Nussbaumer [Fri, 6 Jan 2012 15:44:48 +0000 (16:44 +0100)]
LUInstanceCreate: self.disks is a dict not a disk object

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agoqa-sample.json: Fix a typo
René Nussbaumer [Fri, 6 Jan 2012 14:00:32 +0000 (15:00 +0100)]
qa-sample.json: Fix a typo

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agoFix wrong variable name
Iustin Pop [Tue, 10 Jan 2012 11:58:27 +0000 (12:58 +0100)]
Fix wrong variable name

Commit bc5d0215 added support for disk params, but due to copy-paste
it tries to enforce the hvparams into disk params values, leading to:

$ gnt-cluster modify -H kvm:initrd_path=/boot/initrd-2.6-kvmU
Parameter Error: Unknown parameter 'initrd_path'

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>

12 years agoInstanceStartup: use overridden beparams in prereq
Guido Trotter [Mon, 12 Dec 2011 15:45:32 +0000 (15:45 +0000)]
InstanceStartup: use overridden beparams in prereq

Without doing this memory checks happen on instance default parameters
rather than any overridden ones.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agogen-coverage: Change filter
Michael Hanselmann [Mon, 9 Jan 2012 14:12:50 +0000 (15:12 +0100)]
gen-coverage: Change filter

It is not necessary to run a Python script to generate the filter, which
didn't work very well anyway on systems with modules in
/usr/share/pyshared.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

12 years agorpc: More unittests
Michael Hanselmann [Fri, 6 Jan 2012 15:01:49 +0000 (16:01 +0100)]
rpc: More unittests

- Fix: Don't check st_atime for temporary file for upload tests
- Add tests for five encoders

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

12 years agoDocument "gnt-node add" behavior in case of failure
Bernardo Dal Seno [Thu, 5 Jan 2012 14:05:29 +0000 (15:05 +0100)]
Document "gnt-node add" behavior in case of failure

Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

12 years agocmdlib: Adapt LUClusterVerifyGroup for instance policy
René Nussbaumer [Tue, 13 Dec 2011 09:39:23 +0000 (10:39 +0100)]
cmdlib: Adapt LUClusterVerifyGroup for instance policy

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agognt-node: Add instance policy to migrate
René Nussbaumer [Tue, 13 Dec 2011 09:35:40 +0000 (10:35 +0100)]
gnt-node: Add instance policy to migrate

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agognt-backup: Adding force option to import
René Nussbaumer [Fri, 16 Dec 2011 13:51:29 +0000 (14:51 +0100)]
gnt-backup: Adding force option to import

This makes it possible to bypass the instance policy

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

12 years agognt-instance: Adding instance policy to add
René Nussbaumer [Wed, 14 Dec 2011 13:39:49 +0000 (14:39 +0100)]
gnt-instance: Adding instance policy to add

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agognt-instance: Adding verification of instance policy to modify
René Nussbaumer [Fri, 16 Dec 2011 14:01:30 +0000 (15:01 +0100)]
gnt-instance: Adding verification of instance policy to modify

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agognt-instance: Adding instance policy to move
René Nussbaumer [Tue, 13 Dec 2011 09:34:27 +0000 (10:34 +0100)]
gnt-instance: Adding instance policy to move

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agognt-instance: Adding instance policy to migrate
René Nussbaumer [Tue, 13 Dec 2011 09:31:04 +0000 (10:31 +0100)]
gnt-instance: Adding instance policy to migrate

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agognt-instance: Adding instance policy to failover
René Nussbaumer [Tue, 13 Dec 2011 09:12:49 +0000 (10:12 +0100)]
gnt-instance: Adding instance policy to failover

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agocmdlib: Adding helper for instance policy
René Nussbaumer [Tue, 13 Dec 2011 08:38:47 +0000 (09:38 +0100)]
cmdlib: Adding helper for instance policy

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agorpc.ConfigRunner: Fix uploading file
Michael Hanselmann [Fri, 6 Jan 2012 12:29:41 +0000 (13:29 +0100)]
rpc.ConfigRunner: Fix uploading file

Commit 601dfcb made some changes to the default encoders. This made
“ConfigRunner.call_upload_file” fail due to a missing encoder. This
patch applies the necessary changes to ConfigRunner and updates the
unittest for “call_upload_file”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

12 years agoUpdate GrowDisk docstring
Guido Trotter [Thu, 5 Jan 2012 15:55:40 +0000 (15:55 +0000)]
Update GrowDisk docstring

It forgot about one argument.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

12 years agoMerge branch 'devel-2.5'
Guido Trotter [Fri, 6 Jan 2012 11:38:29 +0000 (11:38 +0000)]
Merge branch 'devel-2.5'

* devel-2.5:
  KVM: support version reported by 1.0
  doc/admin: Clarify archived jobs

Conflicts:
NEWS: trivial

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>