Iustin Pop [Thu, 22 Mar 2012 12:34:53 +0000 (12:34 +0000)]
Rework exit model
While updating the confd code, I realised that we have _lots_ of
duplication in the exit model for the various programs.
So this patch attempts to abstract all the exits via a couple of new
functions; sorry for the somewhat big patch, but I hope the payoff is
worth the change: the actual exit conditions are much clearer.
Note that the patch (also) moves the exitIfBad function to Utils.hs,
since that is more logical.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 22 Mar 2012 14:55:01 +0000 (14:55 +0000)]
Fix out-of-tree builds
The new shell tests do not succeed out-of-tree, due to static paths
and other issues. This trivial patch fixes these issue, make distcheck
now passes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Wed, 21 Mar 2012 01:06:29 +0000 (01:06 +0000)]
Convert manual shell tests to shelltestrunner
This is more of a RFC… Basically most of the shell-based tests are
converted from exec+grep to shelltestrunner.
Things are not all fine and nice though:
- we have dependencies between tests, as some generate some data files
needed later; this is not nice, and we depend on serial execution in
testrunner
- we can still fail with no so nice messages in the offline-test
script (when we generate most of the data)
But overall, I think the tests are much nicer to
define/read/debug:
- each test is standalone, with the only dependency being an optional
input data file; this is much better than a single monolithic shell
script
- in case of failures, the failure is clearly shown by shell test,
both for exit code and stdout/stderr
- shelltest can run in --debug mode, where the exact details are shown
much better than the alternative of "set -x" for the shell script
Comments welcome!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Wed, 21 Mar 2012 16:48:07 +0000 (16:48 +0000)]
Add command line option for controlling syslog use
… and enable it in hconfd.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Wed, 21 Mar 2012 15:08:28 +0000 (15:08 +0000)]
Add support for syslog logging to Ganeti.Logging
Currently this is initialised to no from Daemon.hs, but will in the
future allow command-line options for controlling it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Wed, 21 Mar 2012 11:45:42 +0000 (11:45 +0000)]
Only build hconfd if --enable-confd was passed
A later, more complete patch, will allow selecting between either the
Python version or the Haskell version. This is just a temporary
solution to help building without all the needed Haskell libraries.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 20 Mar 2012 13:13:11 +0000 (13:13 +0000)]
Build test helpers that point to hpc-htools
Instead of using just shell constructs to run hpc-htools correctly
(i.e. HTOOLS=role htools/hpc-htools …), let's add some shell fragments
that do this for us.
This will ease the running of tests directly.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Tue, 20 Mar 2012 10:48:08 +0000 (11:48 +0100)]
Allow hail to read data from stdin
This patch makes hail treat '-' as denoting stdin, per the usual Unix
convention. This will help with testing.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 19 Mar 2012 17:02:56 +0000 (18:02 +0100)]
Update hconfd bind address handling
Instead of hardcoded IPv4 INADDR_ANY, this patch changes hconfd to use
either the any network for the configured cluster address family
(ipv4/ipv6), or whatever the user passes in via the --bind option.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 19 Mar 2012 17:01:52 +0000 (18:01 +0100)]
Add the bind-address option
This implements the same logic as the Python code: if the option is
not used, use the default appropriate for the cluster, otherwise try
to parse and use whatever was passed in.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 20 Mar 2012 17:57:33 +0000 (17:57 +0000)]
Add two utility functions
These both are work with/on the Result type, so we add them to
BasicTypes. The functions will be used as more generic versions of
some more specialised functions that are right now spread across the
modules.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 19 Mar 2012 16:47:45 +0000 (17:47 +0100)]
Add skeleton ssconf module
This currently has only one export function in it, which will be used
for future bind address functionality in daemons.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Alexander Schreiber [Tue, 20 Mar 2012 11:00:00 +0000 (12:00 +0100)]
Typo fix: s/aditional/additional/
Trivial fix for a typo in message output of LUInstanceSetParams
Signed-off-by: Alexander Schreiber <als@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 19 Mar 2012 15:41:22 +0000 (16:41 +0100)]
Fix exported constants
I "forgot" to run the unittests before commit :(
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 19 Mar 2012 13:27:08 +0000 (14:27 +0100)]
Export the AF_INET/AF_INET6 constants
These are needed to ensure that htools uses the same numeric values as
Python.
By the way, I'm not sure what's the best option: importing socket into
constants (this patch), or adding custom exports to
autotools/convert-constants.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 19 Mar 2012 10:04:56 +0000 (11:04 +0100)]
Rename htools-hpc to hpc-tools to fix laziness
This is a stupid/trivial patch. I didn't realise when adding
htools-hpc that this will "break" tab completion on htools; as in, I
will have to add / manually always when completing files on the
command line.
So to fix my laziness, let's rename htools-hpc to hpc-htools,
eliminating this completion conflict.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 19 Mar 2012 09:55:19 +0000 (10:55 +0100)]
htools: fix long version of --port for daemons
The dashes do not need to be in, if they are then the resulting option
is:
-p PORT ----port=PORT Network port (default: 1814)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 19 Mar 2012 09:45:51 +0000 (10:45 +0100)]
Add htools/rapi test directory to DIRS
Sorry!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 19 Mar 2012 09:17:39 +0000 (10:17 +0100)]
Update gnt-node evacuate man page/help text
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 16 Mar 2012 10:59:01 +0000 (11:59 +0100)]
Fix gnt-group --help display
Copy-paste mismatch :)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
René Nussbaumer [Fri, 16 Mar 2012 10:14:48 +0000 (11:14 +0100)]
hspace: Make use of the spindle_use
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 15 Mar 2012 17:48:47 +0000 (18:48 +0100)]
htools: Fix some compiler warnings
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 9 Mar 2012 14:47:04 +0000 (15:47 +0100)]
Add tests for custom text files
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 9 Mar 2012 14:20:39 +0000 (15:20 +0100)]
Add utilisation and instance selection tests
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 9 Mar 2012 14:03:13 +0000 (15:03 +0100)]
Enable pipefail mode for shell tests
These allow more accurate checks.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 9 Mar 2012 13:09:07 +0000 (15:09 +0200)]
Add RAPI shell-level unittest
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 9 Mar 2012 11:19:32 +0000 (13:19 +0200)]
Enhance the RAPI backend to accept file:// URLs
This will allow offline testing of this backend (except for the actual
curl part), also in the case where we didn't compile against curl.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Thu, 8 Mar 2012 21:05:15 +0000 (23:05 +0200)]
Add new hail (shell) tests
Using some hand-crafted files (manually modified from a real cluster
running master branch), we test that hail behaves as we expect:
doesn't load obvious wrong data, can do relocations/allocations, etc.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
René Nussbaumer [Thu, 15 Mar 2012 13:58:30 +0000 (14:58 +0100)]
htools: Reflect the rename of spindle_usage to spindle_use
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Thu, 15 Mar 2012 13:55:55 +0000 (14:55 +0100)]
Rename spindle_usage to spindle_use on Ganeti side
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Tue, 6 Mar 2012 23:49:52 +0000 (01:49 +0200)]
Add a helper function for mirrorType computations
We most always use `templateMirrorType . diskTemplate`, so let's add a
helper function for this.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 23:40:18 +0000 (01:40 +0200)]
Fix TLMigrateInstance use of IAllocator
hail now expects correctly that relocate_from is of equal length with
the number of required nodes (fixme: there's a lot of not well
documented behaviour here… not nice for any other potential
IAllocators). As such, we _need_ to pass just the instance's primary
node.
Additionally, update the iallocator doc to correctly specify what this
list (`relocate_from`) contains.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 23:29:49 +0000 (01:29 +0200)]
Enable all disk templates in unit-tests
After the many patches implementing the shared storage functionality
in iallocator, balances, etc., and after the patches preparing the
unit-tests to do the right thing for shared storage, we can finally
flip the bits that make instances have "random" disk templates as
opposed to just DRBD. Yes, the unit-tests still pass ☺
We also need to enable all disk templates in the IPolicy (for
allocation/relocation).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 22:03:28 +0000 (00:03 +0200)]
Enable evacuation/relocation for shared storage
This patch toggles the final bit and enables nodeEvacInstance to work
on shared storage instances. Diskless instances are handled the same
(as in, None is the same everywhere :).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 21:53:33 +0000 (23:53 +0200)]
Replace explicit case with 'either'
Instead of manually case-ing on the Either contents, let's just use
either with const functions (not really readable, but…).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 21:50:50 +0000 (23:50 +0200)]
Rename evacDrbdSecondaryInner to evacOneNodeInner
The function is now generic, so let's rename it and cleanup/improve
the docstrings.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 20:41:21 +0000 (22:41 +0200)]
Abstract the secondary evac into a separate function
This is generic enough (no longer depending on the replace type) that
it will be used for other replace modes too. So all we need to do is
to abstract it to a separate function (that can be called from
multiple branches) and to make the imove op_fn computation more
generic, instead of hard-coding it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 20:28:04 +0000 (22:28 +0200)]
Fix Node_addPriFD test for arbitrary disk templates
Currently, this test assumes localStorageTemplates, so let's make sure
we enforce that on the given instance
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 18:45:46 +0000 (20:45 +0200)]
Extend the possible move types with FailoverToAny
This brings together all previous pieces and allows balancing to work
(with the known caveats related to disk space calculation) for shared
storage disk templates.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
René Nussbaumer [Wed, 14 Mar 2012 10:15:59 +0000 (11:15 +0100)]
htools: Add the new ipolicy field
Also add the spindle usage to QC
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Tue, 13 Mar 2012 12:39:44 +0000 (13:39 +0100)]
ipolicy: Keep track of spindle usage
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Wed, 14 Mar 2012 10:38:21 +0000 (11:38 +0100)]
htools: Use the spindle usage in the algorithms
In the same transition also do a simple refactor of the caculation
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Tue, 13 Mar 2012 10:52:27 +0000 (11:52 +0100)]
htools: Convert all the other backends as this is trivial
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Tue, 13 Mar 2012 10:47:31 +0000 (11:47 +0100)]
htools: Adapt the Text backend to spindle usage
The old text files remain compatible with this change.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Tue, 13 Mar 2012 10:31:33 +0000 (11:31 +0100)]
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backend
later.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Mon, 12 Mar 2012 14:01:21 +0000 (15:01 +0100)]
iallocator: Add the spindle_usage to iallocator
This adapts the Ganeti side to export the spindle_usage
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Mon, 12 Mar 2012 13:18:47 +0000 (14:18 +0100)]
constants: Add new spindle_usage parameter to BE
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 14 Mar 2012 16:23:54 +0000 (17:23 +0100)]
Rework unit parsing
Due to how conversions were implemented previously, 1TB failed to
parse on 32-bit, as we were overflowing during computation, even
though the final result would fit easily.
This patch moves the parsing of the scaling factor to a separate
function, and all the conversions are done via the Rational type
(which has unlimited arbitrary precision), and conversion to the
desired type only happens at the last step.
The unit-tests are adjusted too, unfortunately they use the same
algorithm as the code… suggestions on how to improve things are
welcome.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Alexander Schreiber [Tue, 13 Mar 2012 16:34:52 +0000 (17:34 +0100)]
Enable lvmstrap to run under Linux 3.x
Extend the kernel version check to also accept Linux 3.x as valid.
Signed-off-by: Alexander Schreiber <als@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 14 Mar 2012 10:15:02 +0000 (11:15 +0100)]
Fix make distcheck after the new shell tests
With the new shell tests, we have for the first time the situation
where tests require a binary built (via make), but the binary itself
is not the test. Hence, we need to declare and use check_SCRIPTS for
it.
Also corrects a wrong file name (introduced during rebase/editing the
patch series).
The end result is that `make distcheck` now passes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 8 Mar 2012 01:03:14 +0000 (03:03 +0200)]
Add more rebalance offline tests
These test that we can actually rebalance (and that it results in
replace-disks and failovers/migrations), and that we don't
double-rebalance, etc.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Thu, 8 Mar 2012 00:12:50 +0000 (02:12 +0200)]
Enhance the offline tests
This adds new offline tests:
- checks that files generated by hspace can be read by hbal/hinfo
- checks that hbal handles node groups as expected
- checks that enabling multiple backends results in failure
- checks hspace machine readable code and higher verbosity
- enables a "FAIL" message on error exit
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Thu, 8 Mar 2012 00:11:14 +0000 (02:11 +0200)]
Build a coverage-enabled version of the tools
This allows the offline tests to also generate coverage data, and the
hs-coverage target is changed to show both unit-test and offline-tests
coverage.
The downside is that now we build yet-another-binary, which makes the
unit-test time slower…
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Wed, 7 Mar 2012 23:19:53 +0000 (01:19 +0200)]
Enable htools offline tests as Haskell tests
This enables the offline-test.sh run both in the special "hs-check"
rule and during normal "make check".
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Wed, 7 Mar 2012 23:11:42 +0000 (01:11 +0200)]
Add a simple binary tester for htools
This only tests the command line basic options. In the future, it can
be expanded to check other things.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Wed, 7 Mar 2012 23:10:30 +0000 (01:10 +0200)]
Add a helper for htools CLI tests
… and update live-test.sh to use it, instead of hard-coding the
program paths.
A few additional style fixes are also done (indentation).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Thu, 24 Nov 2011 14:05:18 +0000 (15:05 +0100)]
htools: add confd daemon binary
This is currently not installed anywhere, just built.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 24 Nov 2011 14:04:32 +0000 (15:04 +0100)]
htools: add confd server module
This contains a more-or-less complete implementation for the
server-side confd.
Note that most of the code is behaving identical to the Python code,
with a notable exception: the asyncore/inotify code was changed (since
Haskell doesn't have asyncore) to 3-thread system. While otherwise I
tried to keep the code flow identical, I found the original Python
code not easily to reason about, and as such I changed the
architecture in this regard.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 24 Nov 2011 13:50:02 +0000 (14:50 +0100)]
htools: add basic daemon-related functionality
This is not complete for now, just the basic functionality has been
implemented:
- daemonize
- check we're running under the correct user
- call setup logging
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Wed, 23 Nov 2011 16:17:08 +0000 (17:17 +0100)]
htools: add logging functionality
This adds a dependency on hslogger 1.1+, which is not available in
Debian Squeeze, only in later versions.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 21 Nov 2011 15:33:00 +0000 (16:33 +0100)]
htools: add runtime functionality
This duplicates the lib/runtime.py functionality, allowing us to check
for runtime users and groups consistency.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Sun, 20 Nov 2011 17:16:18 +0000 (18:16 +0100)]
htools: add a small module for crypto functions
This adds a dependency on the following new libraries:
- Data.HMAC module from the 'crypto' library
- Data.Text from the 'text' library
- Data.ByteString from the 'bytestring' library
In Debian, the bytestring library comes with the ghc compiler, whereas
the other two are available separately in wheezy/unstable.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Fri, 18 Nov 2011 10:36:31 +0000 (11:36 +0100)]
htools: add definitions for confd types
While we have some of these as plain types in Constants.hs, we add
proper ADT definitions for them in a new file. Furthermore, we add the
ConfdRequest and ConfdReply types here (in Python they are in
objects.py).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Fri, 18 Nov 2011 10:13:37 +0000 (11:13 +0100)]
htools: add a very basic lib/config.py equivalent
This only is able to load the configuration data for now.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Fri, 18 Nov 2011 09:59:12 +0000 (10:59 +0100)]
htools: add partial implementation of lib/objects.py
This is partial since not all object types can be easily converted for
now (will need some changes on the Python side for this).
Most importantly, the *Params types do not have a good solution now:
the Python code, due to its dynamic typing, hides the fact that we
actually have two different types at play: a full type which needs to
have all keys, and the 'partial' type which has slightly different
behaviour. I've implemented these in Haskell as two different types,
Full* and Partial*, which are derived automatically from a single
Parameter type, together with the associated Fill* functions.
Furthermore, HVParams is even more special, as its contents is not
fixed but varies per hypervisor type, plus it has the HV_GLOBALS part
which should not be customisable at instance type (yay for
exceptions). As such, this should be written in Haskell as a
multi-constructor type, but it's the only one so far and thus we don't
have support for it yet.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
René Nussbaumer [Fri, 2 Mar 2012 12:15:13 +0000 (13:15 +0100)]
htools: Make it possible to load an IAlloc request
This is only useful in the stand-alone command line tools:
- hbal
- hspace
- hinfo
This change also documents the flag in all man-pages.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 7 Mar 2012 21:43:50 +0000 (23:43 +0200)]
Fix lint error in commit
035b33e2
Commit
035b33e2 forgot one blank space, and current pylint in Debian
Sid doesn't run on our code… so I didn't realise this until after
commit, sorry.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 20:55:49 +0000 (22:55 +0200)]
Fix relocation test ClusterAllocRelocate
As for the evacuate tests, we require a mirrored disk template, but
otherwise the test should work for both mirror types.
Additionally, we perform a simplification that was left as TODO.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 23:24:28 +0000 (01:24 +0200)]
Fix IAlloc.processRelocate to support shared storage
Currently, this function expects (but doesn't really check! only via
secondary node being defined) a DRBD8 template type.
This patch (not very readable, sorry) changes it so that we check
whether we're internally mirrored, externally mirrored, or not
mirrored at all. It then proceeds to do the right thing™ per the
mirror type, so that it can work well with both.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 22:20:32 +0000 (00:20 +0200)]
Fix up Cluster nodeEvac unittests for shared storage
Currently, these presume that the instance being generated always is
DRBD/relocatable (but won't be in the future), and also presume that
all relocation modes can be used (hint: not ☺).
So we add a couple of helper functions to restrict the used disk
templates/relocation modes, and these two tests are ready for shared
storage.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 22:11:19 +0000 (00:11 +0200)]
Small improvement for nicer unit-testing
In case you just run "make htools/test" to recompile the unittests,
the old .tix file should be removed, otherwise you get the message
about inconsistent tick boxes. Let's make the compilation process
automatically remove this, for easier testing.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 20:32:40 +0000 (22:32 +0200)]
Rework evacDrbdSecondaryInner to take the imove
Instead of hard-coding the instance move operation, let's accept is as
a parameter from the caller. This is the single thing that makes this
DRBD-specific, so removing it will help for future
patches. Furthermore, this abstract the imove op type into a single
place instead of two (the inner function and the nodeEvacInstance).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 20:13:24 +0000 (22:13 +0200)]
Add a simple type alias for nicer type signatures
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 20:46:56 +0000 (22:46 +0200)]
Fix a few tests that hard-coded the number of nodes
Instead of randomly generating the number of required nodes or
hard-coding it, compute it from the disk template of the
instance. Otherwise tests will fail when the disk template is not
DRBD8 and we ask for allocation on 2 nodes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 20:43:42 +0000 (22:43 +0200)]
Fix test Node_addSec_idempotent
Another test that only works for DRBD8 instances.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 20:41:38 +0000 (22:41 +0200)]
Fix the Node_rMem test to specify DRBD8 templates
… since memory reservation only is used for this template.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 20:39:37 +0000 (22:39 +0200)]
Split the Node_addOffline test into pri/sec
Currently, this (single) test checks whether we can add an instance as
primary and secondary to a given node. This won't work for instance
templates other than DRBD8, so we split it into:
- an add to primary test, which should work for all disk templates
- an add to secondary test, for which we set the disk template to DRBD
explicitly
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 13:54:19 +0000 (14:54 +0100)]
Use the mirror type in Cluster.possibleMoves
Currently the function possibleMoves always presumes DtDrbd, which is
wrong for shared storage. To make it work with all disk templates, we
additionally pass the mirror type to it (we assume that the move
behaviour only depends on the mirror type, and not the individual
disk template; currently this is true, and we should aim to keep it
that way).
For none and external mirrors, we don't give any solutions; for
internal mirrors, we keep the previous algorithm.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 13:50:25 +0000 (14:50 +0100)]
iPolicy: enable by default all storage templates
… otherwise we can't run tests or simulations using non-default
templates. Tests still pass.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 13:11:15 +0000 (14:11 +0100)]
Fix printSolutionLine to handle big scores
Currently we don't set a boundary on the length of the score field,
which breaks all our nicely-aligned output efforts.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 13:06:57 +0000 (14:06 +0100)]
Fix printSolutionLine to handle non-DRBD steps
Currently Cluster.printSolutionLine function always wants to print the
old and new secondary, and as such it fails (ugly) for moves of
non-DRBD instances. Change it so that it looks explicitly for
noSecondary nodes, and format the output different in that case.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 12:55:54 +0000 (13:55 +0100)]
Add a helper type and function for the mirror type
Currently we track the mirroring type as simply yes/no, with "yes"
meaning DRBD. This is not enough for handling shared storage, so we
need to add a type that gives the exact mirroring type (none,
internal, external), and a function that ties the disk template type
to the mirroring type.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 12:09:24 +0000 (13:09 +0100)]
Small simplification in tryBalance
Instead of computing the bad/good node lists separately (in effect
iterating twice over the all_nodes list and also creating that list
twice from the container), let's use 'partition' and do a small
improvement.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 11:58:47 +0000 (12:58 +0100)]
Mark Rbd as a movable disk template
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Sun, 19 Feb 2012 21:15:17 +0000 (22:15 +0100)]
Do not rely on no-secondary for movable tests
Currently, the code in Loader.hs marks as un-movable any instances
which don't have a secondary. This is not valid for any shared storage
templates, so let's remove that check completely. The initial state of
the instance will reflect correctly the movable status, based on the
disk template of the instance, so we don't need to re-do this check.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Sun, 19 Feb 2012 20:59:48 +0000 (21:59 +0100)]
Handle nodes without local storage
Our current cluster model is based on per-node storage, and as such we
don't handle shared disk templates. Until we can improve the model,
let's handle nodes without local storage better: in order to still
compute a valid cluster score, we need to handle nodes with tDsk == 0.
For this, we abstract the in-line computation of pDsk (which is used
in the cluster score) to a separate function which, for tDsk == 0,
returns instead 1 (i.e. disk fully used).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 6 Mar 2012 19:38:18 +0000 (21:38 +0200)]
Fix tempfile reset code & test on newer Python
Python 2.7.3 (rc status) and 3.2.3/3.3 (rc, respectively alpha status)
have fixed http://bugs.python.org/issue12856 which we worked around
ourselves.
This means two things:
- we don't need to manually reset the module
- we can't test for the no-reset case
Unfortunately current Debian Sid has the 2.7.3 RC but still labeled as
2.7.2+ ☹, so we have to manually do an extra check (in the unit test
only); I expect Debian will update to official 2.7.3 as soon as it's
released, and then we can remove this override.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 5 Mar 2012 21:48:16 +0000 (23:48 +0200)]
Merge remote-tracking branch 'origin/devel-2.5'
Conflicts:
Makefile.am (trivial, test data files added on both branches)
lib/opcodes.py (trivial, master renamed filter→qfilter, 2.5
fixed the type of the parameter)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 5 Mar 2012 10:49:41 +0000 (11:49 +0100)]
Split check-man-warnings into two
This moves the dashes check to a new check-man-dashes script.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 5 Mar 2012 10:44:51 +0000 (11:44 +0100)]
Rename check-man to check-man-warnings
Currently the check-man test is dependent on the MAN_HAS_WARNINGS
test, which made sense as long as the script only did the warnings
check.
However, we should run the [em] tests always, and in general we could
run other tests too, even if man doesn't support --warnings; so let's
split the check man. As a first prerequisite, we rename check-man to
check-man-warnings (and the respective Makefile variables).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Mon, 5 Mar 2012 10:38:35 +0000 (11:38 +0100)]
Makefile: improve the fix for .DELETE_ON_ERROR
I've investigated more and it seems that .DELETE_ON_ERROR not working
is a bug/regression in some versions of make; testing with make 3.82
from Debian experimental makes this special target work as
expected. Also, make 3.81 works on trivial Makefiles, but not on our
complex one.
In the meantime, let's improve the fix for the building of man pages:
make the removal implicit, instead of explicit, via the use of 'trap'
on EXIT. I just hope this behaves consistently across all bash
versions :)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
René Nussbaumer [Fri, 2 Mar 2012 10:52:30 +0000 (11:52 +0100)]
htools/ExtLoader: Bring the pieces together
This make it possible to load from an Ialloc file
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Fri, 2 Mar 2012 10:51:32 +0000 (11:51 +0100)]
htools/CLI: Adding new options to use ialloc as source
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Fri, 2 Mar 2012 10:50:55 +0000 (11:50 +0100)]
htools/IAlloc: Adding loadData method for use as backend
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Fri, 2 Mar 2012 10:49:34 +0000 (11:49 +0100)]
htools: Refactor some of IAlloc.readRequest to Hail
This makes it possible to reuse that function when we want to use IAlloc
as a data source backend
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Thu, 1 Mar 2012 09:54:54 +0000 (10:54 +0100)]
hinfo: Adjust the verbosity
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Wed, 29 Feb 2012 15:29:44 +0000 (16:29 +0100)]
hinfo: Gather and print group statistics
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 29 Feb 2012 14:55:58 +0000 (15:55 +0100)]
Explicitly remove the .x.in man pages on failure
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 29 Feb 2012 14:55:18 +0000 (15:55 +0100)]
More -- fixes
Because .DELETE_ON_ERROR doesn't work consistently (works when the
target we're talking about is a final target, but not for
intermediates), I missed a lot of man checks.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Wed, 29 Feb 2012 13:33:49 +0000 (14:33 +0100)]
htools: Adding printTable helper
This helper abstract away all the unlines and unwords call you've to do
after you got a formatTable back.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>