Agata Murawska [Thu, 13 Sep 2012 08:34:25 +0000 (10:34 +0200)]
Correct top comment in Query/Server
I assumed this was a copy-paste+forgetting to change the header comment.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Thu, 20 Sep 2012 14:15:17 +0000 (16:15 +0200)]
Add nodedCertFile to Path
This patch adds missing Path entry and uses it in Rpc. It was not in
place before due to different build settings - i.e. if curl was
disabled, there were no problems.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
René Nussbaumer [Wed, 26 Sep 2012 12:53:53 +0000 (14:53 +0200)]
Verify the node count just if the allocation was successful
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 25 Sep 2012 10:30:07 +0000 (12:30 +0200)]
Enable bash completion for htools scripts
This patch ties together the previous commits, by extending
build-bash-completion to call all htools scripts and get their
completion information, then use that to build fake cli_options
representing them and finally generate the bash completion
information.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 25 Sep 2012 09:42:49 +0000 (11:42 +0200)]
Add option for displaying completion information
This patch adds support for a --help-completion option, which will
display the defined options and their completion information, in a
format designed to be parsed easily from Python, for integration into
build-bash-completion.
The alternative would have been to duplicate the build-bash-completion
functionality on the Haskell side, but that would have been lots of
duplication for not enough gain (since that is only run at build
time).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Tue, 25 Sep 2012 09:33:13 +0000 (11:33 +0200)]
Annotate options with completion information
This is a quite boring patch, just adding annotation information to
all existing options. Some of the annotations are not very good; but
we don't have support for more precise completion in
build-bash-completion, so this is good enough.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Tue, 25 Sep 2012 09:30:29 +0000 (11:30 +0200)]
Add a data type for completion information
This can be used for either option or argument completion (although
OptComplNone doesn't make sense for an argument :).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Tue, 25 Sep 2012 08:37:48 +0000 (10:37 +0200)]
Remove generic options from individual programs
Currently, we test and require that each individual program (hbal,
etc.) defines/supports the generic options (currently --help and
--version). Even with the test, this is not optimal, since it requires
changes in many places whenever we modify the list of generic options,
hence we move these out of the individual programs and into the
generic CLI handling code.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Tue, 25 Sep 2012 14:54:12 +0000 (16:54 +0200)]
Add printing of group score in hinfo
The global cluster score is less interesting than individual group
scores, for multi-group allocation purposes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 25 Sep 2012 14:49:09 +0000 (16:49 +0200)]
Replace a big tuple with a custom data type
This will make it easier to add new parameters.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Michael Hanselmann [Thu, 20 Sep 2012 16:52:31 +0000 (18:52 +0200)]
Implement virtual cluster support in Python code
- pathutils: Prepend node-specific prefix path
- RPC: Use virtual paths (see vcluster.py)
- SSH: Pass environment variables, use destination's node directory when
copying files using scp, use GANETI_HOSTNAME to determine hostname
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Tue, 25 Sep 2012 13:28:45 +0000 (15:28 +0200)]
vcluster: Add more details to docstrings
As requested by René Nussbaumer. Hopefully things are more clear now.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Tue, 25 Sep 2012 09:48:46 +0000 (11:48 +0200)]
Add tests for bash completion
Given the number of settings that need to be controlled a plain bash
script was a better choice over shelltestrunner. Just a few completions
are attempted, but among them should be the most critical ones (e.g.
“gnt-instance add --node …”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Sep 2012 14:06:45 +0000 (16:06 +0200)]
Fix shelltests for ganeti-*
They were never run.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Sep 2012 13:03:53 +0000 (15:03 +0200)]
bash_completion: Enable extglob while parsing file
In older versions of GNU Bash extended patterns, such as “@(…)”, are only
available with the “extglob” shell option. As pointed out in [1] and [2],
“extglob” must be enabled while parsing the code. Therefore the flag must be
enabled at the beginning of the script and be reset to its original value at
the end as to not interfere with other code on shell initialization.
[1] http://unix.stackexchange.com/questions/45957
[2] http://mywiki.wooledge.org/glob
Reported by Sascha Lucas.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Mon, 24 Sep 2012 12:43:20 +0000 (14:43 +0200)]
Revert "bash_completion: Always enable extglob"
This reverts commit
94014b6328f813b674a1126ffd221705d7e315b3. Turns out
I misunderstood the “extglob” shell option.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
René Nussbaumer [Wed, 19 Sep 2012 11:45:59 +0000 (13:45 +0200)]
Adding RAPI resource for multi-allocation
This is straightforward.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Mon, 17 Sep 2012 12:23:20 +0000 (14:23 +0200)]
Adding multi-allocation logic
This patch adds the Logical Unit for the instance multi allocation
opcode.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Thu, 13 Sep 2012 11:18:57 +0000 (13:18 +0200)]
Adding the new opcode for multi-allocation
Skeleton for the multi-allocation opcode
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Fri, 14 Sep 2012 12:41:18 +0000 (14:41 +0200)]
Refactor shared code
We've some work which we need to do for the multi allocation request as
well, so lets refactor it to separate functions.
_ComputeNics is kinda ugly, as it reserves the MAC. I added a TODO to
think more about how we factor this behaviour out.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Wed, 19 Sep 2012 09:14:14 +0000 (11:14 +0200)]
Adding some fundamental unittests for iallocator
This test covers the bug fixes found in the previous two patches
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Tue, 18 Sep 2012 14:58:54 +0000 (16:58 +0200)]
Fix REQ_RESULT for MultiInstanceAllocRequest
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Tue, 18 Sep 2012 14:11:28 +0000 (16:11 +0200)]
Fix a validation issue
We want to validate the result only if the iallocation run was a
success.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Thu, 20 Sep 2012 16:50:56 +0000 (18:50 +0200)]
netutils: Use virtual hostname if set
This enables the hostname seen by other code through Hostname.GetFqdn to
be changed through the environment variable “GANETI_HOSTNAME”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 20 Sep 2012 16:49:32 +0000 (18:49 +0200)]
daemon-util: Support virtual clusters
GANETI_ROOTDIR contains the root directory for the current “virtual node”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 20 Sep 2012 16:47:28 +0000 (18:47 +0200)]
Add new module for virtual clusters
This module will take care of managing paths for virtual clusters.
Unittests are included (100% coverage).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 20 Sep 2012 16:52:49 +0000 (18:52 +0200)]
bootstrap.SetupNodeDaemon: Stop hardcoding bind address
Commit
b43dcc5a11 added support for IPv6. To have the node daemon bind
to an IPv6 address on startup it changed the code to pass a static bind
address (0.0.0.0 or ::0). If a user had configured another bind address
using /etc/default/ganeti, that address would not be used until the
daemons were restarted.
By copying ssconf files before attempting to start the daemons we can
ensure that the built-in IP family detection in daemon.py works.
This fixes issue 267.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Thu, 20 Sep 2012 16:44:21 +0000 (18:44 +0200)]
hv_kvm: Use pu.CONF_DIR variable
… instead of using SYSCONFDIR directly.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Thu, 20 Sep 2012 16:42:49 +0000 (18:42 +0200)]
utils.filelock: Remove executable bit from lock files
There's no need for lock files to be executable.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Thu, 20 Sep 2012 16:45:50 +0000 (18:45 +0200)]
LUInstanceCreate: Include group name in error message
“pnode.group” contains the UUID, which isn't very user-friendly.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Tue, 18 Sep 2012 13:25:07 +0000 (15:25 +0200)]
bash_completion: Generate more compact version
First the numbers:
$ stat --format '%s %n' doc/examples/bash_completion*
77847 doc/examples/bash_completion
86492 doc/examples/bash_completion-debug
The non-debug version doesn't use indentation and does not have some
(primitive) facilities for debugging. The savings are about 8.5 kB or
10 %. The “-debug” version is used by “devel/upload”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Tue, 18 Sep 2012 13:06:25 +0000 (15:06 +0200)]
ShellWriter: Add parameter to disable indentation
This will be used to write a more compact bash completion script.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Tue, 18 Sep 2012 12:56:52 +0000 (14:56 +0200)]
bash_completion: Always enable extglob
In older versions of GNU Bash extended patterns, such as “@(…)” are only
available with the “extglob” shell option. This patch adds a wrapper
function so that extglob is always enabled while doing any completion.
Due to early returns inside the main completion function this seemed
like the best option.
Reported by Sascha Lucas.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:38 +0000 (18:28 +0200)]
constants: Stop using wildcard import for pathutils
It still needs to be imported for hypervisor parameter defaults, but
after this change paths are no longer exported from constants.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 17:19:28 +0000 (19:19 +0200)]
Use autoconf-based paths from Haskell instead of constants
Future changes will change Path.hs to use an environment variable.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:57 +0000 (18:28 +0200)]
Migrate tools/* from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:57 +0000 (18:28 +0200)]
Migrate test/ganeti.*.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:57 +0000 (18:28 +0200)]
Migrate lib/watcher/__init__.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:56 +0000 (18:28 +0200)]
Migrate lib/utils/*.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:56 +0000 (18:28 +0200)]
Migrate lib/uidpool.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:56 +0000 (18:28 +0200)]
Migrate lib/tools/ensure_dirs.py to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:56 +0000 (18:28 +0200)]
Migrate lib/ssh.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:56 +0000 (18:28 +0200)]
Migrate lib/ssconf.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:56 +0000 (18:28 +0200)]
Migrate lib/server/*.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:56 +0000 (18:28 +0200)]
Migrate lib/rpc.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:56 +0000 (18:28 +0200)]
Migrate lib/rapi/baserlib.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:55 +0000 (18:28 +0200)]
Migrate lib/ovf.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:55 +0000 (18:28 +0200)]
Migrate lib/mcpu.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:55 +0000 (18:28 +0200)]
Migrate lib/masterd/instance.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:55 +0000 (18:28 +0200)]
Migrate lib/luxi.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:55 +0000 (18:28 +0200)]
Migrate lib/{jqueue,jstore}.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:55 +0000 (18:28 +0200)]
Migrate lib/hypervisor/*.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:54 +0000 (18:28 +0200)]
Migrate lib/config.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:54 +0000 (18:28 +0200)]
Migrate lib/confd/*.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:54 +0000 (18:28 +0200)]
Migrate lib/cmdlib.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:54 +0000 (18:28 +0200)]
Migrate lib/client/*.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:54 +0000 (18:28 +0200)]
Migrate lib/cli.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:54 +0000 (18:28 +0200)]
Migrate lib/bootstrap.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:54 +0000 (18:28 +0200)]
Migrate lib/backend.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:53 +0000 (18:28 +0200)]
Migrate build-bash-completion from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 16:28:53 +0000 (18:28 +0200)]
Migrate qa/*.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 14:26:03 +0000 (16:26 +0200)]
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Many
paths will change based on an environment variable and are no longer
constant and should no longer be in “constants.py”. Since “constants.py”
is already huge a number of other paths are also moved in the process.
For now a wildcard import is used to re-export all paths from
“constants.py” (this will change over coming patches). Daemon log paths
have been changed to use a function. “RUN_GANETI_DIR” was moved to
“RUN_DIR” as the latter was only used in “constants.py”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 12:24:00 +0000 (14:24 +0200)]
configure: Provide options to change default user/group
This is needed for virtual clusters where a non-root user is used to run
the daemons.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 17 Sep 2012 15:40:50 +0000 (17:40 +0200)]
constants: Add missing word in comment
It was missing in commit
243cdbcc on May 7, 2008.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Bernardo Dal Seno [Thu, 13 Sep 2012 21:15:15 +0000 (23:15 +0200)]
QA for instance migration made more compact
Also, the value of always_failover is not hard-coded any more, and it's
possible to disable changing the always_failover parameter, if needed.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Bernardo Dal Seno [Thu, 13 Sep 2012 20:25:32 +0000 (22:25 +0200)]
Always_failover doesn't require --allow-failover anymore
If an administrator sets always_failover, it means that there is no need
for another explicit approval to failover instead of migrating.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Bernardo Dal Seno [Mon, 3 Sep 2012 08:26:36 +0000 (10:26 +0200)]
Remove useless lock in recreate-disks
The old primary node was being locked to do RPC, but we don't lock nodes
for RPC anymore, so the lock is useless.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Wed, 12 Sep 2012 11:47:10 +0000 (13:47 +0200)]
Update design for virtual clusters
The biggest change is going from a command line parameter for specifying
a daemon's directory to using an environment variable. Some other things
are clarified.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
René Nussbaumer [Wed, 12 Sep 2012 12:43:08 +0000 (14:43 +0200)]
Small style adjustment
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Wed, 12 Sep 2012 12:41:25 +0000 (14:41 +0200)]
Make iallocator request types more strict
This adapts the type of some request input fields to more stricter ones.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Wed, 12 Sep 2012 11:43:28 +0000 (13:43 +0200)]
Rework the required_nodes field
This was used to verify the result of instance allocation. However, this
is not logic which belongs to the cmdlib rather than to the iallocator
request itself. So abstract it by its own function. And update the
ValidateResult accordingly.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 12 Sep 2012 12:42:40 +0000 (14:42 +0200)]
objects, rpc: Code cleanup
objects.Disk.ComputeLDParams: Simplify structure, don't use and
overwrite local variables.
rpc.AnnotateDiskParams: Replace explicit loop.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Wed, 12 Sep 2012 09:53:09 +0000 (11:53 +0200)]
RAPI client: Fix typo in docstring
Also update the copyright header.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
René Nussbaumer [Tue, 11 Sep 2012 08:37:59 +0000 (10:37 +0200)]
Disable pylint warnings for unknown members
For some reason pylint can't determine, that the members are calculated
upon runtime and complains. This is weird as the same mechanism works on
opcode/objects. I haven't found the issue for it yet, so let's just
ignore the warnings for now.
This is a separate patch for easier revert later on, once it's fixed.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Thu, 6 Sep 2012 14:18:05 +0000 (16:18 +0200)]
Adapt gnt-debug iallocator
Add the new multi-alloc request to the test allocation framework. For
now we just create --count identical instances. This might need further
improvements later.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Thu, 6 Sep 2012 14:17:21 +0000 (16:17 +0200)]
Adding new multi-allocation request
Now that we've the refactoring done, we can easily add the new mode and
everything works.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Thu, 6 Sep 2012 11:45:04 +0000 (13:45 +0200)]
Switch all iallocator calls to the new one
After the refactor, the old calls are not working anymore, so we switch
to the new ones.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Thu, 6 Sep 2012 11:23:49 +0000 (13:23 +0200)]
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passing
in parameters on top level which works, but is hard to maintain and not
flexible.
With the upcoming change to the IAllocator for MultiAllocate we can't
use the toplevel parameters anymore. Therefore, we refactor the code
into separate classes, which just do their own stuff. It works similiar
to Opcode containers.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Fri, 7 Sep 2012 12:53:03 +0000 (14:53 +0200)]
Make the __slots__ functionality more modular
As we will introduce another set of containers using the __slots__ trick
we abstract away as much as possible to separate bases classes. The
child classes then adapt them for their needs. This leads to less code
duplication.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Fri, 7 Sep 2012 11:52:26 +0000 (13:52 +0200)]
Move _MakeLegacyNodeInfo to rpc
This function will also be needed for the iallocator refactor. Also this
function does not belong to cmdlib but rather to rpc. Lets move it
there.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
René Nussbaumer [Fri, 7 Sep 2012 11:26:32 +0000 (13:26 +0200)]
Move _CalculateGroupIPolicy and _ComputeDiskSize
This makes it possible to better untangle the IAllocator code, which
would otherwise lead to a recursive import structure as we need those
helpers there as well.
Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Mon, 10 Sep 2012 12:51:59 +0000 (14:51 +0200)]
Update documentation for "gnt-job list"
Commit
f037e9d7 added command line flags to filter jobs, but the
documentation was not updated. Also mention the ability to use
more complex filters.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 6 Sep 2012 21:57:17 +0000 (06:57 +0900)]
Simplify a bit more the test harness
We can build the test groups directly in the `testSuite' helper,
instead of doing it (much later) in the test harness.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Sep 2012 21:47:28 +0000 (06:47 +0900)]
Remove the slow/fast tests functionality
Since the recent commits improved the speed of the two "slow" test
groups to regular test speed, we can remove this kludge and simplify
significantly our test runner, yay!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Sep 2012 20:57:41 +0000 (05:57 +0900)]
Fix arbitrary ConfigData object generation
The Cluster object, as it is defined right now, has many '[String]'
members, which means that in a standard arbitrary generator these will
become very big, which is the reason for the current slowness of the
test 'Config_serialisation'.
By resizing the generator to 8 (arbitrary chosen to limit the list
length and the string sizes), and by reducing a bit the node count, we
can make this test be as fast as the others (about 10x
improvement). This means we can test more cases, for the same cost.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Sep 2012 18:17:12 +0000 (03:17 +0900)]
Improve the `AllocPolicy' test
This test has a few deficiencies, which this patch addresses:
- using arbitrary 1 or 2 node count for allocation is obsolete,
nowadays we need to use a number appropriate for the instance's disk
template (and we should remove that parameter…)
- generating a random node is sub-optimal, since we could generate an
offline node, and no instance will fit on a cluster composed of only
offline nodes
- generating arbitrary instances "such that" they can be allocated is
an expensive test; let's rather generate instances smaller than our
template node, and add a check that they indeed can be allocated
- using boolean return type, instead of nicely annotated properties
For the nice annotation and the extra check, we need to change a
helper function's signature, so that we can extract a bit more
information out of it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Sep 2012 17:19:46 +0000 (02:19 +0900)]
Improve the `CanTieredAlloc' test
Currently, this test is very slow. Upon investigation, this is due to
how `tieredAlloc' works:
- tries to allocate one instance
- if failed, shrink the instance by the "most failed" resource
- restart
In this algorithm, if the "most failed" resource is e.g. memory, and
the maximum available memory is much smaller than the current
template, it means we will have to shrink and try to allocate many
many times until we finally get with the to-be-allocated instance
memory size to a reasonable value. In the real world, this is not the
case, but when testing with arbitrary memory/node values, it can be
that we execute the shrink hundreds of thousands of times per test.
So we "improve" the test by directly generating an instance just
slightly bigger than the node, so that we don't have to shrink too
many times. This requires a new export from test/…/Instance.hs.
Additionally, we allow up to 5 instances to be tiered-allocated, and
we cleanup the test checks, making the conditions much, much more
readable (IMHO).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Sep 2012 16:01:21 +0000 (01:01 +0900)]
Add new test for checking multi-allocations
This test expands the "single-alloc-no-rebalance" by allocating a few
instances on a small cluster, and ensuring that after we allocate all
of them, either we can't rebalance or if we rebalance the score
improvement is very small.
The last condition is needed because sometime rounding errors (we're
using double-precision floating point) can accumulate and result in
what is a no real change in the cluster state, but with an
infinitesimal score decrease (e.g. 1e-14).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Sep 2012 15:59:50 +0000 (00:59 +0900)]
Improve the prop_Alloc_sane test to detect mis-allocations
Currently, this just checks that a cluster cannot be rebalanced after
a single instance allocation. However, we can also test whether the
allocation decision computed a correct new cluster score, by checking
that against the one computed from the actual new node list.
Also, for nicer display, we convert the test from a Boolean to a
Property, with nice annotations.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Fri, 7 Sep 2012 21:19:30 +0000 (06:19 +0900)]
Merge branch 'devel-2.6' into submit
* devel-2.6:
Fix bug in non-mirrored instance allocation
Fix gnt-debug iallocator
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Fri, 7 Sep 2012 21:13:15 +0000 (06:13 +0900)]
Merge branch 'stable-2.6' into devel-2.6
* stable-2.6:
Fix bug in non-mirrored instance allocation
Fix gnt-debug iallocator
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Thu, 6 Sep 2012 16:08:43 +0000 (01:08 +0900)]
Fix bug in non-mirrored instance allocation
The function `allocateOnSingle' has a bug in the calculation of the
cluster score used for deciding which of the many target nodes to use
in placing the instance: it uses the original node list for the score
calculation.
Due to this, since the original node list is the same for all target
nodes, it means that basically `allocateOnSingle' returns the same
score, no matter the target node, and hence the choosing of the node
is arbitrary, instead of being done on the basis of the algorithm.
This has gone uncaught until reported because the unittests only test
1 allocation at a time on an empty cluster, and do not check the
consistency of the score. I'll send separate patches on the master
branch for adding more checks to prevent this in the future.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Tue, 4 Sep 2012 19:55:11 +0000 (21:55 +0200)]
Further hlint fixes
Commit 2cdaf22, “Re-enable standard hlint warnings”, got it almost
right. The only problem is that (confusingly) the default set of hints
is not in HLint.Default, but in HLint.HLint (it includes Default and
some built-ins).
After changing the lint file to correctly include the defaults, we had
another 128 suggestions:
- Error: Eta reduce (2)
- Error: Redundant bracket (4)
- Error: Redundant do (17)
- Error: Redundant lambda (7)
- Error: Redundant return (1)
- Warning: Avoid lambda (2)
- Warning: Redundant $ (42)
- Warning: Redundant bracket (35)
- Warning: Use : (1)
- Warning: Use String (4)
- Warning: Use camelCase (10)
- Warning: Use section (3)
which are fixed by the current patch. Note that the 10 "Use camelCase"
were all due to hlint not “knowing” the idiom of ‘case_’ (it does for
‘prop_’), for which I filled
http://code.google.com/p/ndmitchell/issues/detail?id=558.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Sun, 2 Sep 2012 16:03:52 +0000 (18:03 +0200)]
Rework CLI modules and tests
While investigating how we could test the Daemon.hs module, I realised
that we have a very, erm, sub-optimal situation:
- HTools/CLI.hs has a nice IO/pure separation testing in cmdline
parsing, which allows some basic functionality to be tested, but
uses direct 'read' in many options, which fails at runtime when
evaluating the argument, and not when parsing the options
- Daemon.hs lacks that, but has a much nicer 'reqWithConversion'
helper that can be used for nicer option parsing, and uses that +
tryRead instead of plain 'read'
Since this situation is very bad, let's clean it up. We introduce yet
another module, Common.hs, that holds functionality common to all
command line programs (daemons or not). We move the parsing to this
module, and introduce a type class to handle option types which
support --help/--version. This allows removal of duplicated code from
CLI.hs and Daemon.hs.
The other part of the patch is cleanup/rework of the tests for this
code: we introduce some helpers (checkOpt, passFailOpt,
checkEarlyExit) that can be used from the much-slimmer now tests for
CLI and Daemon. In the common module, we just test the yes/no helper
we have. Many new tests for boolean options and numeric options are
added.
A side change is the removal of the obsolete `--replay-count',
`--test-size' options (unused since commit
95f6c931, “Switch Haskell
test harness to test-framework”).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Sat, 1 Sep 2012 22:21:32 +0000 (00:21 +0200)]
Fix deserialisation bug in ResultEntry
Found via the newly added unit-tests, which test most of the
serialisation code in Query/Language (except for QueryResult, for
which we already tests both sub-components separately).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 31 Aug 2012 12:54:38 +0000 (14:54 +0200)]
Add query filter tests
These tests are node specific only because we don't have other query
types implemented yet, but what they actually test is the various
filter types.
The tests are trying to cover most filter functionality; missing for
now is proper checking for ContainsFilter and TrueFilter, the rest
should be more or less covered.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 31 Aug 2012 12:32:11 +0000 (14:32 +0200)]
Add some unittests for node queries
These new tests check that:
- no known fields return unknown
- any unknown field returns unknown
- the type of the fields is consistent between the getters and the
field definition
- the length of each result row corresponds with the number of fields
queried, and the length of the field definitions returned
- the length of the rows corresponds to the number of nodes
- querying fields on empty fields returns all fields
Finally this patch found a bug, in that the pinst_list/sinst_list
fields were declared as QFTNumber (copy-paste error from
pinst_cnt/sinst_cnt), yay!
I also changed genEmptyCluster to ensure that it generates unique node
names, so that the number of result rows is consistent with what we
requested, and switched ResultEntry from a normal constructor to
record syntax, so that we can extract the fields without having to use
pattern matching.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 31 Aug 2012 12:10:48 +0000 (14:10 +0200)]
Add a small 'passTest' helper
This is symmetric to failTest, and allows us to use it in cases where
we need to return a property.
While replacing 'property True' with it, I saw a case where we can
simplify the use and thus reworked that check.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 31 Aug 2012 10:54:07 +0000 (12:54 +0200)]
Add entire ConfigData serialisation tests
Using the recently-added genArbitrary, we can now implement Arbitrary
instances for even "huge" objects like Cluster, so let's use that to
implement entire ConfigData serialisation tests.
Note that, as we don't have yet proper types for some of the Params
fields, we have to cheat via FlexibleInstances and
TypeSynonymInstances, using either empty items or real arbitrary
values.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Iustin Pop [Fri, 31 Aug 2012 10:42:04 +0000 (12:42 +0200)]
Replace manual arbitrary instances with genArbitrary
There are a few more that could be replaces, once we start using
appropriate (new)types.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>