Iustin Pop [Thu, 13 Dec 2012 17:05:26 +0000 (18:05 +0100)]
Log only partial response in Luxi when in debug mode
Currently, we log the entire response (at debug level) in the Luxi
replies. This is not a good idea; the logging library operates on
strings, and as such it will use huge amounts of memory: without debug
logging, a certain gnt-job list invocation uses 295MB (+RTS -s) and
2m35s time, when in debug mode, it's 1525MB and 48m!
So we make two changes:
- first, we switch from "show (pp_value a)" to "encode a", which
generates a non-formatted string rather than a indented one
- second we log only the first 2000 characters; this should be enough
to understand the first part of the response
We could go for higher, or for logging in batches (that would be
faster, as well).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Constantinos Venetsanopoulos [Thu, 13 Dec 2012 16:06:05 +0000 (18:06 +0200)]
Pass check_ip and not hostname.ip to _ComputeNics
This should be done because in the case of --no-name-check there
is no 'hostname.ip' attribute, causing an execution error.
'check_ip' is always set (in CheckArguments) even if --no-name-check
is passed in the command line.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Dec 2012 15:03:30 +0000 (16:03 +0100)]
Add version check for doc/design-draft.rst
This is similar to README or “doc/security.rst”. Having a comment that
needs updating for a new version is more likely going to trigger a check
of the lists of design drafts.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Dec 2012 15:03:25 +0000 (16:03 +0100)]
Makefile/check-local: List all errors before failing
Use the “error” variable in multiple places and fail only once all tests
have been completed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Dec 2012 14:49:51 +0000 (15:49 +0100)]
Move four designs from draft to 2.7
These are the ones I implemented and for which I'm certain they're
complete.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Dec 2012 14:20:34 +0000 (15:20 +0100)]
RAPI documentation fix and update
- Mention user capabilities in security.rst
- Replace “query” with “read” in RAPI documentation
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Thu, 13 Dec 2012 11:48:55 +0000 (12:48 +0100)]
Add hpc-mon-collector to .gitignore
It is a binary file and it should never be added to the git repository.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 17:00:25 +0000 (18:00 +0100)]
Read watcher pause using RPC, not directly
The master daemon should not directly read files written by the node
daemon. This patch adds a new RPC to read the watcher pause file and
changes the master code to use it.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 15:48:03 +0000 (16:48 +0100)]
Set watcher pause on all nodes
Instead of just setting the watcher pause file, which isn't replicated,
RPC is used to set it on all nodes (where possible). This means that
after an outage of the master node and a following master-failover, the
watcher will still be paused.
The RPC is attempted on all nodes. Errors for nodes marked offline are
ignored. All other errors are returned in the form of an exception
(there is no other option given the LUXI API).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 16:17:20 +0000 (17:17 +0100)]
Add RPC for setting watcher pause
The watcher pause file should be set/unset on all nodes at once, not
only the master node. For that a new RPC is needed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 16:03:38 +0000 (17:03 +0100)]
Generated RPC code: Don't write whitespace after paren
There is no obvious need for this space--the line would wrap at the next
space, which comes soon after.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 16:09:44 +0000 (17:09 +0100)]
Rename test for backend._CommonRestrictedCmdCheck
“TestWriteFile” was not renamed when adding the file based on
“ganeti.utils.io_unittest-runasroot.py”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 13:32:13 +0000 (14:32 +0100)]
jqueue: Improve inotify error reporting
This addresses issue 218. When the number of inotify watches is
exhausted, for example by being set too low from the beginning or by
other programs, waiting for a job to change would just report a lost job
(e.g. “Error checking job status: Job with id 7817 lost”).
This patch changes the job watcher to no longer catch
“errors.InotifyError” and, this is by far the larger part of this patch,
adds unittests for this situation.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 14:52:56 +0000 (15:52 +0100)]
Improve test for tools.ensure_dirs
- Add more checks, some of them are deliberately redundant
- Descriptive error messages
- Add comment describing order to “tools.ensure_dirs”
- Avoid copying a list in an assertion in “tools.ensure_dirs”
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Dato Simó [Wed, 12 Dec 2012 15:32:46 +0000 (15:32 +0000)]
Jobs.hs: make L.Client the last argument for all functions
Having the Luxi client be the last argument of the functions allows for
easier use with `Exception.bracket L.getClient L.closeClient ...`.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Wed, 12 Dec 2012 13:46:27 +0000 (14:46 +0100)]
Fix a bug in the hs-coverage rule of Makefile.am
The coverage information for mon-collector was not merged correctly.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 12 Dec 2012 10:00:23 +0000 (11:00 +0100)]
uidpool: Remove roman number support
Doing so simplifies to code a bit and never had a practical use.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Constantinos Venetsanopoulos [Tue, 4 Dec 2012 11:50:16 +0000 (13:50 +0200)]
Remove checks wrt IDISK_PARAMS from OpCode level
Change the "--disks" option validation, to just check the format
of the dict and do not check whether the keys are included in the
IDISK_PARAMS constant at OpCode level. This allows the passing of
arbitrary parameters at the CLI, which will then be logically
checked at LU level (inside CheckArguments).
Currently, the check wrt to IDISK_PARAMS already happens in all
related LUs (InstanceSetParams, InstanceCreate, InstanceRecreateDisks),
with the use of ForceDictType(), so there is no need for additional
checking code.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed a unittest that needed adjustement]
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Wed, 5 Dec 2012 09:20:18 +0000 (10:20 +0100)]
Add shelltests for the mon-collector
This commit adds shelltests for the mon-collector binary and for the DRBD
data collector.
Also, it fixes a small bug in the DRBD parser found thanks to the tests.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Tue, 27 Nov 2012 09:16:45 +0000 (10:16 +0100)]
Add the stand-alone data collector for DRBD
It is implemented as a single binary receiving, as its first parameter,
the name of the actual data collector to be run.
This way, its structure can be used for all the future data collectors.
Also, factored out of bdev.py into constants.py the location of the DRBD
status file.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Wed, 12 Dec 2012 08:45:29 +0000 (09:45 +0100)]
Move the path of the DRBD status file to the Constants file
It will be needed by the DRBD data collector, that will be added shortly.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Wed, 12 Dec 2012 09:02:42 +0000 (10:02 +0100)]
Make configure check for haskell vector library
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Helga Velroyen [Wed, 12 Dec 2012 08:38:39 +0000 (09:38 +0100)]
Remove usage of reserved key word 'family'
An old hlint version (1.8.28) was complaining about the
word 'family' being used in the network tests. Thanks to
an epiphany of Iustin this was due to that it is a reserved
key word.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Tue, 11 Dec 2012 17:37:36 +0000 (18:37 +0100)]
Add documentation about new dependency to vector lib
Updates the INSTALL file to add the haskell library
libghc-vector-dev.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Helga Velroyen [Mon, 10 Dec 2012 10:36:27 +0000 (11:36 +0100)]
Network and address pool (again)
Implementation of the network objects and address pool. Functionality
as in the reverted commit
b9a616e132af9bba718d2da8c94eeec7af886814, but
now using only the vector library and not the bit-vector library.
Tested with vector library version 0.9 and 0.10., which are
also installed on the buildbot machines now.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 14:50:45 +0000 (15:50 +0100)]
Fix ordering of entries in tools.ensure_dirs
Commit ebd437a added two new entries to tools.ensure_dirs, but did so in
the wrong order. Patch forthcoming to improve the unittest's error
message.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 13:29:45 +0000 (14:29 +0100)]
Set owner on watcher pause and queue drain files
If the files were created by a user different, e.g. due to a switch from
running masterd as root to running it as a dedicated user, they couldn't
be modified/removed anymore.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 13:23:22 +0000 (14:23 +0100)]
lib/tools/ensure_dirs.py: Code formatting
Wrap lines in a consistent manner (uid/gid on the second line) if
wrapping is necessary at all. “git diff --color-words” shows no
difference at all as only whitespace changed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 12:56:35 +0000 (13:56 +0100)]
Replicate queue drain flag across all master candidates
Until now, the flag was unset on a master failover unless the
“$localstatedir/lib/ganeti/queue/drain” file existed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 15:52:57 +0000 (16:52 +0100)]
Add utility function to create frozenset with unique values
When used instead of a plain call to “frozenset”, this would have
avoided the issue fixed in commit e2dd6ec. The new function is located
in the “compat” module as it will be used at module load time in most
places and should therefore reside in a place with very few
dependencies.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 17:30:51 +0000 (18:30 +0100)]
constants: Remove duplicate DRBD barrier option
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 16:42:43 +0000 (17:42 +0100)]
doc/rapi.rst: Add ECODE_TEMP_NORES
The assertion was failing as ECODE_TEMP_NORES was not documented.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 15:41:42 +0000 (16:41 +0100)]
Change value for ECODE_TEMP_NORES
Unfortunately there was a bug in commit 518a45e whereby ECODE_TEMP_NORES
had the same value as ECODE_NORES, leading to failures in a Haskell
test. Of course this would also have affected other users of the constant.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 15:40:38 +0000 (16:40 +0100)]
Make htools/Ganeti/Constants.hs depend on errors.py
It also uses values from “lib/errors.py”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 14:30:32 +0000 (15:30 +0100)]
masterd: Remove duplicate code
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 13:34:15 +0000 (14:34 +0100)]
Improve error message for when adding inotify watcher fails
Explicitely mention the fs.inotify.max_user_watches sysctl value.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 14:14:24 +0000 (15:14 +0100)]
htools/Ganeti/Errors: Add ECodeTempNoRes
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 12:09:33 +0000 (13:09 +0100)]
Update pyinotify website
It has moved to github.com.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Dato Simó [Thu, 6 Dec 2012 18:43:25 +0000 (18:43 +0000)]
htest/Types.hs: check ordering of some ADTs
In some ADT, ordering of constructors is important because a certain
meaning is attached to ordinality (e.g., "clusters with lesser AllocPolicy
are preferred").
To ensure constructor order is not accidentally changed, this commit
introduces assertions that verify the ordering of these types is as
expected.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 7 Dec 2012 13:17:44 +0000 (14:17 +0100)]
htools: Add missing parameter to OpInstanceMultiAlloc
Somehow this went missing in commit 1f1188c.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 16:57:13 +0000 (17:57 +0100)]
Export error codes from RAPI client module
Until now the error codes were not available from the RAPI client
module. A newly added unit test ensures all error codes are contained in
“ECODE_ALL”, as well as ensuring consistency between the RAPI client and
the authoritative “errors” module.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 15:17:05 +0000 (16:17 +0100)]
Add error code for temporary lack of resources
When an instance creation uses opportunistic locks, the iallocator might
not be able to find an allocation solution if not enough node locks (or
a suboptimal subset thereof) were acquired. As per the design document
<doc/design-opportunistic-locking.rst> a new error code denoting
temporary failures is added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 14:53:55 +0000 (15:53 +0100)]
cmdlib: Use locked nodes as node whitelist
Also actually start using opportunistic locks (if requested).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 7 Dec 2012 12:29:49 +0000 (13:29 +0100)]
cmdlib: Opportunistic locking on instance creation
Adds a new parameter to “OpInstanceCreate” and “OpInstanceMultiAlloc” to
use opportunistic locks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 06:33:26 +0000 (07:33 +0100)]
cmdlib: Node whitelist support for allocation request
Forward the node whitelist to the iallocator plugin.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 26 Nov 2012 06:02:35 +0000 (07:02 +0100)]
mcpu: Verify node allocation lock mode
Add verification code to mcpu to check an LU's locks. Two whitelists are
provided to exclude LUs from the two tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Mon, 26 Nov 2012 09:09:26 +0000 (10:09 +0100)]
_ExportQuery: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 26 Nov 2012 09:09:38 +0000 (10:09 +0100)]
LUBackupExport: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:44:58 +0000 (04:44 +0100)]
LUBackupRemove: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:43:48 +0000 (04:43 +0100)]
LUInstanceMultiAlloc: Use node allocation lock
Avoid conflicts between instance allocations.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:43:08 +0000 (04:43 +0100)]
LUInstanceRecreateDisks: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:42:27 +0000 (04:42 +0100)]
LUNodeSetParams: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:41:52 +0000 (04:41 +0100)]
LUNodeQueryvols: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:41:34 +0000 (04:41 +0100)]
LUOobCommand: Use node allocation lock
If no node names are given, all node locks are acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michele Tartara [Thu, 6 Dec 2012 16:45:40 +0000 (17:45 +0100)]
Fix typo in comment
There was a typo in the docstring of the parseMessage function of the
confd server.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Helga Velroyen [Thu, 6 Dec 2012 18:35:44 +0000 (19:35 +0100)]
Revert "Network and address pool"
This reverts commit
b9a616e132af9bba718d2da8c94eeec7af886814, which
depends on the "bitvec" library. We need to investigate a bit further that
dependency, as it in turns depends on "vector" 0.9.1 or *below*, since
0.10.* removed support for Data.Vector.Unboxed.Safe which it uses.
Signed-off-by: Helga Velroyen <helgav@google.com>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Helga Velroyen [Tue, 27 Nov 2012 09:34:02 +0000 (10:34 +0100)]
Network and address pool
Implementation of the network and address pool class in
Haskell. Not complete yet. Includes unit tests that cover
all functionality that is so far implemented.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 6 Dec 2012 15:43:53 +0000 (16:43 +0100)]
Fix node-daemon-setup test with older pyOpenSSL
Older versions use “-----BEGIN RSA PRIVATE KEY-----” instead of
“-----BEGIN PRIVATE KEY-----”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Wed, 5 Dec 2012 15:02:11 +0000 (16:02 +0100)]
Add tool to clean up node
Sometimes a node is not removed properly from a cluster (especially
during development). This new tool stops all daemons and removes (after
making copies) the most critical files.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 30 Nov 2012 10:34:39 +0000 (11:34 +0100)]
Switch from scp/ssh to node daemon setup utility
This patch does away with many calls to scp and by means of using
“tools/node-daemon-setup”, verifies most of the values before writing
them to files.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 10:43:57 +0000 (11:43 +0100)]
Add tool to configure node daemon
The design for this is in “doc/design-node-add.rst”. The tool receives a
JSON data structure on stdin and configures the node's daemon after
verifying the received values.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Thu, 6 Dec 2012 11:32:16 +0000 (12:32 +0100)]
make genMaybe more Just
There is a common conception that Just something is more worth than
Nothing. So we're biasing our tests towards that. As such let's generate
Nothing fewer times, and Just subgen more times. The values were copied
from the "official" maybe generator.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Dimitris Aragiorgis [Wed, 5 Dec 2012 16:28:14 +0000 (18:28 +0200)]
Locking related fixes for networks
Use GetNetwork() only when having already aquired the lock,
i.e. in CheckPrereq().
In LUNetworkConnect/Disconnect do not include Network info in Hooks
environment, so that network locking can be avoided if conflicts
are not checked.
Invoke _CheckNodeGroupInstances() only in case of conflicts check,
because otherwise owned instances are None, which results to an error.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 6 Dec 2012 08:58:21 +0000 (09:58 +0100)]
Fix build breakage in Jobs.hs test code
Commit
3bdbe4b3 (“Jobs.hs: move OpStatus and JobStatus ADTs to
Types.hs”) removed the TemplateHaskell language pragma from
htest/Test/Ganeti/Jobs.hs due to a hlint warning, but that is bad: it
means the testSuite call is no longer interpreted as a splice, so it results in:
Parse error: naked expression at top level
With newer GHCs. To fix this, we restore the pragma, and add an hlint
ignore.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Dato Simó [Wed, 5 Dec 2012 15:16:02 +0000 (15:16 +0000)]
configure.ac: fix reading of HLINT environment variable
configure.ac advertised that the HLINT environment variable could be set to
specify the path to the 'hlint' tool. However, HLINT was being initialized
to "no" inside configure.ac, which broke this usage.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Dato Simó [Tue, 20 Nov 2012 14:54:11 +0000 (14:54 +0000)]
Hbal.hs: move job execution functions to Jobs.hs
Ganeti.Jobs now holds functions that can be used to submit and monitor the
status of jobs. In particular, execJobsWait and waitForJobs are factored
out of Hbal.hs.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Dato Simó [Wed, 7 Nov 2012 19:40:01 +0000 (19:40 +0000)]
Hbal.hs: use Result (), not Bool, as return value for exec*
Previously, functions in Hbal.hs related to execution of jobsets were
returning only IO Bool, and printing any errors they found directly to
stderr on their own.
I'm going to be moving some of these functions to a library module in
future commits, and it makes sense that they won't print to stderr, but
rather return an error condition. To make diffs more readable, I change the
return value in Hbal.hs itself, so that the next commit deals only with the
move. It's now `main` that prints any Bad result to stderr.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Dato Simó [Thu, 15 Nov 2012 12:45:48 +0000 (12:45 +0000)]
Hbal.hs: rename functions that deal with early termination
In Hbal.hs, it's easy to get lost in the flow of exec* functions
because their names are similar and don't convey their different
purposes (e.g. runJobSet, execJobSet, execWrapper).
This patch renames 'runJobSet' to 'execWithCancel', and 'execWrapper'
to 'execCancelWrapper', since these two functions deal, in particular,
with early termination when the user presses ^C.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Dato Simó [Wed, 7 Nov 2012 17:38:01 +0000 (17:38 +0000)]
Hbal.hs: fix typo in signal handler names (hangle -> handle)
'hangleSigInt' and 'hangleSigTerm' are renamed to 'handleSigInt' and
'handleSigTerm', respectively.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Dato Simó [Tue, 20 Nov 2012 14:34:48 +0000 (14:34 +0000)]
Jobs.hs: move OpStatus and JobStatus ADTs to Types.hs
This leaves Ganeti/Jobs.hs and Test/Ganeti/Jobs.hs empty, but they're the
target of a future move of some functions, so we leave them around, and
don't delete them, to avoid unnecessary delete/create diffs.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 5 Dec 2012 16:41:54 +0000 (17:41 +0100)]
Makefile: Reset environment for epydoc
epydoc allows overriding configuration values via environment variables.
While this might be useful in certain use cases, but as no prefix
whatsoever is used, conflicts are easily created. Some people have the
environment variable “NAME” set, effectively overriding the project name
set in epydoc.conf. A bug in epydoc causes an error if non-ASCII
characters, such as German umlauts, are used in NAME.
$ NAME=Täscht make py-apidoc
[…]
UNEXPECTED ERROR:
'ascii' codec can't decode byte 0xc3 in position 73: ordinal not in range(128)
$ parse=false make py-apidoc
[…]
epydoc: error: Invalid option combination: --parse-only and --introspect-only.
This patch changes the call in Makefile to reset the environment given
to epydoc save for PATH and PYTHONPATH.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Wed, 5 Dec 2012 12:42:45 +0000 (13:42 +0100)]
jqueue: Don't modify input opcode when changing priority
Commit 4679547 implemented the ability to change job's priority after it
was submitted. The code contained a bug whereby it would modify the
input data for an opcode, something the job queue shouldn't do (logical
units do for historical reasons).
This patch removes the line modifying the opcode input and adjusts the
tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Dimitris Aragiorgis [Wed, 5 Dec 2012 14:08:10 +0000 (16:08 +0200)]
Use new util function for mac_prefix validation
Use new NormalizeAndValidateThreeOctetMacPrefix() util function in
LUNetworkAdd/LUNetworkSetParams to validate network's MAC prefix.
Additionally, move the check in CheckArguments() in the case of
LUNetworkAdd.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:41:15 +0000 (04:41 +0100)]
LUClusterRedistConf: Use node allocation lock
All node locks are acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:40:17 +0000 (04:40 +0100)]
LUClusterRepairDiskSizes: Use node allocation lock
This opcode acquires all node resource locks, which conflicts with
instance allocations.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:39:46 +0000 (04:39 +0100)]
LUGroupVerifyDisks: Use node allocation lock
See comment in code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:39:20 +0000 (04:39 +0100)]
LUClusterVerifyGroup: Use node allocation lock
See comment in code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 01:47:13 +0000 (02:47 +0100)]
LUInstanceReplaceDisks: Acquire node allocation lock
If the lock was acquired in the first place (only when an iallocator is
used), it is released as soon as possible.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Thu, 22 Nov 2012 23:08:32 +0000 (00:08 +0100)]
LUInstanceChangeGroup: Acquire node allocation lock
Changing instances' groups shouldn't conflict with instance allocations.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Thu, 22 Nov 2012 22:29:07 +0000 (23:29 +0100)]
Acquire node allocation lock during node query
If locking is used (usually by ganeti-watcher), node allocations must be
temporarily blocked.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Wed, 5 Dec 2012 10:36:22 +0000 (11:36 +0100)]
Mention dsh/cssh in man page for "gnt-cluster command"
Until now they are only mentioned in “doc/admin.rst” and the man page
for “ganeti-listrunner”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 01:56:04 +0000 (02:56 +0100)]
iallocator: Add node whitelist
In the future instance creations might have a lock on all nodes as was
the case until the implementation of opportunistic locking. Nodes for
which the lock is not held will be shown to the iallocator plugin as if
they were marked offline.
This patch adds a new parameter named “node_whitelist” to
“IAReqInstanceAlloc”. If set to a list, only nodes contained within are
shown as online.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 17:10:15 +0000 (18:10 +0100)]
Allow ignoring successful commands in "gnt-cluster command"
In some cases it is useful to ignore the output of and avoid mentioning
successful commands. One would be when looking for a certain string in
a file:
$ gnt-cluster command egrep -q '^testing$' /etc/...
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Guido Trotter [Mon, 3 Dec 2012 13:13:34 +0000 (14:13 +0100)]
Add "proper coloring" unittest check
We have to check that for each edge its vertices have different colors.
This is very easy to do with a vertex-to-color map, but not so easy with
a color-to-vertex one. Since all our coloring algorithms created a
vertex-to-color map behind the scenes and then converted it, we flip
them back to returning it directly, and do the conversion explicitly
where we need it (which for now is everywhere except when testing this
property).
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 3 Dec 2012 08:25:32 +0000 (09:25 +0100)]
Fix Dsatur and add Dcolor
Our Dsatur implementation was incorrect: while the paper defined the
degree of saturation of a vertex as the number of different colors it is
adjacent to, we were using the number of colors, without considering
uniqueness. This effectively implemented a different algorithm, which is
very similar to the previous one, and while it performs slightly worse
on average it still beats Dsatur on some cases.
So we refactor the implementation to effectively support both algorithms
without code duplication, and then we export both the old algorithms as
"Dcolor" and the new one as "Dsatur". Since these are all fast
algorithms in hroller we will still be able to pick the best result.
Note that the new Dsatur implementation uses an IntSet to calculate the
uniqueness. Results with nub + length on a list were significantly
slower.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Sat, 1 Dec 2012 09:45:02 +0000 (10:45 +0100)]
HTools/Node: add mkNodeGraph function
This function helps treating node node problems as graph problems.
It can transform a list of nodes plus a list of instances into a graph
which uses the nodes as vertices, and instances as edges connecting them
(as long as they have both a primary and a secondary node)
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Thu, 29 Nov 2012 08:57:59 +0000 (09:57 +0100)]
Add tests for verticesByDegree{Asc,Desc}
This brings our coverage of Graph.hs to 100%
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 28 Nov 2012 09:28:53 +0000 (10:28 +0100)]
Ganeti/HTools/Graph Add isColorable
Check whether coloring on a given graph makes sense. This is the case
only if there are no loops and the graph is undirected.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Fri, 30 Nov 2012 17:15:29 +0000 (18:15 +0100)]
Add Dsatur implementation
Implement the Dsatur algorithm for Graph coloring. This also abstracts
the neighColors function into two subfunctions that this algorithm can
reuse.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Sat, 24 Nov 2012 10:47:05 +0000 (11:47 +0100)]
Add Ganeti.HTools.Graph
This module implements some algorithms on Data.Graph data structures.
At the moment its main functionality is an LF-color implementation
(greedy coloring in descending order of degree). There are also a few
extra functions to calculate the degree order, and convert the node to
color mapping to color to nodes.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 15:06:23 +0000 (16:06 +0100)]
errors: Show error descriptions in API documentation
Comments with a colon after the hash sign (“#:”) show up in the
epydoc output.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Fri, 30 Nov 2012 00:30:18 +0000 (01:30 +0100)]
Small improvement to rpc-test
This will show smaller/better error messages: full node dumps are no
longer included.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 12:38:55 +0000 (13:38 +0100)]
Fix locking mistake introduced in commit 5cc1f88
The node resource locks were not set correctly on instance import.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 11:22:00 +0000 (12:22 +0100)]
Switch luxi submit job calls to use MetaOpCode
This patch changes the luxi submit job calls to use wrapped opcodes,
and therefore it changes Hbal to submit actual meta opcodes. For
nicety, hbal also submits a comment now, showing who generated the
job.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 11:41:15 +0000 (12:41 +0100)]
hbal: restore nice job ID display
This fixes:
Got job IDs JobId {fromJobId = 1052613}
And restores to:
Got job IDs 1052624
Other improvements could be done here, of course.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 10:59:52 +0000 (11:59 +0100)]
Add safety check on job dependency/TIsLength
If TIsLength is applied to a non-container item, it will fail (type
error) due to invalid application of len(). Since this can happen on
user-supplied data, we add an explicit TList/TTuple check (the TTuple
test is a new one).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 10:30:12 +0000 (11:30 +0100)]
Add CommonOpParams and MetaOpCode types
This patch adds the "meta" opcode type and the common op
params. Compatibility tests with Python are changed to pass Meta
opcodes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 10:19:39 +0000 (11:19 +0100)]
Remove unused THH function 'saveLuxiField'
This was deprecated via commit
88609f00, “Switch Luxi TH code from
simple to custom fields”.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 4 Dec 2012 09:55:02 +0000 (10:55 +0100)]
Change opcode/luxi showJSON generation in THH
Currently, the opcode and luxi "showJSON" functions generate directly
a JSValue; in contrast, the object (single-constructor) types generate
a 'toDict' function, and then `showJSON = makeObj . toDict`. This is
useful, as the 'dict' form can be manipulated if needed.
This patch changes the opcode and luxi types to behave the same; we
generate a dict, and then (since this differs between opcodes and
luxi) generate showJSON either as `makeObj . toDict` (for opcodes), or
(for luxi) `showJSON . map sn . toDict`.
The change is needed for MetaOpCode implementation.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 09:41:13 +0000 (10:41 +0100)]
Add a type alias for simpler THH signatures
This is reused in more than just a few places, so adding it makes the
signatures much nicer.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>