Iustin Pop [Tue, 18 Dec 2012 13:41:28 +0000 (14:41 +0100)]
Improve mon-collector drbd CLI handling
Currently, this doesn't abort if multiple arguments are passed, and
the error message for errors during file read has a duplicate "Error".
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 17 Dec 2012 16:31:20 +0000 (17:31 +0100)]
Enable bash completion for mon-collector
This just ties in all the pieces so far and enables bash completion
for it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 17 Dec 2012 16:25:18 +0000 (17:25 +0100)]
bash-completion: add support for multi-cmd Haskell binaries
This patch adds support for parsing the command list out of a binary
(very strict format), and then iterating over that and building the
sub-commands options/arguments.
It also does a bit of general cleanup in the script.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 17 Dec 2012 14:34:51 +0000 (15:34 +0100)]
Implement fake completion support for multi-cmd binaries
This will allow build-bash-completion to understand what is the list
of commands that each binary supports, and then provide recursively
completion support for each command.
Note: by "fake", I mean we use hand-built output and hardcoded
"--help-completion" matching, as opposed to automatically doing it via
regular options. It could be possible in the future to use a
two-staged command line parser, but for now not sure if it's worth it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 17 Dec 2012 14:22:24 +0000 (15:22 +0100)]
Simplify htools.hs personality selection
No need to do a boolean-based lookup, we can use the regular
association list lookup.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 17 Dec 2012 15:15:01 +0000 (16:15 +0100)]
Add description to personality definitions
This allows usage information to display nicer help (like in Python).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 17 Dec 2012 13:44:28 +0000 (14:44 +0100)]
Introduce generic multi-command binary handling
Currently, the mon-collector binary is the only multi-command binary
we have in Haskell. Not surprisingly therefore, its command line
handling is not as robust as we need, not being able to support
standard "--help" commands.
This patch introduces generic multi-command handling, similar to the
Python functionality (some parts missing, will be added in future
patches), and switches mon-collector over to it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Fri, 14 Dec 2012 22:51:14 +0000 (23:51 +0100)]
Add a type alias for the personality lists
The signature of the personality definitions is so ugly that, at
least, we should hide it a bit behind a type alias.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Fri, 14 Dec 2012 21:03:49 +0000 (22:03 +0100)]
Add a custom CLI module for DataCollectors
Currently, the mon-collector binary uses the HTools/CLI module, which
is OK but mean it links in lots of htools code. By copying that module
to DataCollectors/CLI and removing the unneeded code, we reduce the
number of modules it depends on fro 20 to 12, meaning both a shorter
compiler time (24s to 9s) and a smaller binary (~9.6MB to ~7MB).
Also fixes a typo in the original HTools/CLI module, thanks Michele!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Fri, 14 Dec 2012 19:39:18 +0000 (20:39 +0100)]
Fix profiling targets
As noted by Guido, there are problems when using the hs-prof and
hs-prof-quick targets in the default configuration (compiling all
programs). The errors manifest in the form of wrong symbols during
compilation.
I knew that the hs-prof targets, which compiled multiple objects with
the '.o' suffix, could be problematic; but the objects that are
actually needed in the Template Haskell phase are very standard and
don't differ between the binaries (Constants, BasicTypes, JSON,
THH). What I didn't realise (although it's obvious) is that also the
hs_prof_quick targets (the final binaries) are also compiled with a
single suffix ('.prof_o'), which means that the object files are
actually compiled for the last binary.
This means that targets later in the HS_ALL_PROGS list would work
correct, but early targets, especially htools/htools, would fail.
So the obvious, and the single simple solution is to make these two
rules only work on a single binary at a time. This should be fine,
since one is looking at a specific problem usually, and it has the
advantage that the hs-prof step is much faster (since it wasn't
buildable in parallel anyway).
Thanks to Guido for finding and making the initial diagnose on this!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Wed, 19 Dec 2012 09:28:01 +0000 (10:28 +0100)]
Allow shutting down offline instance
If an instance is offline we definitely shouldn't start it up.
But shutting it down, should it be up by mistake is not "that" bad.
Still, we only allow it with --force, as it still performs an action on
an instance we shouldn't touch. This should make everybody happy.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 19 Dec 2012 09:05:26 +0000 (10:05 +0100)]
Allow running instances to be put online
If an instance is running (eg. ERROR_up) and at the same time offline,
there's no way to either shut it down or reonline it. This allows
onlining it. Offlining is still disabled for running instances.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Bernardo Dal Seno [Mon, 8 Oct 2012 23:43:17 +0000 (01:43 +0200)]
Fixes and improvements to comments
Some fixes, added more information in a few points, removed a stale (5+
year old) TODO comment.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 18 Dec 2012 13:08:00 +0000 (14:08 +0100)]
Explain how to use GHCI to see spliced TH code
This can be more useful than the "make" version.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 12:19:18 +0000 (13:19 +0100)]
Fix error during cluster initialization due to hv_kvm
Commit 141d148 was a bit too enthusiastic. The three parameters added to
the list of parameters to be checked default to a value not evaluating
to false, leading to a failure on cluster initialization.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Dimitris Aragiorgis [Tue, 18 Dec 2012 09:23:27 +0000 (11:23 +0200)]
Make LUNetworkAdd pass _VerifyLocks()
LEVEL_NODE_ALLOC should be aquired too if LEVEL_NODE is ALL_SET.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Helga Velroyen [Thu, 13 Dec 2012 13:02:28 +0000 (14:02 +0100)]
Haskell/python compatibility test for networks
This patch contains the HUnit test that checks the compatibility
of Haskell-generated networks with the python code. For that the
generation of test instances of networks was enhanced to meet
the validation steps of the python implementation. Also, so far
networks were generated at two different places in the code; this
is now consolidated.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 17 Dec 2012 16:16:27 +0000 (17:16 +0100)]
Add test for SPICE parameter list, add missing ones
“_SPICE_ADDITIONAL_PARAMS” is supposed to be the full list of
SPICE-related KVM hypervisor parameters with the exception of
“HV_KVM_SPICE_BIND”. The new test checks if all parameters starting with
“HV_KVM_SPICE_*” are included. Three previously missing parameters are
added to the list.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 17 Dec 2012 16:06:24 +0000 (17:06 +0100)]
hv_kvm: List of SPICE parameters should be module-global
This list is static at runtime and doesn't need to be recreated every
time.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 17 Dec 2012 15:51:27 +0000 (16:51 +0100)]
_DeclareLocksForMigration: Fix non-DRBD locking issue
When non-DRBD disks are used for an instance,
“lu.needed_locks[locking.LEVEL_NODE]” is set to “locking.ALL_SET” (which
is None). The assertion will then fail as None evaluates to False.
Reported by Constantinos Venetsanopoulos.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 17 Dec 2012 13:41:11 +0000 (14:41 +0100)]
LUOobCommand: Always declare NAL in shared mode
Doing so avoids triggering an assertion in mcpu which cross-checks the
node lock and node allocation lock acquisition mode.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 12 Dec 2012 03:40:46 +0000 (04:40 +0100)]
Fix two logging messages in TLReplaceDisks
Commit
f0f8d060 (“Show old primary/secondary node on disk
replacement”) added two wrong uses of feedback_fn, which results in
log entries such like these:
"log": [
[
7, [
1351258326, 466214], "message", "Replacing disk(s) 0 for instance 'instance1.example.com'"
],
[
8, [
1351258326, 749391], "Current primary node: %s", "node1"
],
[
9, [
1351258326, 774829], "Current seconary node: %s", "node4"
],
[
10, [
1351258329, 620410], "message", "STEP 1/6 Check device existence"
],
(the first and fourth are OK, the 2nd and 3rd are broken).
The API of jqueue._OpExecCallbacks.Feedback should probably be changed
to prevent such an occurrence in the future.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Tue, 11 Dec 2012 00:37:10 +0000 (01:37 +0100)]
Split and extend Confd types tests
Some confd types were not tested for serialisation, so let's move
these (type-related) tests to their own module and extend them.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 01:50:33 +0000 (02:50 +0100)]
Switch Luxi sendMsg from strict to lazy ByteStrings
Commit
e821050d (“Switch the Luxi interface from Strings to
ByteStrings”) was designed to optimise the receive interface, but has
an unfortunate side-effect: when sending non-trivial messages, it
means that both the entire String and the ByteString versions must be
in memory at the same time, leading to much increased memory usage.
By changing the "hPut" from strict to lazy ByteStrings, it means that
both the String and the ByteString values can be evaluated lazily,
with significant effects: for a test query answer, instead of having
a peak from ~600MB to 1.4G during the entire Luxi send operation,
memory consumption actually decreased during the send operation, as
the ByteString chunks are released individually and even the backing
storage of the items that create the JSON string serialisation is
released lazily as well. So instead of slow growth 10→550MB then quick
peak to 1.4GB during Luxi send, we now have an even slower growth to
~580MB and then decrease of memory as the Luxi send progresses.
The only downside is of a small increase in CPU time of a few percents
for the above case; for our use cases, I think this is much desirable.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 00:46:08 +0000 (01:46 +0100)]
Add NFData instances for query results
This allows us to ensure that query results are strict as we build
them, instead of being lazy and only evaluated when the reply is sent
over the Luxi interface.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 00:29:28 +0000 (01:29 +0100)]
Introduce NFData instances for JSON types
The JSValue/JSObject types don't come with a NFData instance, so let's
add one ourselves, so that we can force evaluation of JSValues (either
when building or when reading them).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 03:55:18 +0000 (04:55 +0100)]
Fix parsing of absolute job dependencies
It seems that Python code generates (sometimes) absolute job
dependencies which are strings, instead of integers, so we should be
able to parse these as well. We simply change from explicit int-based
parsing (makeJobId) to the generic one (parseJobId).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 03:27:46 +0000 (04:27 +0100)]
Make two common opcode parameters always serialised
This changes deps and comment fields to always be shown, to match the
Python behaviour for (at least) job listing/ops field.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 03:20:01 +0000 (04:20 +0100)]
Fix disk_template OpInstSetParams parameter
This is not a required field, but rather an optional one; we add a new
parameter and use it instead.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 23:27:21 +0000 (00:27 +0100)]
Support 'null' in Luxi QueryJobs names field
Python code sometimes sends this, so let's support it even though it's
non-standard.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 18:35:09 +0000 (19:35 +0100)]
Abstract code related to job ID creation
Having makeJobIdS as a separate function will allow us to use it
outside of json encoding.
The patch also exports one more function from the Types module.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Tue, 11 Dec 2012 00:11:39 +0000 (01:11 +0100)]
Add new Haskell library dependency for tests
The 'temporary' library add support for create temporary
directories/files beyond what is offered in the base library (which is
somewhat limited, and hard to use in tests).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 10 Dec 2012 18:32:27 +0000 (19:32 +0100)]
Reduce logging level in tests
This allows tests to execute IO actions which have logging as a side
effect, without polluting the stderr too much.
A better solution would be that we have fine-grained control over
loggers, so that tests can run with their own logging, etc. etc..
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 10 Dec 2012 16:25:30 +0000 (17:25 +0100)]
Export jstore constants as well to Constants.hs
This single constant could be as well moved to constants.py. For now I
choose to just export the module as well.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 10 Dec 2012 16:23:54 +0000 (17:23 +0100)]
Generalise the test helper 'genUniquesList'
Instead of always using "arbitrary", make it require a generator (that
can then use more complex rules for building the list elements), and
also convert it to use a temporary set instead of list membership.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 10 Dec 2012 10:06:21 +0000 (11:06 +0100)]
Export more paths from Path.hs
Also adds a small helper for building the paths.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 10 Dec 2012 08:46:43 +0000 (09:46 +0100)]
Simplify a bit the FilterRegex JSON instance
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 08:32:13 +0000 (09:32 +0100)]
Implement opcode summary support
This implements in the Haskell codebase the opcode summary. As opposed
to Python, we always use custom code for formatting, since we don't
want to use dynamic attribute lookup.
To test this properly, we need to change MetaOpCode to record-syntax,
so that we get accessor functions.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 08:27:07 +0000 (09:27 +0100)]
Add optional formatting for OP_DSC_FIELD
For some opcodes, the output is not "stable", and depends on the exact
input values; this makes it harder to check consistency against
Haskell code.
To compensate for this, we add a way to override the formatting of the
OP_DSC_FIELD; by default, this is always "%s", but if the
OP_DSC_FORMATTER is defined (must be a callable), it is used to format
the actual value.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Dec 2012 15:13:18 +0000 (16:13 +0100)]
Add a type for the log type in job/opcode logs
Also fixes indentation for OP_STATUS.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Dec 2012 14:23:35 +0000 (15:23 +0100)]
Fix small typo in OpTestDelay/duration definition
The extra space results in inaccessible names; currently GHC doesn't
flag this as an error, but I've filled an upstream bug for that
(http://hackage.haskell.org/trac/ghc/ticket/7484).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 14 Dec 2012 00:31:53 +0000 (01:31 +0100)]
Add constants for two values which we re-evaluate
Two constants which we use as Integer are defined as Int in
Constants.hs (coming from constants.py), so we do the conversion every
time we use it. Let's move them to top-level names, so that the
conversion is only done once.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 13 Dec 2012 21:33:47 +0000 (22:33 +0100)]
Fix an small but quite nasty typo
Introduced in commit
d4752785.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Mon, 13 Dec 2010 11:57:25 +0000 (11:57 +0000)]
update gnt-instance manpage about self-rename
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 13 Dec 2010 11:53:39 +0000 (11:53 +0000)]
burnin: rename instances to same name
this (a) allows us to test renaming in burnin even if we don't have an
extra instance name and (b) is a valid operation, as it allows to "fix"
an instance who went name-astray
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 20 Jun 2012 14:03:37 +0000 (15:03 +0100)]
Very very very basic openvswitch support
This is a "better than nothing" support, just for kvm and just joining
the machine to the opevswitch bridge with the right command.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Guido Trotter [Wed, 12 Dec 2012 14:29:50 +0000 (06:29 -0800)]
Improve a few Graph test properties
Return type is changed from Property to Bool, and the ==? True at the
end is dropped.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 12 Dec 2012 14:27:19 +0000 (06:27 -0800)]
remove duplicate code in Graph.hs
Also update the docstring of a function.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 12 Dec 2012 12:43:17 +0000 (04:43 -0800)]
Add profiling dependencies note
As discussed on the list, these are needed to do a make hs-prof.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 13 Dec 2012 17:05:26 +0000 (18:05 +0100)]
Log only partial response in Luxi when in debug mode
Currently, we log the entire response (at debug level) in the Luxi
replies. This is not a good idea; the logging library operates on
strings, and as such it will use huge amounts of memory: without debug
logging, a certain gnt-job list invocation uses 295MB (+RTS -s) and
2m35s time, when in debug mode, it's 1525MB and 48m!
So we make two changes:
- first, we switch from "show (pp_value a)" to "encode a", which
generates a non-formatted string rather than a indented one
- second we log only the first 2000 characters; this should be enough
to understand the first part of the response
We could go for higher, or for logging in batches (that would be
faster, as well).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Constantinos Venetsanopoulos [Thu, 13 Dec 2012 16:06:05 +0000 (18:06 +0200)]
Pass check_ip and not hostname.ip to _ComputeNics
This should be done because in the case of --no-name-check there
is no 'hostname.ip' attribute, causing an execution error.
'check_ip' is always set (in CheckArguments) even if --no-name-check
is passed in the command line.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Dec 2012 15:03:30 +0000 (16:03 +0100)]
Add version check for doc/design-draft.rst
This is similar to README or “doc/security.rst”. Having a comment that
needs updating for a new version is more likely going to trigger a check
of the lists of design drafts.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Dec 2012 15:03:25 +0000 (16:03 +0100)]
Makefile/check-local: List all errors before failing
Use the “error” variable in multiple places and fail only once all tests
have been completed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Dec 2012 14:49:51 +0000 (15:49 +0100)]
Move four designs from draft to 2.7
These are the ones I implemented and for which I'm certain they're
complete.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Dec 2012 14:20:34 +0000 (15:20 +0100)]
RAPI documentation fix and update
- Mention user capabilities in security.rst
- Replace “query” with “read” in RAPI documentation
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Thu, 13 Dec 2012 11:48:55 +0000 (12:48 +0100)]
Add hpc-mon-collector to .gitignore
It is a binary file and it should never be added to the git repository.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 17:00:25 +0000 (18:00 +0100)]
Read watcher pause using RPC, not directly
The master daemon should not directly read files written by the node
daemon. This patch adds a new RPC to read the watcher pause file and
changes the master code to use it.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 15:48:03 +0000 (16:48 +0100)]
Set watcher pause on all nodes
Instead of just setting the watcher pause file, which isn't replicated,
RPC is used to set it on all nodes (where possible). This means that
after an outage of the master node and a following master-failover, the
watcher will still be paused.
The RPC is attempted on all nodes. Errors for nodes marked offline are
ignored. All other errors are returned in the form of an exception
(there is no other option given the LUXI API).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 16:17:20 +0000 (17:17 +0100)]
Add RPC for setting watcher pause
The watcher pause file should be set/unset on all nodes at once, not
only the master node. For that a new RPC is needed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 16:03:38 +0000 (17:03 +0100)]
Generated RPC code: Don't write whitespace after paren
There is no obvious need for this space--the line would wrap at the next
space, which comes soon after.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 16:09:44 +0000 (17:09 +0100)]
Rename test for backend._CommonRestrictedCmdCheck
“TestWriteFile” was not renamed when adding the file based on
“ganeti.utils.io_unittest-runasroot.py”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 13:32:13 +0000 (14:32 +0100)]
jqueue: Improve inotify error reporting
This addresses issue 218. When the number of inotify watches is
exhausted, for example by being set too low from the beginning or by
other programs, waiting for a job to change would just report a lost job
(e.g. “Error checking job status: Job with id 7817 lost”).
This patch changes the job watcher to no longer catch
“errors.InotifyError” and, this is by far the larger part of this patch,
adds unittests for this situation.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 14:52:56 +0000 (15:52 +0100)]
Improve test for tools.ensure_dirs
- Add more checks, some of them are deliberately redundant
- Descriptive error messages
- Add comment describing order to “tools.ensure_dirs”
- Avoid copying a list in an assertion in “tools.ensure_dirs”
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Dato Simó [Wed, 12 Dec 2012 15:32:46 +0000 (15:32 +0000)]
Jobs.hs: make L.Client the last argument for all functions
Having the Luxi client be the last argument of the functions allows for
easier use with `Exception.bracket L.getClient L.closeClient ...`.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Wed, 12 Dec 2012 13:46:27 +0000 (14:46 +0100)]
Fix a bug in the hs-coverage rule of Makefile.am
The coverage information for mon-collector was not merged correctly.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 12 Dec 2012 10:00:23 +0000 (11:00 +0100)]
uidpool: Remove roman number support
Doing so simplifies to code a bit and never had a practical use.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Constantinos Venetsanopoulos [Tue, 4 Dec 2012 11:50:16 +0000 (13:50 +0200)]
Remove checks wrt IDISK_PARAMS from OpCode level
Change the "--disks" option validation, to just check the format
of the dict and do not check whether the keys are included in the
IDISK_PARAMS constant at OpCode level. This allows the passing of
arbitrary parameters at the CLI, which will then be logically
checked at LU level (inside CheckArguments).
Currently, the check wrt to IDISK_PARAMS already happens in all
related LUs (InstanceSetParams, InstanceCreate, InstanceRecreateDisks),
with the use of ForceDictType(), so there is no need for additional
checking code.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed a unittest that needed adjustement]
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Wed, 5 Dec 2012 09:20:18 +0000 (10:20 +0100)]
Add shelltests for the mon-collector
This commit adds shelltests for the mon-collector binary and for the DRBD
data collector.
Also, it fixes a small bug in the DRBD parser found thanks to the tests.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Tue, 27 Nov 2012 09:16:45 +0000 (10:16 +0100)]
Add the stand-alone data collector for DRBD
It is implemented as a single binary receiving, as its first parameter,
the name of the actual data collector to be run.
This way, its structure can be used for all the future data collectors.
Also, factored out of bdev.py into constants.py the location of the DRBD
status file.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Wed, 12 Dec 2012 08:45:29 +0000 (09:45 +0100)]
Move the path of the DRBD status file to the Constants file
It will be needed by the DRBD data collector, that will be added shortly.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Wed, 12 Dec 2012 09:02:42 +0000 (10:02 +0100)]
Make configure check for haskell vector library
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Helga Velroyen [Wed, 12 Dec 2012 08:38:39 +0000 (09:38 +0100)]
Remove usage of reserved key word 'family'
An old hlint version (1.8.28) was complaining about the
word 'family' being used in the network tests. Thanks to
an epiphany of Iustin this was due to that it is a reserved
key word.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Tue, 11 Dec 2012 17:37:36 +0000 (18:37 +0100)]
Add documentation about new dependency to vector lib
Updates the INSTALL file to add the haskell library
libghc-vector-dev.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Helga Velroyen [Mon, 10 Dec 2012 10:36:27 +0000 (11:36 +0100)]
Network and address pool (again)
Implementation of the network objects and address pool. Functionality
as in the reverted commit
b9a616e132af9bba718d2da8c94eeec7af886814, but
now using only the vector library and not the bit-vector library.
Tested with vector library version 0.9 and 0.10., which are
also installed on the buildbot machines now.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 14:50:45 +0000 (15:50 +0100)]
Fix ordering of entries in tools.ensure_dirs
Commit ebd437a added two new entries to tools.ensure_dirs, but did so in
the wrong order. Patch forthcoming to improve the unittest's error
message.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 13:29:45 +0000 (14:29 +0100)]
Set owner on watcher pause and queue drain files
If the files were created by a user different, e.g. due to a switch from
running masterd as root to running it as a dedicated user, they couldn't
be modified/removed anymore.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 13:23:22 +0000 (14:23 +0100)]
lib/tools/ensure_dirs.py: Code formatting
Wrap lines in a consistent manner (uid/gid on the second line) if
wrapping is necessary at all. “git diff --color-words” shows no
difference at all as only whitespace changed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 11 Dec 2012 12:56:35 +0000 (13:56 +0100)]
Replicate queue drain flag across all master candidates
Until now, the flag was unset on a master failover unless the
“$localstatedir/lib/ganeti/queue/drain” file existed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 15:52:57 +0000 (16:52 +0100)]
Add utility function to create frozenset with unique values
When used instead of a plain call to “frozenset”, this would have
avoided the issue fixed in commit e2dd6ec. The new function is located
in the “compat” module as it will be used at module load time in most
places and should therefore reside in a place with very few
dependencies.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 17:30:51 +0000 (18:30 +0100)]
constants: Remove duplicate DRBD barrier option
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 16:42:43 +0000 (17:42 +0100)]
doc/rapi.rst: Add ECODE_TEMP_NORES
The assertion was failing as ECODE_TEMP_NORES was not documented.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 15:41:42 +0000 (16:41 +0100)]
Change value for ECODE_TEMP_NORES
Unfortunately there was a bug in commit 518a45e whereby ECODE_TEMP_NORES
had the same value as ECODE_NORES, leading to failures in a Haskell
test. Of course this would also have affected other users of the constant.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 15:40:38 +0000 (16:40 +0100)]
Make htools/Ganeti/Constants.hs depend on errors.py
It also uses values from “lib/errors.py”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 14:30:32 +0000 (15:30 +0100)]
masterd: Remove duplicate code
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 13:34:15 +0000 (14:34 +0100)]
Improve error message for when adding inotify watcher fails
Explicitely mention the fs.inotify.max_user_watches sysctl value.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 14:14:24 +0000 (15:14 +0100)]
htools/Ganeti/Errors: Add ECodeTempNoRes
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 12:09:33 +0000 (13:09 +0100)]
Update pyinotify website
It has moved to github.com.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Dato Simó [Thu, 6 Dec 2012 18:43:25 +0000 (18:43 +0000)]
htest/Types.hs: check ordering of some ADTs
In some ADT, ordering of constructors is important because a certain
meaning is attached to ordinality (e.g., "clusters with lesser AllocPolicy
are preferred").
To ensure constructor order is not accidentally changed, this commit
introduces assertions that verify the ordering of these types is as
expected.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 7 Dec 2012 13:17:44 +0000 (14:17 +0100)]
htools: Add missing parameter to OpInstanceMultiAlloc
Somehow this went missing in commit 1f1188c.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 16:57:13 +0000 (17:57 +0100)]
Export error codes from RAPI client module
Until now the error codes were not available from the RAPI client
module. A newly added unit test ensures all error codes are contained in
“ECODE_ALL”, as well as ensuring consistency between the RAPI client and
the authoritative “errors” module.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 15:17:05 +0000 (16:17 +0100)]
Add error code for temporary lack of resources
When an instance creation uses opportunistic locks, the iallocator might
not be able to find an allocation solution if not enough node locks (or
a suboptimal subset thereof) were acquired. As per the design document
<doc/design-opportunistic-locking.rst> a new error code denoting
temporary failures is added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 4 Dec 2012 14:53:55 +0000 (15:53 +0100)]
cmdlib: Use locked nodes as node whitelist
Also actually start using opportunistic locks (if requested).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 7 Dec 2012 12:29:49 +0000 (13:29 +0100)]
cmdlib: Opportunistic locking on instance creation
Adds a new parameter to “OpInstanceCreate” and “OpInstanceMultiAlloc” to
use opportunistic locks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 06:33:26 +0000 (07:33 +0100)]
cmdlib: Node whitelist support for allocation request
Forward the node whitelist to the iallocator plugin.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 26 Nov 2012 06:02:35 +0000 (07:02 +0100)]
mcpu: Verify node allocation lock mode
Add verification code to mcpu to check an LU's locks. Two whitelists are
provided to exclude LUs from the two tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Mon, 26 Nov 2012 09:09:26 +0000 (10:09 +0100)]
_ExportQuery: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 26 Nov 2012 09:09:38 +0000 (10:09 +0100)]
LUBackupExport: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:44:58 +0000 (04:44 +0100)]
LUBackupRemove: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:43:48 +0000 (04:43 +0100)]
LUInstanceMultiAlloc: Use node allocation lock
Avoid conflicts between instance allocations.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:43:08 +0000 (04:43 +0100)]
LUInstanceRecreateDisks: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>