Iustin Pop [Wed, 28 Nov 2012 08:33:06 +0000 (09:33 +0100)]
Add a negative type
This mirrors the positive one, and will be needed for relative job
IDs.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 27 Nov 2012 15:45:39 +0000 (16:45 +0100)]
Add a 'real' type for JobIds
Currently, the job ID is a simple type alias. This is suboptimal, as
it means we can't use a custom JSON (or Arbitrary) instance for it.
The patch changes it into a newtype, and then a) simplifies some
deserialisation code and b) changes some more fields to this new type
(rather than plain 'Int').
We also move the JobId to types, since it will be needed in opcodes as
well.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 27 Nov 2012 14:57:24 +0000 (15:57 +0100)]
Add type for finalised job statuses
For now, we don't need a pending job status type as well, so we'll
delay adding that until later.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 26 Nov 2012 15:55:26 +0000 (16:55 +0100)]
Add test for opcode fields equivalence
As opposed to the existing test, which tests the type/serialisation of
fields, this one simply tests the equivalence of the list of fields
for each opcode.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 12:53:01 +0000 (13:53 +0100)]
Fix Haskell OpNetworkAdd
Commit 213076f (“Fix locking in networks”) changed Python OpNetworkAdd
without corresponding Haskell definition changes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 26 Nov 2012 15:50:02 +0000 (16:50 +0100)]
Add a few missing fields from opcodes
Due to manual conversion, a few fields were missing from the
conversion, but as they were optional our type equivalence checking
didn't detect this.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 26 Nov 2012 15:44:56 +0000 (16:44 +0100)]
Rework/enhance original htools opcodes
The original htools opcodes were minimalistic and not 1:1 equivalent
with the Python ones. Let's add all missing fields and, since we
changed the order, switch to more readable record syntax for building
the opcodes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 26 Nov 2012 13:38:35 +0000 (14:38 +0100)]
Add a function for all fields of a given OP_ID
This patch changes THH to export a new function which defines all
fields of a given OP_ID. Not very clean, since for an invalid OP_ID we
return empty list, but since it will only be used in tests it should
be good enough.
The generated code looks as follows:
allOpFields :: String -> [String]
allOpFields "OP_TEST_DELAY"
= ["duration", "on_master", "on_nodes", "repeat"]
allOpFields "OP_INSTANCE_REPLACE_DISKS"
= ["disks", "early_release", "iallocator", "ignore_ipolicy",
"instance_name", "mode", "remote_node"]
…
allOpFields _ = []
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 26 Nov 2012 13:16:13 +0000 (14:16 +0100)]
Rework custom fields handling
This patch changes a bit the handling of custom fields. Since in
general we use custom fields to aggregate multiple entries in the JSON
object into a safer data-type, we should also have a way to declare
which extra entries this field covers (so that in the future we can
say what are all the JSON keys for an object).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 20:26:46 +0000 (21:26 +0100)]
Two small haskell-related Makefile improvements
Just some cleanup:
- don't run hlint over hpc-htools, since it's just a symlink to
htools.hs (so we'd get duplicate warnings)
- build rpc-test in make really-all, via HS_ALL_PROGS variable
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 20:20:00 +0000 (21:20 +0100)]
Fix long lines in Ganeti.HTools.Types and add Make check
Since hlint doesn't check style issues but rather code only issues,
let's add a simple Makefile check for too-long lines, and fix the
single bad case we have currently.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 20:13:14 +0000 (21:13 +0100)]
Small Makefile.am cleanups
Indentation within rules is fixed to always be 2 spaces (as opposed to
mixed tabs/2 spaces/4 spaces). Additionally, the check-dirs rule,
which was more complex than necessary, has been simplified: moved
auto4mte.cache to DIRCHECK_EXCLUDE, and simplify the find call and the
exit condition (no need for if test …; then exit 0; else exit 1; fi).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 3 Dec 2012 12:28:12 +0000 (13:28 +0100)]
Fix generation of clusters via 'genEmptyCluster'
The current uniqueness algorithm (generate random node names, suffix
them with node index) is actually wrong: a node named "21" at index 5
will end up with the same name as a node named "2" at position 15.
To fix this, we also add a character from a different "set" ("-"), so
that such mixups can't happen again, and also add an explicit check
for it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:40:58 +0000 (04:40 +0100)]
LUClusterSetParams: Use node allocation lock
All resources are acquired and opportunistic instance creations would
fail. Also add a TODO.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 22 Nov 2012 17:56:26 +0000 (18:56 +0100)]
LUInstanceCreate: Acquire node allocation lock
Opportunistic locks are not yet used. This patch changes
LUInstanceCreate to acquire the node allocation lock to avoid conflicts
with other opcodes acquiring many node locks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 01:46:16 +0000 (02:46 +0100)]
Acquire node allocation lock for failover/migration
See code for an explanatory comment. The lock is released as soon as
possible.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Dimitris Aragiorgis [Fri, 30 Nov 2012 15:45:05 +0000 (17:45 +0200)]
Use GetMultiInstanceInfo in LUNetwork* opcodes
LUNetworkConnect/Disconnect looks up a nodegroup's instances for
conflicting IPs. To do so, use GetNodeGroupInstances() and
GetMulitInstanceInfo().
Additionally, check if the correct locks were acquired.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Mon, 3 Dec 2012 16:08:42 +0000 (17:08 +0100)]
utils.text: Function to verify MAC address prefix
The network management code needs to verify a MAC address prefix.
Instead of (ab)using NormalizeAndValidateMac, clean code should be used.
Unit tests for NormalizeAndValidateMac are updated and new ones for
NormalizeAndValidateThreeOctetMacPrefix are added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michele Tartara [Mon, 3 Dec 2012 17:03:11 +0000 (18:03 +0100)]
Improve the Haskell api doc directory creation system
Instead of having two different lists of directories, now there is only one.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Wed, 28 Nov 2012 11:17:27 +0000 (12:17 +0100)]
Factorize code for checking node daemon certificate
This code is going to be used by a new utility for setting up the node
daemon. Unit tests are updated/added.
Additionally, the certificate and key stored in “server.pem” are
verified, too.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 22 Nov 2012 17:37:15 +0000 (18:37 +0100)]
Support opportunistic locks in mcpu/LUs
Similar to “share_locks”, a new dictionary containing booleans for each
locking level is added to “cmdlib.LogicalUnit”. Logical units wanting to
make use of opportunistic locks will be able to configure this
dictionary accordingly.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 22 Nov 2012 17:44:33 +0000 (18:44 +0100)]
Add opportunistic locking to GanetiLockManager
Just forwarding the parameter, nothing more.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 19 Nov 2012 20:59:47 +0000 (21:59 +0100)]
locking: Implement opportunistic locking in LockSet
This patch adds a new parameter to “LockSet.acquire” named
“opportunistic”. When enabled the lockset will try to acquire as many
locks as possible, but it won't wait for them (with the exception of the
lockset-internal lock in case the whole set is acquired). This is
implemented by using a timeout of 0 seconds when acquiring individual
locks. Commit 03c5291 made such acquisitions significantly cheaper.
The most complicated code included in this patch is probably the helper
function used to determine which mode to use and which timeout functions
are needed.
Full unit tests are included for the new and changed code
(“locking.py”'s overall coverage is at 97%).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 30 Nov 2012 10:41:48 +0000 (11:41 +0100)]
Add ssconf function to read all files
Configuring a node daemon on a newly added node will need all ssconf
values.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 30 Nov 2012 10:36:40 +0000 (11:36 +0100)]
bootstrap.RunNodeSetupCmd: Add IPv6 support
Commit 224ff0f modified the node SSH setup to use the system's SSH
client. Before that Paramiko was used. It's not entirely clear whehter
the latter ever supported IPv6 properly, but with this patch
“bootstrap.RunNodeSetupCmd” is changed to use it if configured. The code
for doing so is taken from “bootstrap.SetupNodeDaemon”, which is going
to use “bootstrap.RunNodeSetupCmd”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 30 Nov 2012 09:30:25 +0000 (10:30 +0100)]
Factorize running node setup command
Part of the code used for running “prepare-node-join” can be re-used for
running a tool to configure the node daemon.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 29 Nov 2012 07:27:01 +0000 (08:27 +0100)]
ssconf: Add dry-run support for writing files
A new utility for configuring the node daemon will support a dry-run
mode. This patch adds the necessary functionality to
“ssconf.SimpleStore” and provides comprehensive tests for
“SimpleStore.WriteFiles”. To enable the latter, a testing-only parameter
specifying the lockfile was added to the class.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Wed, 28 Nov 2012 12:11:08 +0000 (13:11 +0100)]
ssconf: Add function to verify keys
The new utility for configuring the node daemon will have to check
whether it received valid ssconf names.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Fri, 30 Nov 2012 08:12:12 +0000 (09:12 +0100)]
Design for opportunistic locking
Add the design document for opportunistic locking.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Sat, 1 Dec 2012 07:43:32 +0000 (08:43 +0100)]
Fix variable names in Loader.assignIndices
This function can be applied to any type of class "Element" but its
internal variables used to refer to a list of tuples containing these
elements and their names as "nodes". Moreover it built a list of name to
indices tuples referring to it as "na" (which is not very clear).
Thus nodes becomes name_element, and na name_idx.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Dimitris Aragiorgis [Fri, 30 Nov 2012 15:45:08 +0000 (17:45 +0200)]
LUNetworkAdd: Log warning when needed
In case conflicts are checked, log warnings if nodes' IPs cannot
be reserved.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Dimitris Aragiorgis [Fri, 30 Nov 2012 15:45:07 +0000 (17:45 +0200)]
Use constants.IP4_VERSION in LUNetworkAdd
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Dimitris Aragiorgis [Fri, 30 Nov 2012 15:45:04 +0000 (17:45 +0200)]
Fix locking in networks
Ensure that locks are held only if needed.
Add conflicts_check in OpNetworkAdd. This is needed if we want to
check whether nodes/master IPs are included in network.
Depending on conflicts_check value, we have to hold node/instance locks
during OpNetworkAdd/OpNetworkConnect or not.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michele Tartara [Mon, 26 Nov 2012 08:37:39 +0000 (09:37 +0100)]
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checking
that Text.JSON objects are created correctly from the DRBD parser data
structures.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Mon, 3 Dec 2012 08:45:30 +0000 (09:45 +0100)]
Add JSON serialization capabilities for DRBD status
The serialized JSON is not a 1:1 dump of the data structures
populated by the parser. This is done intentionally, with the
aim of producing a more stable and more meaningful output to
be used by the (future) monitoring agent and stand-alone data
collectors.
Also:
* shorten the names of some data types.
* change data type for the fields of the Time data structure
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Mon, 3 Dec 2012 08:31:04 +0000 (09:31 +0100)]
Add two support functions for building JSON objects
The new support functions in module Ganeti.JSON will be used by the next
commits to build JSON objects with optional (Maybe) fields, excluding
them when they contain Nothing.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Mon, 3 Dec 2012 08:54:59 +0000 (09:54 +0100)]
Reduce hostname length in tests and opcode py_compat count
This improves Issue 325 - new runtime and memory consumption is about
1/10 compared to before.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Fri, 30 Nov 2012 11:19:46 +0000 (12:19 +0100)]
Add generator for an instance on a node list
Given a node list in input, we get an instance that had nodes in it.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Fri, 30 Nov 2012 08:17:19 +0000 (09:17 +0100)]
Add generator for list of arbitrary instances
This takes an instance generator and produces a possibly empty list of
instances.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Thu, 29 Nov 2012 17:23:28 +0000 (18:23 +0100)]
Add generator for list of arbitrary nodes
This generates a minimum of one node, because legal clusters never have
zero nodes.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Sat, 1 Dec 2012 10:39:26 +0000 (11:39 +0100)]
Fix options in hbal manpage
hbal doesn't have any "-o" option, but on the other hand supports -S
(described below, but not present in the short summary)
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Fri, 30 Nov 2012 08:15:22 +0000 (09:15 +0100)]
Create a genInstanceMaybeBiggerThan function
This is used only once when testing Cluster.hs, but having it abstracted
clarifies there what that call is about, makes that test shorter, and
allows us to better do refactoring of the main genInstanceSmallerThan
generator.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 19 Nov 2012 09:06:06 +0000 (10:06 +0100)]
Change -G option description
The new one is more consistent with the rest of the nodegroup
terminology and more generic for use in different htools.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Fri, 30 Nov 2012 00:03:57 +0000 (01:03 +0100)]
Fix Haskell profiling targets
It turns out that in newer GHC versions, the suffix shouldn't be
prefixed with a dot; otherwise name mangling is broken.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Thu, 29 Nov 2012 23:53:04 +0000 (00:53 +0100)]
Optimise recursive Query filters
Currently, the And and Or filters use very nice code, e.g. in case of
OrFilter:
any id <$> mapM evaluateFilter flts
However, looking at the memory profiles shows that application of
any/id to monadic values via '<$>' does not work nicely, losing the
'early' success property. This results in too much memory being used
for thunks in monadic sequencing.
Rather than trying to add more strictness (not sure exactly how, TBH),
switching these to explicit recursion solves the problem, since we
take the 'early' exit problem in our hands and we are explicit about
it.
Memory usage in case of big (e.g. 1000 'Or' elements) is reduced
significantly, and thus also the runtime.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Thu, 29 Nov 2012 23:32:52 +0000 (00:32 +0100)]
Make Query operators enforce strictness
Currently, the query operators (binop, etc.) create thunks, instead of
forcing the evaluation of the simple boolean results. This results in
higher than needed memory use.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Thu, 29 Nov 2012 19:23:58 +0000 (20:23 +0100)]
Fix htools/rpc-test after recent changes
Since this program is not built normally, a few recent commits broke
it:
- 707cd3d (“Use exitErr instead of explicit error message and
exitWith”) introduced wrong syntax (missing '$')
- 29a3053 (“Remove unsafePerformIO usage”) changed the signature of
G.Path.clusterConfFile
- 5183e8b (“Convert query path from string errors to GanetiException”)
changed the signature of 'getNode'
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 26 Nov 2012 12:34:31 +0000 (13:34 +0100)]
Add explicit test for wrong OpCode arbitrary instances
I've been bitten a couple of times with arbitrary opcodes working on
UTF-8 locale, but failing on buildbot (ASCII). So let's add an
explicit test that checks always (even with UTF-8) for correct
arbitrary values, showing explicitly which opcodes fail.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 26 Nov 2012 11:50:08 +0000 (12:50 +0100)]
Remove read instances from our Haskell code
It turns out that optimising 'read' derived instances (via -O) for
complex data types (like OpCode, or the various objects) can be slow
to very slow. Disabling such instances results in (time make
$all_our_haskell_binaries) large compile-time savings and also smaller
(unstripped) binaries (by a significant amount):
ghc 6.12: time htools sz hconfd sz
with read: 4m50s 12,244,694 14,927,928
no read: 3m30s 10,234,305 12,536,745
ghc 7.6:
with read: 14m45s 13,694,761 15,741,755
no read: 3m40s 11,631,373 13,245,134
So let's remove these instances, since we never use read in production
for our custom types, and even when debugging in GHCI, we can simply
use the 'show' representation to create the types, without needing to
actually parse from strings.
Note: for the very slow ghc 7.6 compilation time, I filled a ticket
(ghc #7450), since it is surprising(ly slow).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Sun, 25 Nov 2012 22:58:31 +0000 (23:58 +0100)]
Replace hand-coded 'live' field of OpInstanceMigrate
This is the last inline-written field; I've done it separately from
the last patch due to changes being needed in HTools (field changed
type).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sun, 25 Nov 2012 22:28:31 +0000 (23:28 +0100)]
Remove remaining in-line parameters
… except one, and replace them with separately-defined ones in
OpParams.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sun, 25 Nov 2012 21:44:25 +0000 (22:44 +0100)]
Abstract query common opcode definitions
It would be even better if the opcodes would actually have all the
same definitions, until then we have two sets of definitions.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sun, 25 Nov 2012 15:24:59 +0000 (16:24 +0100)]
Make QuickCheck generator names more uniform
Due to lack of attention, we have two styles for generators of
arbitrary values: get* and gen* (e.g. getFQDN and genDiskIndices). In
order to make this more obvious that we deal with a function in the
Gen monad, let's rename all get* functions to gen*.
A few other simplifications were done on existing tests, switching to
helpers that were introduced later.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sun, 25 Nov 2012 15:11:26 +0000 (16:11 +0100)]
Enable equivalence checks of opcode list
This patch enhances the opcode list checks - instead of spawning a
Python interpreter to display the opcode list, we export it statically
in Constants.hs via a slight convert-constants change.
Furthermore, since we now have opcode parity, we enable full opcode
list checks.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sun, 25 Nov 2012 15:10:12 +0000 (16:10 +0100)]
Add two last missing opcodes
OpInstanceQuery was missing accidentally, whereas OpRestrictedCommand
was just recently added without Haskell definitions.
The patch also slightly improves the OpNodeQuery arbitrary generation.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sun, 25 Nov 2012 13:29:11 +0000 (14:29 +0100)]
Add types, parameters and the opcodes for networks
This completes the last missing opcode group. The only difficulty was
with the ip addresses, where we used simple strings to represent them
and (for IPv4) a few helpers to generate arbitrary instances;
otherwise, the patch is trivial.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sun, 25 Nov 2012 13:28:25 +0000 (14:28 +0100)]
Move NICMode from Objects.hs to Types.hs
Also add some unittests for this type.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sat, 24 Nov 2012 21:35:07 +0000 (22:35 +0100)]
Complete the Test opcodes
This adds the OpTestAllocator, OpTestJqueue and OpTestDummy
opcodes. The OpTestAllocator seems to need some cleanup (on the Python
side), for now we implement it as is. As for the other two, while not
used in production, we should have full coverage for them as well.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Sat, 24 Nov 2012 00:55:12 +0000 (01:55 +0100)]
Rename OpTestAllocator.allocator to iallocator
This makes the OpCode more consistent with the other opcodes. The
downside is incompatibility when upgrading from 2.6, but since this is
a test opcode it shouldn't be problematic.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Sat, 24 Nov 2012 00:39:45 +0000 (01:39 +0100)]
Add a helper for the "iallocator" opcode field
This field is used with just changed description in about 10 opcodes,
so unifying it makes things simpler for future potential changes to
the field type.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 23 Nov 2012 16:44:30 +0000 (17:44 +0100)]
Complete the Tag opcodes and fix OpTestDelay missing param
The regexp in OpTagsSearch is loaded as is, without testing for
validity; the rest of the patch is trivial.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Fri, 23 Nov 2012 16:16:14 +0000 (17:16 +0100)]
Add Group, OS and Backup opcodes
This also corrects a docstring in OpBackupExport on the Python side.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Fri, 23 Nov 2012 14:21:01 +0000 (15:21 +0100)]
Complete the instance OpCodes and parameters
Only the original instance opcodes (used in htools) are left
non-converted to only parameter style; they'll be cleaned up later,
once the htools codebase itself migrates to safer types.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Thu, 22 Nov 2012 13:31:47 +0000 (14:31 +0100)]
Add test for mutable default values in opcode parameters
This is not comprehensive, since in Python one can't determine what is
and what is not mutable; but I've added a few base cases (list, dict,
set).
The patch also improves (makes more uniform) the error messages in the
parameter definitions.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 22 Nov 2012 13:25:18 +0000 (14:25 +0100)]
Fix empty list as default value in OpInstanceMultiAlloc
Commit
12e62af5 (“Adding the new opcode for multi-allocation”)
introduced a "bad" default value; while porting this to Haskell, I
realised this is wrong.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 22 Nov 2012 13:20:18 +0000 (14:20 +0100)]
Add types and parameters for OpInstanceCreate
This is a "big" opcode, so sending it separately.
A few types needed changing, and a few parameters were renamed to make
it more clear which are cluster-level and which are instance-level
parameters.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Thu, 22 Nov 2012 10:41:22 +0000 (11:41 +0100)]
Move FileDriver from Objects to Types
Another base type that we need in opcodes as well.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Fri, 30 Nov 2012 09:04:42 +0000 (10:04 +0100)]
Fix breakage introduced in commit a8b3b09
The order of the calls to “ctx.use_privatekey” and “ctx.use_certificate”
was wrong, leading to an exception being thrown.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Wed, 28 Nov 2012 10:09:08 +0000 (11:09 +0100)]
Factorize SSL context setup for certificate check
This code will also be used by the node daemon setup utility.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Dimitris Aragiorgis [Thu, 29 Nov 2012 11:49:38 +0000 (13:49 +0200)]
Introduce ht.TMaybeValueNone and ht.TValueNone
TValueNone checks if a value is "none" and TMaybeValueNone is a wrapper
of TOr(TValueNone, x). This is used by OpNetworkSetParam in order to
reset a network value (e.g. mac_prefix, gateway, etc.)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Thu, 29 Nov 2012 11:33:46 +0000 (12:33 +0100)]
opcodes: Replace manual loop with map
Also remove a superfluous empty line in test file.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 29 Nov 2012 11:23:50 +0000 (12:23 +0100)]
Fix type descriptions in RAPI documentation
This patch adds descriptors to the “_CheckCIDR*” functions in opcodes
and improves the descriptions generated by “ht.TInstanceOf”, thereby
indirectly fixing bad type descriptions in the RAPI documentation.
Before this patch:
- (String and (<function _CheckCIDRAddrNotation at 0x2f…>))
- (Instance of (<class 'ganeti.opcodes.OpInstanceCreate'>))
After this patch:
- (String and (IPv4 address))
- (Instance of (ganeti.opcodes.OpInstanceCreate))
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Helga Velroyen [Thu, 29 Nov 2012 10:06:42 +0000 (11:06 +0100)]
Fixing crash when removing disks
This bug was introduced by my commit "Check ispecs against ipolicy on
instance modify", because I did not filter for the 'add' action. Thus it
would try to read the size parameter also when removing disks.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Thu, 29 Nov 2012 09:19:17 +0000 (10:19 +0100)]
locking: Change locking order, move NAL after instances
Some opcodes, for example LUInstanceFailover and LUInstanceMigrate,
can't know whether they need to acquire all nodes until they have a lock
on the instance. In turn they would have to acquire the node allocation
lock “just in case” and could only release it once the instance had been
acquired.
Since there is no good reason for the node allocation lock to be before
instances in the locking order, this patch changes the order so that
instances are locked first.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michele Tartara [Thu, 29 Nov 2012 09:36:40 +0000 (10:36 +0100)]
Fix Attoparsec test coverage check compatibility
The hpc tool, on squeeze, does not handle utf8 encoded files correctly.
The Attoparsec test is just needed to verify whether the parser handles
uft8 correctly, so the solution is to exclude it from coverage checks.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 29 Nov 2012 07:40:04 +0000 (08:40 +0100)]
Add constant for node certificate mode
A new utility for configuring the node daemon will have to write the
node certificate as well. To not split information about the certificate
file even more, the constant is added to “pathutils”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Wed, 28 Nov 2012 09:03:40 +0000 (10:03 +0100)]
Move cluster verification out of prepare-node-join
A new tool for configuring the node daemon will also have to verify the
cluster name, so it's better to have this function in a central place.
In the process of moving it to ssconf it is also changed to use
“SimpleStore” instead of reading the ssconf directly. Tests are updated.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Wed, 28 Nov 2012 08:23:28 +0000 (09:23 +0100)]
ssconf: Verify file size when reading, add some tests
Until now ssconf would limit the amount read from files to 128 KiB and
silently ignored files larger than that. With this patch a check is
added by using fstat(2) on the file descriptor while it's being read.
There were no tests file ssconf at all, so some are added.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Wed, 28 Nov 2012 06:40:19 +0000 (07:40 +0100)]
Fix pylint error introduced in commit 9675661
Missing spaces around operator.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 11:44:03 +0000 (12:44 +0100)]
ssconf: Move class-level constants to module
There is no good reason why these two constants should be inside the
class. This patch moves them to the module so they can be used without
“self”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 11:45:46 +0000 (12:45 +0100)]
Remove ssconf.SimpleConfigReader
Commit b0dcdc1 removed the last user of this class.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 11:27:38 +0000 (12:27 +0100)]
ssconf: Small error message fixes
- Include filename in error message
- s/ip/IP/
- Stop using superfluous variable
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Helga Velroyen [Tue, 27 Nov 2012 17:33:40 +0000 (18:33 +0100)]
Fix network query for extra stats fields
Extend _GetNetworkStatsField() to accept 4 arguments.
This fixes Issue 323.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Helga Velroyen <helgav@google.com>
Dimitris Aragiorgis [Tue, 27 Nov 2012 12:54:42 +0000 (14:54 +0200)]
Export UUID and serial no in network queries
Add uuid and serial_no to _NETWORK_SIMPLE_FIELDS in order to export
them in network queries. Modify gnt-network to list them too.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 11:40:36 +0000 (12:40 +0100)]
ssconf: Remove unused regular expression
RE_VALID_SSCONF_NAME is not used anywhere.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 09:30:04 +0000 (10:30 +0100)]
Fix build after commit 9b7e05a
Commit 9b7e05a didn't quite work as expected: glm.is_owned expects a
locking level, not a lock name.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 10:43:07 +0000 (11:43 +0100)]
Factorize code to load and verify JSON
A new tool to configure the node daemon will also have to load and
verify JSON data.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 09:27:00 +0000 (10:27 +0100)]
Factorize logging setup in tools
Most tools had their own “SetupLogging” function, but they were all
essentially the same. This patch adds a generic version to “utils.log”
and provides unit tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michele Tartara [Fri, 23 Nov 2012 13:26:46 +0000 (14:26 +0100)]
Small improvements to the DRBD parser and tests
* Now the parser completely consumes the input, up to the end of the text.
* Name of the test suite module changed to adhere to naming conventions.
* Some reformatting of the source code.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 23 Nov 2012 03:42:09 +0000 (04:42 +0100)]
LUNodeQueryStorage: Use node allocation lock
Block instance allocations when all node locks will be acquired.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 07:01:37 +0000 (08:01 +0100)]
cmdlib: Don't pass processor to _ComputeNics
The execution context ID can be passed right away.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 06:57:28 +0000 (07:57 +0100)]
cmdlib.LUInstanceRename: Another assertion for BGL
Check whether the BGL is actually held instead of just testing
“REQ_BGL”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 06:55:56 +0000 (07:55 +0100)]
cmdlib: Stop using proc.Log*
The Log* functions are re-exported through the LU. Inline use of string
formatting (“%”) was converted to additional arguments to Log* (unlike
“feedback_fn”, these functions support it). Some punctuation has been
removed from messages, as well as some small re-wordings.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 07:49:07 +0000 (08:49 +0100)]
cmdlib: Use CommaJoin where appropriate
Don't use inline versions of CommaJoin.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 27 Nov 2012 07:52:52 +0000 (08:52 +0100)]
LUNetwork*: Build dictionaries right away
… instead of constructing an object, only to fill it separately.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Michael Hanselmann [Mon, 26 Nov 2012 05:34:52 +0000 (06:34 +0100)]
locking: Method to check if LockSet is fully acquired
A new method is added to check whether the LockSet-internal lock is
held. This is the case after LockSet.acquire was called with
locking.ALL_SET.
Unit tests are updated, including one where the list of names must be
empty.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 26 Nov 2012 06:02:39 +0000 (07:02 +0100)]
mcpu: Start locking at correct level
Commit 8716b1d added a new lock level, LEVEL_NODE_ALLOC. It is ahead of
LEVEL_INSTANCE. The latter was hardcoded in mcpu to be locked right
after the BGL, effectively ignoring LEVEL_NODE_ALLOC.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michele Tartara [Fri, 23 Nov 2012 16:36:37 +0000 (16:36 +0000)]
Fix wrong edge case in a QuickChek Drbd property
The bug was in the test itself, not in the tested code.
Also, fixed a line longer than 80 characters in the same file.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 22 Nov 2012 21:41:44 +0000 (22:41 +0100)]
Add new lock level for node allocations
The new lock is similar to the BGL in the sense that it has its own
level and there is only one. It is called “node allocation lock”.
Logical units will use it to synchronize with instance creations, which
in turn will start using opportunistic locks on nodes.
Additionally, an assertion in GanetiLockManager gained a message.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michele Tartara [Wed, 21 Nov 2012 11:53:00 +0000 (11:53 +0000)]
QuickCheck Tests for the commaIntParser, part of DRBD Parser
These new tests substitute the HTests, as by the suggestion
received after submitting the previous patch.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>