Michele Tartara [Thu, 20 Dec 2012 09:03:15 +0000 (09:03 +0000)]
Add function for generating UUIDs in the Haskell codebase
Its first use will be to generate the salt for ConfD requests of the Haskell
client, as in the Python client.
Unit test added as well.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Tue, 11 Dec 2012 11:11:12 +0000 (12:11 +0100)]
Move Confd getCurrentTime to Utils
The Confd server implementation has a getCurrentTime function.
This commit moves it to the Utils.hs file of Confd, so that it will be
available to be used by the future Confd client as well.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 20 Dec 2012 15:43:38 +0000 (16:43 +0100)]
Fixup too-long-line in Objects.hs
Sorry, didn't see this before pushing the previous commit (
277a2ec9).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Constantinos Venetsanopoulos [Fri, 7 Sep 2012 10:15:40 +0000 (13:15 +0300)]
ExtStorage support in htools
This just adds new constructors for the disk template and related
types, enabling the Ext storage.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 20 Dec 2012 12:30:44 +0000 (13:30 +0100)]
confd: strip newlines before logging the request
The Python confd client sends newlines in messages (due to use of
DumpSignedJson/DumpJson), so that results in ugly debug output.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 20 Dec 2012 11:36:56 +0000 (12:36 +0100)]
Move rstripSpace from Ssconf to Utils
Also renames it in the process and adds a few tests for it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Thu, 20 Dec 2012 14:45:57 +0000 (15:45 +0100)]
objects.NIC: Look up mode only once, capitalize acronym
Look up “NIC_MODE” only once, capitalize “NIC” in error messages.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 20 Dec 2012 14:40:57 +0000 (15:40 +0100)]
gnt-cluster(8): Quoting, small fixes
Quote fixed strings where appropriate and replace “nic” (an acronym)
with “network interface”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 20 Dec 2012 14:40:47 +0000 (15:40 +0100)]
Add utility to format dictionary as key=value strings
This will be used in QA to format network interface parameters.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Constantinos Venetsanopoulos [Fri, 7 Sep 2012 10:02:27 +0000 (13:02 +0300)]
Adjust LUInstanceSetParams wrt the `ext' template
Initial support for the `ext' template in gnt-instance modify.
Take care of all needed checks in CheckArguments and CheckPrereqs
wrt ExtStorage Providers and ext-params.
ExtStorage disk addition and specification of arbitrary ext-params
during addition is supported.
Currently, one cannot modify an existing disk's ExtStorage Provider,
neither can change an existing disk's ext-params. Only an existing
disk's `mode' can be modified.
Removal is transparent of the disk template and is also supported.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fix bug in disk changes, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>
Constantinos Venetsanopoulos [Wed, 28 Mar 2012 09:29:22 +0000 (12:29 +0300)]
Multiple ExtStorage Providers and ext-params
Add support for passing parameters to the ext template (ext-params).
Take advantage of disk-params, that don't seem to make much sense in
this template (ExtStorage Providers are not predefined and we don't
know their needs) and use them to pass the ext-params dynamically to
the template.
ext-params are correlated with gnt-os-interface's os-params.
All ext-params are exported to the ExtStorage Provider through it's
environment, with variables prefixed with 'EXTP_' (similarly to the
OS interface's 'OSP_' params).
ext-params are passed via the --disk option. If the disk template
is of type `ext', then any additional options passed to --disk and
are not in IDISK_PARAMS are considered ext-params e.g.:
gnt-instance add -t ext --disk=0:size=2G,param1=value1,param2=value2
Finally, we introduce a new IDISK_PARAM called IDISK_PROVIDER, that is
mandatory for template `ext' and is used to select the desired
ExtStorage Provider. This parameter is not a valid --disk option for
any other template type.
The IDISK_PROVIDER parameter becomes the first element of the
disk's unique_id tuple e.g.:
unique_id = ('sample_provider1', 'UUID.ext.diskX')
Example selecting different ExtStorage Providers for each disk and
passing different ext-params to them:
-t ext --disk=0:size=2G,provider=sample_provider1,param1=value1
--disk=1:size=3G,provider=sample_provider2,param2=value2
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small simplification in bdev code, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>
Constantinos Venetsanopoulos [Mon, 12 Mar 2012 15:49:18 +0000 (17:49 +0200)]
Implement the External Storage Interface
With this commit we introduce the External Storage Interface
to Ganeti, abbreviated: ExtStorage Interface.
The ExtStorage Interface provides Ganeti with the ability to interact
with externally connected shared storage pools, visible by all
VM-capable nodes. This means that Ganeti is able to handle VM disks
that reside inside a NAS/SAN or any distributed block storage provider.
The ExtStorage Interface provides a clear API, heavily inspired by the
gnt-os-interface API, that can be used by storage vendors or sysadmins
to write simple ExtStorage Providers (correlated to gnt-os-interface's
OS Definitions). Those Providers will glue externally attached shared
storage with Ganeti, without the need of preprovisioned block devices
on Ganeti VM-capable nodes as confined be the current `blockdev' disk
template.
To do so, we implement a new disk template called `ext' (of type
DTS_EXT_MIRROR) that passes control to externally provided scripts
(the ExtStorage Provider) for the template's basic functions:
create / attach / detach / remove / grow
The scripts reside under ES_SEARCH_PATH (correlated to OS_SEARCH_PATH)
and only one ExtStorage Provider is supported called `ext'.
The disk's logical id is the tuple ('ext', UUID.ext.diskX), where UUID
is generated as in disk template `plain' and X is the disk's index.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small simplification in bdev code, pylint fixes]
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 20 Dec 2012 13:22:27 +0000 (14:22 +0100)]
Stop all daemons on cluster initialization
Addresses issue 145. When daemons are still running from a previous
cluster instance, starting the daemons again will fail.
“ConfigWriter.IsCluster” only looks at the configuration file, which
could already have been deleted.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Wed, 19 Dec 2012 16:21:57 +0000 (17:21 +0100)]
Make node groups' networks field a dict of nicparams
The networks field of the node groups class is a dictionary
of nic params and not a list of networks. This should fix
the broken QA.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Constantinos Venetsanopoulos [Mon, 2 Apr 2012 17:05:51 +0000 (20:05 +0300)]
Update the shared storage design document
Update the shared storage design document to reflect the current
changes, after the implementation of the ExtStorage interface.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
[iustin@google.com: small style fixes]
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 20 Dec 2012 09:53:36 +0000 (10:53 +0100)]
netutils: Normalize all FQDNs
Addresses issue 147. Only the hostnames resolved using
“netutils.Hostname($name)” were normalized, but those returned by the
class method “GetSysName” weren't.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 19 Dec 2012 22:29:30 +0000 (23:29 +0100)]
Display command description in htools usage message
By abstracting the command/personality formatting, we can easily
switch htools to the same output as the rest of the commands.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Wed, 19 Dec 2012 17:06:14 +0000 (18:06 +0100)]
More style updates for network-related code
Error message formatting, else/if merging to elif, docstrings.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 14:00:12 +0000 (15:00 +0100)]
NEWS: Opportunistic locking, clarifications
Some rewording, some clarifications, and a new item on opportunistic
locking.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 14:00:01 +0000 (15:00 +0100)]
cmdlib: Additional opportunistic locking check
If opportunistic locking is requested in the opcode, but no iallocator
is used (not specified or no default), an error will be raised.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Wed, 19 Dec 2012 17:34:38 +0000 (18:34 +0100)]
Detect wrong Python module structure
If, for one reason or another, a module named "ganeti.ganeti" can be
succesfully imported (e.g. through a symlink in the wrong place), error
which are diffcult to debug can occur. This piece of code tests for the
unlikely case of that happening.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 19 Nov 2012 14:49:48 +0000 (15:49 +0100)]
Use hroller to print nodes to reboot by groups
This is highly incomplete, as per the manpage BUGS section.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Thu, 15 Nov 2012 10:21:36 +0000 (11:21 +0100)]
Add hroller htools personality
This is a new personality that for the moment doesn't do anything.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 19 Dec 2012 12:33:02 +0000 (13:33 +0100)]
Fix QA for --offline semantics modification
The previous patch series made it simpler to deal with ERROR_up offline
instances, but didn't fix qa, that had the old expectations. This fixes
that, and adds more testing about the expected behavior.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 19 Dec 2012 13:43:37 +0000 (14:43 +0100)]
One more offline+shutdown fix
Currently gnt-instance shutdown --force of an offline instance
resets the offline status. Which of course is bad and completely
unintended. Fix. (caught by the new qa tests)
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 19 Dec 2012 14:59:19 +0000 (15:59 +0100)]
Fix network LUs use of error code in OpPrereqError
Currently, most network LUs simply use EINVAL instead of correct error
codes; this patch tries to adjust the obvious cases.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 13:11:05 +0000 (14:11 +0100)]
_NetworkQuery: Simplify statistics computation
The code can loop directly over “self.wanted”, saving a lookup. An
explicit loop is also replaced with a generator.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 17:44:23 +0000 (18:44 +0100)]
cmdlib: Factorize network conflict detection
LUNetworkConnect and LUNetworkDisconnect had very similar code to detect
conflicts between instance's network interfaces and networks. This code
factorizes the common part and does some cleanup:
- Remove single-letter variable for lambda (“l”)
- Don't repeat instance name in warning output
- Fixed error messages to match style guide
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 17:21:33 +0000 (18:21 +0100)]
gnt-network: Fix import for textwrap
The style guide says to use “import foo”-style imports, not “from foo
import bar” unless it's a Ganeti module. There are some places with
exceptions, but this one certainly isn't warranted. Also fix the import
order.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 17:19:59 +0000 (18:19 +0100)]
gnt-network add: "--network" is required
Also do some minor code re-formatting.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 16:48:30 +0000 (17:48 +0100)]
cmdlib: Replace period with colon in error message
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 16:38:54 +0000 (17:38 +0100)]
query: Factorize code for getting statistics value
This was not only copied for the networking fields in commit
306bed0e,
but commit
cfcea7ef fixed wrongly ordered parameters and didn't fix the
original. Either way, this patch merges the two cases again. The newly
added function is already tested through the tests for
_GetLiveNodeField.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 16:42:16 +0000 (17:42 +0100)]
gnt-network(8): s/ipv6/IPv6/
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 16:21:16 +0000 (17:21 +0100)]
gnt-network add: Network mode bash completion
This makes entering the command easier.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 16:21:05 +0000 (17:21 +0100)]
OpNetworkConnect: Check for network mode
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 16:20:36 +0000 (17:20 +0100)]
Use generated field list in gnt-network.8
The list of query fields can be automatically generated.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 16:19:48 +0000 (17:19 +0100)]
Improve descriptions of network query fields
They should be in the same style as other descriptions.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 10 Dec 2012 17:45:26 +0000 (18:45 +0100)]
Replace frozenset with compat.UniqueFrozenset
This is not a trivial s/frozenset/compat.UniqueFrozenset/, but rather
only replaces “frozenset” where appropriate. Most of the places are
“static” information that doesn't change after the module has been
loaded.
Some docstrings and code formatting (e.g. empty lines) issues are
addressed as well. Some lines got too long and were wrapped.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Wed, 19 Dec 2012 15:27:41 +0000 (16:27 +0100)]
Renaming fam -> net_family on test network generation
Small renaming to make the test more readable.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Helga Velroyen [Tue, 18 Dec 2012 13:29:08 +0000 (14:29 +0100)]
Serialization compatibility test for node groups
This creates a number of node groups with up to 3 networks,
serializes them and compares them with their python equvalents.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Tue, 18 Dec 2012 13:33:37 +0000 (14:33 +0100)]
Add a network fields to NIC params and node group type
This adds a network field to the NIC params and a list of networks
to the node group type. It adjusts the nodegroup's unit tests accordingly.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Tue, 18 Dec 2012 13:29:08 +0000 (14:29 +0100)]
Move Network definition ahead of depending definitions
Networks will be properties of NicParams and NodeGroups. Therefore
they need to be defined before those.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 19 Dec 2012 11:18:40 +0000 (12:18 +0100)]
Remove some unused Python code
This patch removes code which is no longer used due to refactoring:
- http.InitSsl, last usage removed in commit
33231500 (“Convert RPC
client to PycURL”)
- rapi.baserlib.MakeParamsDict, last usage remove in commit
4e5a68f8
(“RAPI: rlib1 removal”)
- rapi.baserlib.ExtractField, last usage removed in commit
028c6b76
(“RAPI:Fix root list and unittest for it.”)
- qa.qa_utils.AssertNotEqual, last real usage removed in commit
2f4b4f78 (“Simplify QA commands”) (but it was still imported for a
while, until we extended pylint to run over the QA code as well)
- qlang._IsHostname, added in commit
f8638e28 (“Detect globbing
patterns as query arguments”) but never used
- cmdlib._BuildNetworkHookEnvByObject, last usage removed in commit
f22433c0 (“Locking related fixes for networks”)
- NLD constants, which are obsolete—the nbma project is no longer
maintained and it's not working with current ganeti
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 10 Dec 2012 21:51:10 +0000 (22:51 +0100)]
Enable job queries via confd in gnt-node and RAPI
This patch is enabling split queries for jobs for gnt-node and rapi
access (only for job listing, not job waiting).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 19:22:34 +0000 (20:22 +0100)]
Move a node-specific function to Query.Node
This shouldn't live in the main Query module, since it's too node
specific.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 18:59:35 +0000 (19:59 +0100)]
Add support for job queries in hconfd
This adds support for job queries, including (basic) unit-tests.
I've tested this for memory and cpu usage as follows:
- 3600 jobs (live queue):
- via masterd, default: ~1.1s (masterd: ~60MB ram)
- via confd, default: ~1.1s (hconfd: ~25MB ram)
- via masterd, id only: ~1.0s (masterd: ~57MB ram)
- via confd, id only: ~0.2s (hconfd: ~15MB ram)
- all jobs (128K in total, around 570MB size on disk):
- via masterd, default: 1m22s (masterd cpu: 48s), masterd: 1.4G ram
- via confd, default: 1m16s (hconfd cpu: 51s), hconfd: 570MB ram peak
(peak only right before starting luxi send, hconfd decreases in RSS as
results are streamed over luxi, back to ~18MB after the send)
- via masterd, id only: ~49s (masterd cpu: 45s), masterd: 1.3G ram
- via confd, id only: ~39s (hconfd cpu: 35s), hconfd: 110MB ram peak
(right before luxi send, decreasing as results are sent, back to ~14MB
after the send)
Given this, and that in production it's not likely to have hundreds of
thousand of job files, I believe the implementation is safe from this
point of view.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 23:13:33 +0000 (00:13 +0100)]
Change makeSimpleFilter accept NumericValues as well
This is required for QueryJobs implementation; we change
makeSimpleFilter to support both string and integers as names.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 18:38:49 +0000 (19:38 +0100)]
Add a read-only job queue module
This patch adds implementation for a read-only job queue module,
together with "full" test (as full as can be in a lazy language…).
One note about the behaviour of the job queue is the handling of
opcodes that fail validation: the 'input' opcode actually is a
meta-type, which can hold either a real opcode or a plain JSValue, so
that we can still load jobs with invalid opcodes for querying. The
only downside of this is that, as opposed to Python code, we can't
show the correct summary for such an opcode - we try to parse the
OP_ID but not the extended OP_DSC_FIELD-equivalent.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 09:58:33 +0000 (10:58 +0100)]
Implement support for QFF_SPLIT_TIMESTAMP
I am very very unhappy with this patch. We have to do this, in order
to be compatible with the Python code and behaviour, but there's lots
of duplication; both the Python and the Haskell code will need to be
cleaned up and simplified (we don't need QFF_* at all).
For now though, this implements QffTimestamp and associated code.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Bernardo Dal Seno [Wed, 12 Dec 2012 16:28:49 +0000 (17:28 +0100)]
Add exclusive_storage node parameter
Unit tests updated and expanded with an inheritance check.
The flag has no effect yet.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Bernardo Dal Seno [Sun, 9 Dec 2012 16:56:16 +0000 (17:56 +0100)]
Update design doc for partitioned Ganeti
PVs don't need to be marked as unallocatable, as the code will ignore
partially used PVs. exclusive_storage made a node parameter.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Helga Velroyen [Wed, 19 Dec 2012 12:07:40 +0000 (13:07 +0100)]
Remove superfluous typing to Int
Forgot to fix that after the review of patch
"Haskell/python compatibility test for networks".
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Tue, 11 Dec 2012 00:08:36 +0000 (01:08 +0100)]
Extend job ID parsing tests
Since this is a type that behaves non-standard, let's add some more
test for "wrong" input values.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Tue, 18 Dec 2012 15:33:49 +0000 (16:33 +0100)]
Add a few tests for OpParams types
As OpParams definitions are used at Template Haskell type, they don't
get any coverage (although the values defined by them are actually
used in unittests). However, we can at least test some of the
functions defined in the module.
This patch tests the failure modes of a few of the custom types in
this module.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 18 Dec 2012 14:49:04 +0000 (15:49 +0100)]
Remove two hlint overrides
"Use comparing" was not needed, and "Use on" can be solved by actually
using on in the single case we have (the patch also reorders imports
in that file).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 18 Dec 2012 13:42:42 +0000 (14:42 +0100)]
Add more mon-collector shell tests
These test the newly-changed behaviour, and fix an inconsistency in
the hs-check target (versus hs-coverage).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Tue, 18 Dec 2012 13:41:28 +0000 (14:41 +0100)]
Improve mon-collector drbd CLI handling
Currently, this doesn't abort if multiple arguments are passed, and
the error message for errors during file read has a duplicate "Error".
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 17 Dec 2012 16:31:20 +0000 (17:31 +0100)]
Enable bash completion for mon-collector
This just ties in all the pieces so far and enables bash completion
for it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 17 Dec 2012 16:25:18 +0000 (17:25 +0100)]
bash-completion: add support for multi-cmd Haskell binaries
This patch adds support for parsing the command list out of a binary
(very strict format), and then iterating over that and building the
sub-commands options/arguments.
It also does a bit of general cleanup in the script.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 17 Dec 2012 14:34:51 +0000 (15:34 +0100)]
Implement fake completion support for multi-cmd binaries
This will allow build-bash-completion to understand what is the list
of commands that each binary supports, and then provide recursively
completion support for each command.
Note: by "fake", I mean we use hand-built output and hardcoded
"--help-completion" matching, as opposed to automatically doing it via
regular options. It could be possible in the future to use a
two-staged command line parser, but for now not sure if it's worth it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 17 Dec 2012 14:22:24 +0000 (15:22 +0100)]
Simplify htools.hs personality selection
No need to do a boolean-based lookup, we can use the regular
association list lookup.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 17 Dec 2012 15:15:01 +0000 (16:15 +0100)]
Add description to personality definitions
This allows usage information to display nicer help (like in Python).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 17 Dec 2012 13:44:28 +0000 (14:44 +0100)]
Introduce generic multi-command binary handling
Currently, the mon-collector binary is the only multi-command binary
we have in Haskell. Not surprisingly therefore, its command line
handling is not as robust as we need, not being able to support
standard "--help" commands.
This patch introduces generic multi-command handling, similar to the
Python functionality (some parts missing, will be added in future
patches), and switches mon-collector over to it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Fri, 14 Dec 2012 22:51:14 +0000 (23:51 +0100)]
Add a type alias for the personality lists
The signature of the personality definitions is so ugly that, at
least, we should hide it a bit behind a type alias.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Fri, 14 Dec 2012 21:03:49 +0000 (22:03 +0100)]
Add a custom CLI module for DataCollectors
Currently, the mon-collector binary uses the HTools/CLI module, which
is OK but mean it links in lots of htools code. By copying that module
to DataCollectors/CLI and removing the unneeded code, we reduce the
number of modules it depends on fro 20 to 12, meaning both a shorter
compiler time (24s to 9s) and a smaller binary (~9.6MB to ~7MB).
Also fixes a typo in the original HTools/CLI module, thanks Michele!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Fri, 14 Dec 2012 19:39:18 +0000 (20:39 +0100)]
Fix profiling targets
As noted by Guido, there are problems when using the hs-prof and
hs-prof-quick targets in the default configuration (compiling all
programs). The errors manifest in the form of wrong symbols during
compilation.
I knew that the hs-prof targets, which compiled multiple objects with
the '.o' suffix, could be problematic; but the objects that are
actually needed in the Template Haskell phase are very standard and
don't differ between the binaries (Constants, BasicTypes, JSON,
THH). What I didn't realise (although it's obvious) is that also the
hs_prof_quick targets (the final binaries) are also compiled with a
single suffix ('.prof_o'), which means that the object files are
actually compiled for the last binary.
This means that targets later in the HS_ALL_PROGS list would work
correct, but early targets, especially htools/htools, would fail.
So the obvious, and the single simple solution is to make these two
rules only work on a single binary at a time. This should be fine,
since one is looking at a specific problem usually, and it has the
advantage that the hs-prof step is much faster (since it wasn't
buildable in parallel anyway).
Thanks to Guido for finding and making the initial diagnose on this!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Wed, 19 Dec 2012 09:28:01 +0000 (10:28 +0100)]
Allow shutting down offline instance
If an instance is offline we definitely shouldn't start it up.
But shutting it down, should it be up by mistake is not "that" bad.
Still, we only allow it with --force, as it still performs an action on
an instance we shouldn't touch. This should make everybody happy.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 19 Dec 2012 09:05:26 +0000 (10:05 +0100)]
Allow running instances to be put online
If an instance is running (eg. ERROR_up) and at the same time offline,
there's no way to either shut it down or reonline it. This allows
onlining it. Offlining is still disabled for running instances.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Bernardo Dal Seno [Mon, 8 Oct 2012 23:43:17 +0000 (01:43 +0200)]
Fixes and improvements to comments
Some fixes, added more information in a few points, removed a stale (5+
year old) TODO comment.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 18 Dec 2012 13:08:00 +0000 (14:08 +0100)]
Explain how to use GHCI to see spliced TH code
This can be more useful than the "make" version.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Tue, 18 Dec 2012 12:19:18 +0000 (13:19 +0100)]
Fix error during cluster initialization due to hv_kvm
Commit 141d148 was a bit too enthusiastic. The three parameters added to
the list of parameters to be checked default to a value not evaluating
to false, leading to a failure on cluster initialization.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Dimitris Aragiorgis [Tue, 18 Dec 2012 09:23:27 +0000 (11:23 +0200)]
Make LUNetworkAdd pass _VerifyLocks()
LEVEL_NODE_ALLOC should be aquired too if LEVEL_NODE is ALL_SET.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Helga Velroyen [Thu, 13 Dec 2012 13:02:28 +0000 (14:02 +0100)]
Haskell/python compatibility test for networks
This patch contains the HUnit test that checks the compatibility
of Haskell-generated networks with the python code. For that the
generation of test instances of networks was enhanced to meet
the validation steps of the python implementation. Also, so far
networks were generated at two different places in the code; this
is now consolidated.
Signed-off-by: Helga Velroyen <helgav@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 17 Dec 2012 16:16:27 +0000 (17:16 +0100)]
Add test for SPICE parameter list, add missing ones
“_SPICE_ADDITIONAL_PARAMS” is supposed to be the full list of
SPICE-related KVM hypervisor parameters with the exception of
“HV_KVM_SPICE_BIND”. The new test checks if all parameters starting with
“HV_KVM_SPICE_*” are included. Three previously missing parameters are
added to the list.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 17 Dec 2012 16:06:24 +0000 (17:06 +0100)]
hv_kvm: List of SPICE parameters should be module-global
This list is static at runtime and doesn't need to be recreated every
time.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 17 Dec 2012 15:51:27 +0000 (16:51 +0100)]
_DeclareLocksForMigration: Fix non-DRBD locking issue
When non-DRBD disks are used for an instance,
“lu.needed_locks[locking.LEVEL_NODE]” is set to “locking.ALL_SET” (which
is None). The assertion will then fail as None evaluates to False.
Reported by Constantinos Venetsanopoulos.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Mon, 17 Dec 2012 13:41:11 +0000 (14:41 +0100)]
LUOobCommand: Always declare NAL in shared mode
Doing so avoids triggering an assertion in mcpu which cross-checks the
node lock and node allocation lock acquisition mode.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 12 Dec 2012 03:40:46 +0000 (04:40 +0100)]
Fix two logging messages in TLReplaceDisks
Commit
f0f8d060 (“Show old primary/secondary node on disk
replacement”) added two wrong uses of feedback_fn, which results in
log entries such like these:
"log": [
[
7, [
1351258326, 466214], "message", "Replacing disk(s) 0 for instance 'instance1.example.com'"
],
[
8, [
1351258326, 749391], "Current primary node: %s", "node1"
],
[
9, [
1351258326, 774829], "Current seconary node: %s", "node4"
],
[
10, [
1351258329, 620410], "message", "STEP 1/6 Check device existence"
],
(the first and fourth are OK, the 2nd and 3rd are broken).
The API of jqueue._OpExecCallbacks.Feedback should probably be changed
to prevent such an occurrence in the future.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Tue, 11 Dec 2012 00:37:10 +0000 (01:37 +0100)]
Split and extend Confd types tests
Some confd types were not tested for serialisation, so let's move
these (type-related) tests to their own module and extend them.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 01:50:33 +0000 (02:50 +0100)]
Switch Luxi sendMsg from strict to lazy ByteStrings
Commit
e821050d (“Switch the Luxi interface from Strings to
ByteStrings”) was designed to optimise the receive interface, but has
an unfortunate side-effect: when sending non-trivial messages, it
means that both the entire String and the ByteString versions must be
in memory at the same time, leading to much increased memory usage.
By changing the "hPut" from strict to lazy ByteStrings, it means that
both the String and the ByteString values can be evaluated lazily,
with significant effects: for a test query answer, instead of having
a peak from ~600MB to 1.4G during the entire Luxi send operation,
memory consumption actually decreased during the send operation, as
the ByteString chunks are released individually and even the backing
storage of the items that create the JSON string serialisation is
released lazily as well. So instead of slow growth 10→550MB then quick
peak to 1.4GB during Luxi send, we now have an even slower growth to
~580MB and then decrease of memory as the Luxi send progresses.
The only downside is of a small increase in CPU time of a few percents
for the above case; for our use cases, I think this is much desirable.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 00:46:08 +0000 (01:46 +0100)]
Add NFData instances for query results
This allows us to ensure that query results are strict as we build
them, instead of being lazy and only evaluated when the reply is sent
over the Luxi interface.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 00:29:28 +0000 (01:29 +0100)]
Introduce NFData instances for JSON types
The JSValue/JSObject types don't come with a NFData instance, so let's
add one ourselves, so that we can force evaluation of JSValues (either
when building or when reading them).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 03:55:18 +0000 (04:55 +0100)]
Fix parsing of absolute job dependencies
It seems that Python code generates (sometimes) absolute job
dependencies which are strings, instead of integers, so we should be
able to parse these as well. We simply change from explicit int-based
parsing (makeJobId) to the generic one (parseJobId).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 03:27:46 +0000 (04:27 +0100)]
Make two common opcode parameters always serialised
This changes deps and comment fields to always be shown, to match the
Python behaviour for (at least) job listing/ops field.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 12 Dec 2012 03:20:01 +0000 (04:20 +0100)]
Fix disk_template OpInstSetParams parameter
This is not a required field, but rather an optional one; we add a new
parameter and use it instead.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 23:27:21 +0000 (00:27 +0100)]
Support 'null' in Luxi QueryJobs names field
Python code sometimes sends this, so let's support it even though it's
non-standard.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 18:35:09 +0000 (19:35 +0100)]
Abstract code related to job ID creation
Having makeJobIdS as a separate function will allow us to use it
outside of json encoding.
The patch also exports one more function from the Types module.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Tue, 11 Dec 2012 00:11:39 +0000 (01:11 +0100)]
Add new Haskell library dependency for tests
The 'temporary' library add support for create temporary
directories/files beyond what is offered in the base library (which is
somewhat limited, and hard to use in tests).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 10 Dec 2012 18:32:27 +0000 (19:32 +0100)]
Reduce logging level in tests
This allows tests to execute IO actions which have logging as a side
effect, without polluting the stderr too much.
A better solution would be that we have fine-grained control over
loggers, so that tests can run with their own logging, etc. etc..
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 10 Dec 2012 16:25:30 +0000 (17:25 +0100)]
Export jstore constants as well to Constants.hs
This single constant could be as well moved to constants.py. For now I
choose to just export the module as well.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 10 Dec 2012 16:23:54 +0000 (17:23 +0100)]
Generalise the test helper 'genUniquesList'
Instead of always using "arbitrary", make it require a generator (that
can then use more complex rules for building the list elements), and
also convert it to use a temporary set instead of list membership.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Mon, 10 Dec 2012 10:06:21 +0000 (11:06 +0100)]
Export more paths from Path.hs
Also adds a small helper for building the paths.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Iustin Pop [Mon, 10 Dec 2012 08:46:43 +0000 (09:46 +0100)]
Simplify a bit the FilterRegex JSON instance
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 08:32:13 +0000 (09:32 +0100)]
Implement opcode summary support
This implements in the Haskell codebase the opcode summary. As opposed
to Python, we always use custom code for formatting, since we don't
want to use dynamic attribute lookup.
To test this properly, we need to change MetaOpCode to record-syntax,
so that we get accessor functions.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Mon, 10 Dec 2012 08:27:07 +0000 (09:27 +0100)]
Add optional formatting for OP_DSC_FIELD
For some opcodes, the output is not "stable", and depends on the exact
input values; this makes it harder to check consistency against
Haskell code.
To compensate for this, we add a way to override the formatting of the
OP_DSC_FIELD; by default, this is always "%s", but if the
OP_DSC_FORMATTER is defined (must be a callable), it is used to format
the actual value.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Dec 2012 15:13:18 +0000 (16:13 +0100)]
Add a type for the log type in job/opcode logs
Also fixes indentation for OP_STATUS.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 6 Dec 2012 14:23:35 +0000 (15:23 +0100)]
Fix small typo in OpTestDelay/duration definition
The extra space results in inaccessible names; currently GHC doesn't
flag this as an error, but I've filled an upstream bug for that
(http://hackage.haskell.org/trac/ghc/ticket/7484).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 14 Dec 2012 00:31:53 +0000 (01:31 +0100)]
Add constants for two values which we re-evaluate
Two constants which we use as Integer are defined as Int in
Constants.hs (coming from constants.py), so we do the conversion every
time we use it. Let's move them to top-level names, so that the
conversion is only done once.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 13 Dec 2012 21:33:47 +0000 (22:33 +0100)]
Fix an small but quite nasty typo
Introduced in commit
d4752785.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Mon, 13 Dec 2010 11:57:25 +0000 (11:57 +0000)]
update gnt-instance manpage about self-rename
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>