Michael Hanselmann [Mon, 24 Oct 2011 15:45:06 +0000 (17:45 +0200)]
rpc: Convert file-storage-related calls
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 15:43:53 +0000 (17:43 +0200)]
rpc: Convert node-related calls
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 15:43:27 +0000 (17:43 +0200)]
rpc: Convert OS-related calls
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 15:39:57 +0000 (17:39 +0200)]
rpc: Convert X509 calls
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 15:39:26 +0000 (17:39 +0200)]
rpc: Convert import/export functions
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 16:04:12 +0000 (18:04 +0200)]
rpc: Add helpers
These will be used by the generated RPC wrappers.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 15:37:02 +0000 (17:37 +0200)]
rpc: Add support for generated RPC wrappers
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 15:34:42 +0000 (17:34 +0200)]
Add RPC definitions
These will be used for replacing the tons and tons of boilerplate code
in rpc.py.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 14:32:59 +0000 (16:32 +0200)]
rpc_definitions: Add helpers
These helpers will be used to convert incoming parameters to
JSON-compatible types.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 14:31:59 +0000 (16:31 +0200)]
rpc: Disable timeout check
This is in preparation for the definition-based RPC model, where
the timeout will be stored within the definition.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 21 Oct 2011 13:28:05 +0000 (15:28 +0200)]
Add script to generate RPC wrappers
A new script, “autotools/build-rpc”, will be used to generate code for
RPC client wrappers. This is done because “lib/rpc.py” contains lots and
lots of boilerplate code. Forthcoming patches will start converting
RPC wrappers.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 26 Oct 2011 07:09:27 +0000 (09:09 +0200)]
Small resource model design update re console
Starting with Ganeti 2.5 the console uses a query and no longer acquires
locks. See commit
d6f46b6a94.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 26 Oct 2011 06:46:57 +0000 (08:46 +0200)]
cmdlib: Use constant for DRBD meta device size
… instead of a hardcoded value.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 14:38:00 +0000 (16:38 +0200)]
Fix parameters to RPC "blockdev_rename"
Before this patch it would take a variable number of arguments, which
isn't very good for a definition-based approach.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 24 Oct 2011 14:19:34 +0000 (16:19 +0200)]
Fix parameters to RPC "os_validate"
All other RPC wrappers take the node name(s) as the first parameter.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 21 Oct 2011 11:22:00 +0000 (13:22 +0200)]
Un-revert comments in utils.mlock
These updates and clarifications are still useful.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Michael Hanselmann [Fri, 21 Oct 2011 12:23:48 +0000 (14:23 +0200)]
Remove “node_tcp_ping” RPC
It is not used anywhere.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 21 Oct 2011 11:32:17 +0000 (13:32 +0200)]
utils.ShellWriter: Don't indent empty lines
Empty lines shouldn't get indented. Unittest included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 20 Oct 2011 13:59:05 +0000 (15:59 +0200)]
http.client: Remove use of PycURL's “reset” function
We don't re-use cURL objects anymore, so there's no need to reset them.
PycURL 7.19.0 has a reference counting bug leading to a crash after
a certain number of performed requests.
Since a unittest depended on this function for a test, it is replaced
with a per-request completion callback, which could also be used by
users of the HTTP client.
Unittests are updated and a test for the request object's “nice name”,
used by the lock monitor, is included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Michael Hanselmann [Thu, 20 Oct 2011 12:56:00 +0000 (14:56 +0200)]
Add unittest for PycURL 7.19.0's reset issue
While we won't use this function in Ganeti (yet), users should be aware
of the potential crashes caused by this functioni (discussed offline).
If the installed version of PycURL is not affected, this test is a
no-op.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Agata Murawska [Thu, 20 Oct 2011 15:20:11 +0000 (17:20 +0200)]
ovfconverter: use qemu-img path from constants
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Thu, 20 Oct 2011 13:30:09 +0000 (14:30 +0100)]
Merge branch 'devel-2.5'
* devel-2.5:
Allow per-hypervisor optional files
Add hypervisors ancillary files list
xen: abstract a few hardcoded strings as constants
Conflicts:
lib/cmdlib.py
- trivial
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Agata Murawska [Thu, 20 Oct 2011 11:58:52 +0000 (13:58 +0200)]
Fix for buildbot pylint failure
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Tue, 18 Oct 2011 15:46:17 +0000 (16:46 +0100)]
Allow per-hypervisor optional files
Rather than just allowing files for all nodes to be optional, we allow
optional files to be per-category. The way this works is that they must
be included in both lists (the new code checks for this).
The code also removes a duplicate assert (present both in verify and
compute ancillary files) and checks the new functionality in unittests.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Tue, 18 Oct 2011 15:43:44 +0000 (16:43 +0100)]
Add hypervisors ancillary files list
These lists will be used to declare some of the files not necessarily
needed on all nodes. The files selected are files without which the
various hypervisors can still work, but that when they are present
should be synchronized across the cluster (or node group).
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Tue, 18 Oct 2011 15:41:42 +0000 (16:41 +0100)]
xen: abstract a few hardcoded strings as constants
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Thu, 20 Oct 2011 10:20:02 +0000 (12:20 +0200)]
Check for qemu-img in configure and Makefile
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Agata Murawska [Thu, 20 Oct 2011 10:12:26 +0000 (12:12 +0200)]
configure: fix for two version of QuickCheck present
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 17 Oct 2011 10:28:51 +0000 (11:28 +0100)]
Upload spice files in redist-conf
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Wed, 19 Oct 2011 17:29:22 +0000 (18:29 +0100)]
Revert "Rename utils.mlock to utils.cfunc"
The rename is not needed either, since we're not adding more code as of
now.
This reverts commit
57ca011e1cd2681948969724e2646edaac22da28.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Wed, 19 Oct 2011 17:28:42 +0000 (18:28 +0100)]
Revert "utils.cfunc: Cleanup, more flexibility"
We discussed that this is not needed right now, and it breaks existing
functionality and unittests.
This reverts commit
6915fe26da8dce41fc967d761f005390aa956161.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Wed, 19 Oct 2011 17:05:22 +0000 (18:05 +0100)]
Fix unittest failures with python 2.7
In python 2.7 the ovf unittests fail because OVFReader expects
ElementTree.parse() of an erroneous document to throw an
xml.parsers.expat.ExpatError while instead it throws an
ElementTree.ParseError.
The solution is to "except" for both errors, with the catch that
ParseError didn't exist before, so we need to define it locally and
get it from the module if it exists, while leaving it set to "None"
(thus catching no exception) if it does not.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Guido Trotter [Tue, 18 Oct 2011 15:06:14 +0000 (16:06 +0100)]
Merge branch 'devel-2.5'
* devel-2.5:
Revert "rapi.client.ModifyNode should PUT rather than POST"
- also fix the actual call, which was merged as PUT in master by
mistake.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Tue, 18 Oct 2011 15:01:39 +0000 (16:01 +0100)]
Revert "rapi.client.ModifyNode should PUT rather than POST"
This was a mistake on my side because ModifyGroup and ModifyInstance
were PUT, and I was not aware of the discussion and the rationale why
this one had to be POST.
This reverts commit
55ef0cf6497c570aaab9413851435a7ee744222e.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Tue, 18 Oct 2011 14:24:13 +0000 (15:24 +0100)]
Merge branch 'devel-2.5'
* devel-2.5:
Revert "Added SPICE TLS option and related cert paths"
Revert "Implementation of TLS-protected SPICE connections"
Revert "Updated man pages with new SPICE TLS options"
Revert "Add tls_ciphers and use_vdagent options"
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Mon, 17 Oct 2011 15:47:00 +0000 (16:47 +0100)]
Revert "Added SPICE TLS option and related cert paths"
This reverts commit
bfe86c763a9ff1b481d799537ff0f0cf6740dfd1.
This commit will be readded on master.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Mon, 17 Oct 2011 15:46:41 +0000 (16:46 +0100)]
Revert "Implementation of TLS-protected SPICE connections"
This reverts commit
b6267745ede04b3c943bc02e004bdb9347e0f564.
This commit will be readded on master.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Mon, 17 Oct 2011 15:46:20 +0000 (16:46 +0100)]
Revert "Updated man pages with new SPICE TLS options"
This reverts commit
b8a10435271ec4457cdc254e0a6b466b2d3bff24.
This commit will be readded on master.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Mon, 17 Oct 2011 15:45:45 +0000 (16:45 +0100)]
Revert "Add tls_ciphers and use_vdagent options"
This reverts commit
3e40b5879fa0070d6dd0e689dcfc31f20198a5a8.
This commit will be readded on master.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Tue, 18 Oct 2011 13:00:25 +0000 (14:00 +0100)]
Merge branch 'devel-2.5'
* devel-2.5:
rapi.client.ModifyNode should PUT rather than POST
Fix RAPI node modify client and server calls
xen: changes to facilitate "xl" support (xen 4.1)
xen: abstract instance config file naming
Abstract xen's 'xm' command as a constant
Fix RAPI documentation build
rapi: Allow auto-promotion on node role change
rapi: Add resource for modifying node
opcodes: Add comment to *SetParams result description
Conflicts:
lib/rapi/client.py
- both functions stay, remove one empty line
lib/rapi/rlib2.py
- convert new function to 2.6 rapi style
test/ganeti.rapi.client_unittest.py
- both tests stay, trivial
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Tue, 18 Oct 2011 12:32:12 +0000 (13:32 +0100)]
rapi.client.ModifyNode should PUT rather than POST
This was caught (albeit in a sibylline manner) by unittests on master
which are not present in 2.5.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Guido Trotter [Tue, 18 Oct 2011 08:50:03 +0000 (09:50 +0100)]
Fix RAPI node modify client and server calls
rapi.client.ModifyNode accepts a "group" and not a "node" param.
(this bug is invisible but still not nice)
rlib2.R_2_nodes_name_modify submits the opcode with instance_name rather
than node_name as a param. This would break the call.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Mon, 17 Oct 2011 11:05:00 +0000 (12:05 +0100)]
xen: changes to facilitate "xl" support (xen 4.1)
- Copy the xl config file, in case there's any
- Start instances by config file, not name (also xm compatible)
- Start paused domains with -p and not --paused (also xm compatible)
- Add a fixme for migration (changes are not xm compatible)
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 17 Oct 2011 11:04:18 +0000 (12:04 +0100)]
xen: abstract instance config file naming
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 17 Oct 2011 10:35:52 +0000 (11:35 +0100)]
Abstract xen's 'xm' command as a constant
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Oct 2011 17:04:37 +0000 (19:04 +0200)]
utils.cfunc: Cleanup, more flexibility
- Split code using ctypes directly into a helper class
- Don't load “libc.so.6”, but use handle for main program instead (see
comment in code)
- Clarify comment on errno with older ctypes versions
- Rename unittest since it can't be used for other functions (modifies
process environment at runtime)
- Add boolean return value for “Mlockall”
These changes are leftovers from some experiments with ctypes.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Oct 2011 15:24:48 +0000 (17:24 +0200)]
Rename utils.mlock to utils.cfunc
Renaming so that more code using ctypes could be added to the same file.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Fri, 7 Oct 2011 18:30:09 +0000 (20:30 +0200)]
Add design doc for virtual(ised) clusters
I am currently able to run a 2-node virtual cluster on my machine,
with a very ad-hoc setup. But the results show clearly that this is
doable, and that given the right tools, setting up such a cluster will
be quite easy.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 13 Oct 2011 12:11:59 +0000 (14:11 +0200)]
Document some useful Haskell tips
This improves devnotes.rst with some tricks for Haskell development,
and additionally it does two Makefile improvements:
- properly document lib/_vcsversion.py as a requirement for
Constants.hs (but do not require rebuild when updated)
- move HEXTRA at the end of the GHC invocation, so any command line
options will indeed override the built-in ones (especially -osuf)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 13 Oct 2011 10:39:56 +0000 (12:39 +0200)]
Further cleanup in hspace
This moves the checking of results from the allocation functions to a
separate function, so that we have less code duplication. It also does
a bit of simplification in the printing functions.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 10 Oct 2011 10:49:47 +0000 (12:49 +0200)]
A bit of cleanup in hspace
The node offline/mcpu is identical to hbal's setNodesStatus, so let's
move that to CLI.hs and reuse it in hspace (also, rename it and drop
one 's').
Also, the check for the number of nodes is obsolete, as we compute
that from the disk template.
The patch does a bit of other small cleanups.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 13 Oct 2011 09:14:27 +0000 (11:14 +0200)]
Add a type synonym for the allocation function sig
Both iterateAlloc and tieredAlloc share the same signature, but it's
not documented nor exported (needed for refactoring).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 12 Oct 2011 16:05:19 +0000 (18:05 +0200)]
htools: Simplify Luxi query results parsing
The logic is not entirely correct—the new Query interface exports the
field status, and we don't use that yet. But the new code should be
more readable.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Wed, 12 Oct 2011 11:54:57 +0000 (13:54 +0200)]
Adjust htools code to new Luxi argument format
This partially undoes commit 92678b3, more specifically it removes the
Store data type and the associated code, since all Luxi arguments are
now lists.
Furthermore, since the qfilter field on Query is complex (it's
actually a tree structure), and we don't support it, turn it into a
plain () type, which always gets encoded as JSNull ('null'), so that
we can remove the optional field handling from Luxi (all fields are
always required).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Thu, 13 Oct 2011 14:58:19 +0000 (16:58 +0200)]
Fix RAPI documentation build
*mumble*
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Oct 2011 12:19:08 +0000 (14:19 +0200)]
rapi: Allow auto-promotion on node role change
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Oct 2011 12:11:00 +0000 (14:11 +0200)]
rapi: Add resource for modifying node
A separate patch will add “auto-promote” through
“/2/nodes/[node_name]/role”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Oct 2011 12:10:07 +0000 (14:10 +0200)]
opcodes: Add comment to *SetParams result description
Explicitely say that the second element of the tuple is the new value.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Oct 2011 12:36:41 +0000 (14:36 +0200)]
constants: Verify exported names
The “constants” module is a bit special in the sense that we don't want
to export random stuff from it. This unittest checks the naming
convention and removes imported modules from the module's namespace.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 13 Oct 2011 11:01:09 +0000 (13:01 +0200)]
http.client: Remove HTTP client pool code
This patch removes all remains of the HTTP client pool. Newly added unittests
provide 96% coverage on http.client.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 12 Oct 2011 12:18:01 +0000 (14:18 +0200)]
rpc: Remove thread-local storage with HTTP pool
The HTTP pool is no longer used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 12 Oct 2011 11:39:10 +0000 (13:39 +0200)]
Merge branch 'devel-2.5'
* devel-2.5:
rpc: Disable HTTP client pool and reduce memory consumption
Preserve bridge MTU in KVM ifup script
hail: Fix result for node evacuation
Fix assertion error on unclean master shutdown
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 12 Oct 2011 11:25:42 +0000 (13:25 +0200)]
Tiny optimisation related to filter parsing
Currently, we get a luxi Client, then parse the filter, then execute
the query. If parsing the filter fails, we connected to the masterd
needlessly.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 12 Oct 2011 11:03:15 +0000 (13:03 +0200)]
Merge branch 'stable-2.5' into devel-2.5
* stable-2.5:
rpc: Disable HTTP client pool and reduce memory consumption
hail: Fix result for node evacuation
Fix assertion error on unclean master shutdown
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 12 Oct 2011 10:16:32 +0000 (12:16 +0200)]
Standardise LUXI call argument types
Currently, we have 4 types of arguments in LUXI calls:
- most common, a list of values
- a single argument that is sent as a list of one element
- a single argument that is sent by itself
- a dictionary (only Query and QueryFields)
This inconsistency makes it not only harder to auto-generate the
HTools LUXI interface, but also in general to check the arguments and
(if we ever want to do it) auto-generate the Python LUXI client.
Compare this with the node daemon, which uses consistently a list for
its arguments, and even with way more changes over time had no issues
with extending the interface.
In case we want to extend a call, there are two options:
- preferred: add a new call, keep the old one unchanged
- possible: add further parameters to the current argument list
The patch against HTools will follow—sending separately as the Python
changes are very clear by themselves.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 12 Oct 2011 09:07:18 +0000 (11:07 +0200)]
Rename filter and filter_ to qfilter
We currently use 'filter' as the OpCode, QueryRequest and RAPI field
name for representing a query filter. However, since 'filter' is a
built-in function, we actually have to use filter_ throughout the code
in order to not override the built-in function.
This patch simply goes and does a global sed over the code. Due to the
fact that the RAPI interface already exposed this field, we add
compatibility code for now which handles both forms.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Wed, 12 Oct 2011 11:00:19 +0000 (13:00 +0200)]
Merge branch 'devel-2.4' into stable-2.5
* devel-2.4:
rpc: Disable HTTP client pool and reduce memory consumption
Fix assertion error on unclean master shutdown
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 12 Oct 2011 10:37:43 +0000 (12:37 +0200)]
rpc: Disable HTTP client pool and reduce memory consumption
We noticed that “ganeti-masterd” can use large amounts of memory,
especially on large clusters. Measurements showed a single PycURL client
using about 500 kB of heap memory (the actual usage depends on versions,
build options and settings).
The RPC client uses a per-thread HTTP client pool with one client per
node. At this time there are 41 non-main threads (25 for the job queue
and 16 for client requests). This means the HTTP client pools use a lot
of memory (ca. 200 MB for 10 nodes, ca. 1 GB for 50 nodes).
This patch disables the per-thread HTTP client pool. No cleanup of
unused code is done. That will be done in the master branch only.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 12 Oct 2011 08:08:00 +0000 (10:08 +0200)]
Haskell support for generic Query in Luxi
Untill now htools did not have support for generic Query in Luxi. This
patch introduces Query as a supported Luxi operation and replaces
QueryNodes, QueryInstances and QueryGroups with Query.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Tue, 11 Oct 2011 10:52:12 +0000 (12:52 +0200)]
TH simplification for Luxi
This patch simplifies the generation of save constructors for LuxiOp
by always using showJSON over an array of JSValues, instead of having
to pass showJSON in most cases, except the 5-tuple case.
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
[iustin@google.com: fixed a few issues]
Agata Murawska [Tue, 11 Oct 2011 10:51:03 +0000 (12:51 +0200)]
Dots in docstings and hlint error fixes for htools
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Tue, 20 Sep 2011 04:39:58 +0000 (13:39 +0900)]
Add design doc for the resource model changes
This is not complete, but is as close as I can get it for now. I
expect people actually implementing the various changes to extend the
design doc.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Andrea Spadaccini [Tue, 11 Oct 2011 13:39:05 +0000 (14:39 +0100)]
Preserve bridge MTU in KVM ifup script
Closes: #201 - KVM_IFUP does not set bridge-MTU on tap devices
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 10 Oct 2011 10:12:50 +0000 (12:12 +0200)]
Remove the oneline output option in hbal
This was, AFAIK, never used, and complicates the output code enough
that it's better to remove it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Mon, 10 Oct 2011 09:48:45 +0000 (11:48 +0200)]
Rework/split hbal's main function
This is just moving code around. A subsequent patch will do a bit more
cleanup and changing the output.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>
Iustin Pop [Thu, 6 Oct 2011 14:48:40 +0000 (16:48 +0200)]
Skip application of 'id' in TH code
This is just beautification when dumping splices to stdout, as ghc
will optimise the 'id' away anyway.
Original generate code:
opToArgs QueryTags kind name = J.showJSON (id kind, id name)
Afterwards:
opToArgs QueryTags kind name = J.showJSON (kind, name)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Andrea Spadaccini [Fri, 7 Oct 2011 16:03:21 +0000 (17:03 +0100)]
Don't send gratuitous ARP if master IP setup fails
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Andrea Spadaccini [Thu, 6 Oct 2011 19:28:23 +0000 (20:28 +0100)]
Document --ignore-errors and --error-codes
Update the man page of gnt-cluster to contain the documentation of the
--ignore-errors and --error-codes verify options. Also, include the list
of the error codes and their documentation.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Andrea Spadaccini [Thu, 6 Oct 2011 19:25:54 +0000 (20:25 +0100)]
Add error codes documentation
lib/constants.py
* add to each CV_E* tuple the documentation of the error code
* add the DOCUMENTED_CONSTANTS constant for the doc preprocessor
autotools/docpp
* add a new directive class CONSTANTS_<kind>, that gets data from
constants.DOCUMENTED_CONSTANTS
lib/cmdlib.py
* modify the code that unpacked the CV_E* tuples to ignore the
documentation parameter
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Andrea Spadaccini [Thu, 6 Oct 2011 19:19:43 +0000 (20:19 +0100)]
Generalize docpp and sphinx_ext
autotools/docpp
* handle generic custom directives in the form <class>_<kind>
* adapt handling of query fields
build/sphinx_ext.py
* add the BuildValuesDoc function to output definitions using the sphinx
syntax that was already used for query fields
* adapt BuildQueryFields to use BuildValuesDoc
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Fri, 7 Oct 2011 09:58:09 +0000 (11:58 +0200)]
hail: Fix result for node evacuation
According to the iallocator documentation the “node-evacuate” call needs
to return a list of jobs, not a list of lists of jobs.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Tue, 4 Oct 2011 15:29:39 +0000 (17:29 +0200)]
Use TemplateHaskell to create LUXI operations
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Wed, 5 Oct 2011 09:25:13 +0000 (11:25 +0200)]
Documentation update for ovfconverter
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Agata Murawska [Wed, 5 Oct 2011 09:24:56 +0000 (11:24 +0200)]
Fixes for ovfconverter + vmware
Signed-off-by: Agata Murawska <agatamurawska@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Andrea Spadaccini [Tue, 4 Oct 2011 14:55:48 +0000 (15:55 +0100)]
Demote to warnings the errors in --ignore-errors
Treat the gnt-cluster verify errors identified by the error codes in
--ignore-errors as warnings; just print a warning message for the user.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Andrea Spadaccini [Tue, 4 Oct 2011 12:26:43 +0000 (13:26 +0100)]
Add --ignore-errors parameter to cluster verify
lib/cli.py
- add IGNORE_ERROR_OPT;
client/gnt_cluster.py
- pass the ignore_errors parameter to the opcodes
lib/opcode.py
- update OpClusterVerifyConfig, OpClusterVerify and OpClusterVerifyGroup
to accept the ignore_errors parameter
lib/cmdlib.py
- pass the ignore_errors parameter to the opcodes that need it
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Andrea Spadaccini [Tue, 4 Oct 2011 12:26:43 +0000 (13:26 +0100)]
Move cluster verify error codes to constants
- move the cluster verify error codes from cmdlib._VerifyErrors to
constants;
- add to each of them the CV (Cluster Verify) prefix;
- add the CV_ALL_ECODES and CV_ALL_ECODES_STRINGS constants;
- wrap the lines that exceed 80 characters after changing the error
code names to the new ones.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Andrea Spadaccini [Wed, 5 Oct 2011 10:10:35 +0000 (11:10 +0100)]
Restore backend.GetMasterInfo return values order
Change
5a8648eb609f7e3a8d7ad7f82e93cfdd467a8fb5 changed the order of the
return values of backend.GetMasterInfo(). This broke the users of the
master_info RPC.
This change restores the original order, and adds a comment in
bootstrap.py about the new value added to the return values of
master_info.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Andrea Spadaccini [Mon, 3 Oct 2011 14:23:55 +0000 (15:23 +0100)]
Add cluster netmask parameter
Add the master_netmask cluster parameter, that represents the netmask of
the master IP, encoded as a CIDR suffix.
This parameter can be set via the --master-netmask of gnt-cluster init
and gnt-cluster modify. The default behaviour is to be consistent with
the old default (/32 for IPv4 and /128 for IPv6).
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Andrea Spadaccini [Mon, 3 Oct 2011 14:22:10 +0000 (15:22 +0100)]
Add ValidateNetmask and GetClass IPAddress methods
Add the following methods to netutils.IPAddress:
* ValidateNetmask
* GetClassFromIpVersion
* GetClassFromIpFamily
Also, add related tests to the test suite.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Andrea Spadaccini [Tue, 4 Oct 2011 18:34:27 +0000 (19:34 +0100)]
Merge branch 'devel-2.5'
* devel-2.5:
cluster-merge: log an info message at node readd
Bump version to 2.5.0~rc1
Fix issue when verifying cluster files
Revert "utils.log: Write error messages to stderr"
Fix adding nodes after commit
64c7b3831dc
LUClusterVerifyGroup: Spread SSH checks over more nodes
Optimise cli.JobExecutor with many pending jobs
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Andrea Spadaccini [Tue, 4 Oct 2011 18:31:46 +0000 (19:31 +0100)]
Merge branch 'stable-2.5' into devel-2.5
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
Guido Trotter [Tue, 4 Oct 2011 18:02:39 +0000 (14:02 -0400)]
cluster-merge: log an info message at node readd
node readd can take a long time, it's good to have info messages to see
progress.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>
Michael Hanselmann [Tue, 4 Oct 2011 09:29:34 +0000 (11:29 +0200)]
Bump version to 2.5.0~rc1
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Mon, 3 Oct 2011 16:25:24 +0000 (18:25 +0200)]
Fix Makefile rules for QCHelper.hs
Include QCHelper.hs in the distributed files, and also exclude it and
the THH.hs file from coverage reports.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Andrea Spadaccini <spadaccio@google.com>
Michael Hanselmann [Mon, 3 Oct 2011 14:58:22 +0000 (16:58 +0200)]
Fix issue when verifying cluster files
If a cluster has any non-master-candidate nodes, those don't contain all
files (e.g. config.data). With commit
aef59ae764dc (March 31st, 2011)
the logic was changed and subsequently verifying a cluster with non-mc
nodes would complain.
This patch fixes this issue by changing the algorithm. It also adds an
additional check for files which shouldn't exist on a machine. A newly
added unittest is included.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 3 Oct 2011 10:46:27 +0000 (12:46 +0200)]
Revert "utils.log: Write error messages to stderr"
This reverts commit
34aa8b7c4bb6f5e2e788108e024c9cd70bdb3431. Writing
error messages to stderr would also include backtraces, something we
tried to avoid in the past.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 3 Oct 2011 10:04:09 +0000 (12:04 +0200)]
Fix adding nodes after commit
64c7b3831dc
Commit
64c7b3831dc changed the RPC call for verifying SSH connections.
Unfortunately this case in adding nodes was missed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Fri, 30 Sep 2011 07:46:49 +0000 (09:46 +0200)]
Some TH simplifications
Now that the basic code works, let's use some aliases for simpler code
and less ))))))))).
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Wed, 21 Sep 2011 09:43:57 +0000 (18:43 +0900)]
A few minor test improvements
This patch adds a few niceties to the test suite:
- allows matching test groups case insensitive and emit warnings when
we give test group names that don't match anything
- add a new operator that is similar to assertEqual in Python: it
tests for equality and emits the two values in case of error
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>