Michael Hanselmann [Thu, 8 Nov 2012 18:08:07 +0000 (19:08 +0100)]
sphinx_ext: Allow use of “rapi” module in pyeval
This way constants like “rapi.RAPI_ACCESS_WRITE” can be used in
documentation.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 8 Nov 2012 17:51:03 +0000 (18:51 +0100)]
rlib2: Document two previously undocumented functions
Commit
208a6cff just included empty docstrings.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 24 Oct 2012 01:05:05 +0000 (03:05 +0200)]
jqueue/mcpu: Determine priority using callback
Instead of being given the priority for acquiring locks by means of a
parameter, mcpu will now call back. This is in preparation for
implementing a command to change a job's priority on the fly and allows
to change it while locks are being acquired (taking effect on the next
lock acquire).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Thu, 8 Nov 2012 14:08:47 +0000 (15:08 +0100)]
Merge branch 'devel-2.6'
* devel-2.6:
http/__init__.py: Remove extraneous argument
rapi.testutils: Add utility to format HTTP headers
rapi.testutils: Return headers from mock utility
http: Add wrapper for mimetools.Message
Add missing tests for commit f0d2286
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 8 Nov 2012 13:56:35 +0000 (14:56 +0100)]
Merge branch 'stable-2.6' into devel-2.6
* stable-2.6:
http/__init__.py: Remove extraneous argument
rapi.testutils: Add utility to format HTTP headers
rapi.testutils: Return headers from mock utility
http: Add wrapper for mimetools.Message
Add missing tests for commit f0d2286
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 8 Nov 2012 13:45:46 +0000 (14:45 +0100)]
http/__init__.py: Remove extraneous argument
pylint complained, I fixed it, and unfortunately pushed too early.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 7 Nov 2012 17:34:36 +0000 (18:34 +0100)]
rapi.testutils: Add utility to format HTTP headers
Once again this will be used by forthcoming RAPI test.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 7 Nov 2012 16:22:35 +0000 (17:22 +0100)]
rapi.testutils: Return headers from mock utility
A newly added test for RAPI will also verify the returned headers. A
test in ganeti.rapi.client_unittest.py is split into smaller stand-alone
tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 7 Nov 2012 16:19:46 +0000 (17:19 +0100)]
http: Add wrapper for mimetools.Message
A newly added piece of code will also have to parse headers, so having
this wrapper saves us from copying this part of code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 8 Nov 2012 10:16:36 +0000 (11:16 +0100)]
Rename Confd.hs to Confd/Types.hs
This should be the last module rename, promise!
We rename this to conform to the other hierarchies (e.g. Query), and
to not have both Confd.hs and Confd/*.hs.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Dato Simó [Wed, 7 Nov 2012 19:49:13 +0000 (19:49 +0000)]
OpCodes.hs: add OpTagsSet and OpTagsDel
Also, adjust comment to $(genOpCode) block to avoid repetition of
"only".
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Dato Simó [Wed, 7 Nov 2012 19:49:12 +0000 (19:49 +0000)]
htools: move TagObject from Luxi.hs to OpCodes.hs
In addition to ReqQueryTags in Luxi.hs, the TagObject ADT is also
required for the "kind" attribute of OpTagsSet and OpTagsDel, which
are coming to OpCodes.hs next. Hence, we move TagObject there, and
adjust imports accordingly.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 7 Nov 2012 15:40:47 +0000 (16:40 +0100)]
Add missing tests for commit f0d2286
Commit f0d2286 changed the logic of
gnt_instance._ConvertNicDiskModifications to also allow a parameter
named “modify”. Unfortunately the corresponding unittest was not
updated. An “if”/“else” condition is also merged.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 1 Nov 2012 11:47:01 +0000 (12:47 +0100)]
Implement base checkFn/prepFn/execFn model for daemons
This mirrors the code in the Python code base, and is required for
clean error reporting during startup.
This patch implements the basic infrastructure; the confd daemon is
not yet modified to take advantage of this, just the types are
adjusted.
Also, the pipe-based error reporting will go in a future patch, once
we can actually use that for reporting.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 7 Nov 2012 13:30:42 +0000 (14:30 +0100)]
One more ghc 7.6 fix
This is only in master, so needed to be fixed separately.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Fri, 26 Oct 2012 15:38:03 +0000 (17:38 +0200)]
workerpool: Use itertools.count instead of manual counting
Instead of having to explicitely increment the value (“… += 1”), a call
to next() is enough. These numbers should in no case be re-used (they
are used for ordering tasks). Using “itertools.count” is useful here as
it guarantees that a returned number won't be returned another time.
Manual code for this could, over the course of time, gain unintended
bugs.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Iustin Pop [Wed, 7 Nov 2012 12:32:42 +0000 (13:32 +0100)]
Merge branch 'devel-2.6' into submit
* devel-2.6:
Fix compatibility with newer Haskell libraries
Fix gnt-instance console with xl
Conflicts:
Makefile.am (reordering, fixed)
htools/Ganeti/Confd/Server.hs (hlint fixes on master)
htools/Ganeti/Daemon.hs (hlint)
htools/Ganeti/HTools/Backend/Rapi.hs (hlint)
htools/Ganeti/HTools/ExtLoader.hs (hlint)
htools/Ganeti/HTools/QC.hs (file renamed/split in master, fixed)
htools/test.hs (we don't use maxDiscards, ignored)
lib/constants.py (move to pathutils)
lib/hypervisor/hv_xen.py (move to pathutils)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Wed, 7 Nov 2012 12:03:49 +0000 (13:03 +0100)]
Merge branch 'stable-2.6' into devel-2.6
* stable-2.6:
Fix compatibility with newer Haskell libraries
Fix gnt-instance console with xl
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Wed, 7 Nov 2012 11:49:11 +0000 (12:49 +0100)]
Switch devel/upload to a static file
We had twice in the past days questions about devel/upload being
"broken", since bash re-reads shell scripts during their run and this
file can get regenerated due to Makefile changes.
Since we only need this to be dynamically built for 3 variables, let's
make the file static and read those three variables when it is run,
instead of when it is built, which allows us to re-read the "latest
version" of these vars as well.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 6 Nov 2012 21:25:51 +0000 (22:25 +0100)]
Fix compatibility with newer Haskell libraries
This small patch fixes compatibility with a few newer Haskell libraries:
- base 4.6, included with ghc 7.6, removed the deprecated 'catch'
function from Prelude, so our "import Prelude hiding (catch)" is now
an error; we workaround by using fully-qualified
Control.Exception.catch name
- containers 0.5 changed the signature of 'deleteFindMax'; we
workaround by using separate 'findMax' and 'deleteMax'
- QuickCheck 2.5 removed the 'maxDiscards' test parameter, replacing
it with a much better 'maxDiscardsRatio'; however, until we can
depend on that, we workaround by just removing it (we don't control
anymore the maxDiscards, instead leaving it default; for our default
test size, this is no change, as the default value is already 500,
which is our default as well) and not printing it anymore
Tested on Squeeze (+extra libs), Wheezy and experimental, which covers
all supported GHC versions.
Also, merging this in master will be a pain, but unless we want to
stop supporting 2.6…
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Tue, 6 Nov 2012 17:52:59 +0000 (18:52 +0100)]
Use SSH_LOGIN_USER rather than root for xl ssh
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Tue, 6 Nov 2012 16:54:04 +0000 (17:54 +0100)]
Fix gnt-instance console with xl
- Rename xm-console-wrapper to xen-console-wrapper
- Pass the xen command to use as a parameter
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 6 Nov 2012 17:27:39 +0000 (18:27 +0100)]
Add utility to check if file is executable
This replaces direct calls to “os.access” and
“os.path.exists”/“os.path.isfile”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 6 Nov 2012 17:12:11 +0000 (18:12 +0100)]
Fix NameError in constants.py introduced in merge 46c1f82
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Tue, 6 Nov 2012 15:47:53 +0000 (16:47 +0100)]
Add test for Luxi calls consistency hs/py
This tests that the same Luxi calls are defined in Python and
Haskell. It doesn't test yet that their serialisation is correct
though.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Guido Trotter [Tue, 6 Nov 2012 16:22:40 +0000 (17:22 +0100)]
Merge branch 'devel-2.6'
* devel-2.6:
Disable E1101 on ganeti/http/server.py:424
Fix live migration under xl
Don't check for xend port when using xl
Conflicts:
lib/hypervisor/hv_xen.py: trivial
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Tue, 6 Nov 2012 16:09:40 +0000 (17:09 +0100)]
Merge branch 'stable-2.6' into devel-2.6
* stable-2.6:
Disable E1101 on ganeti/http/server.py:424
Fix live migration under xl
Don't check for xend port when using xl
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Tue, 6 Nov 2012 15:35:34 +0000 (16:35 +0100)]
Disable E1101 on ganeti/http/server.py:424
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Mon, 5 Nov 2012 14:18:33 +0000 (15:18 +0100)]
Fix live migration under xl
Until now the only way to make live migration work in conjunction with
"xl" was to add ssh known_hosts keys for every node's secondary ip on
every other node.
With this command we remove the target key verification: this is not
worse than what we were doing before with "xm", and allows the migration
to happen under either toolstack, without extra manual work. Of course
the full security of ssh is not used by live migration, then.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Mon, 5 Nov 2012 13:32:24 +0000 (14:32 +0100)]
Don't check for xend port when using xl
If the toolstack is set to "xl" we shouldn't ping xend for liveness
before attempting a live migration.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Mon, 5 Nov 2012 18:12:51 +0000 (19:12 +0100)]
utils.io: Improve handling of double and single slashes
Up until now “IsBelowDir("/", …)” would never return True. The reason
was that an additional slash was added to the root path resulting in
“//", which is “implementation-defined” in posix and treated specially
by “os.path.normpath”.
This patch fixes the behaviour for this special case and adds tests
(also for IsNormAbsPath). A typo in the docstring is fixed. Calls to
“assert_” and “assertFalse” are changed to pass a message by keyword
argument.
It is a bit of a mess, but I hope the resulting behaviour is correct.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Dato Simó <dato@google.com>
Michael Hanselmann [Fri, 2 Nov 2012 17:07:28 +0000 (18:07 +0100)]
workerpool: Don't mask variable in AddManyTasks
The name “priority” is already used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Tue, 30 Oct 2012 15:43:24 +0000 (16:43 +0100)]
workerpool: Simplify _WaitForTaskUnlocked
The function in is simplified in its structure and duplicated checks
have been merged.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Dato Simó [Mon, 5 Nov 2012 17:54:59 +0000 (17:54 +0000)]
cli.py: use None as name for tag operations on the cluster
This change is mostly cosmetic. Previously, the literal "cluster" was
used for the 'name' field of tag operations on the cluster (as opposed
to a node or an instance). Since this field has a type of TMaybeString
specifically for the case of the cluster, it seems more correct to use
None, rather than an arbitrary string (that is not used by the callee).
Additionally: note in opcodes.py that groups also expect a name; the
previous comment only referred to nodes and instances.
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Bernardo Dal Seno [Thu, 1 Nov 2012 20:45:18 +0000 (21:45 +0100)]
Fix previous merge
A call to _CalculateGroupIPolicy wasn't refactored during the merge.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Thu, 1 Nov 2012 17:27:27 +0000 (18:27 +0100)]
Merge branch 'devel-2.6'
* devel-2.6:
Fix typo in gnt-instance man page
jqueue: Return jobs to queue when shutting down
gnt-debug delay: Add "--submit" option
Make hostname checks uniform between instance rename and add
Improve logging of new job submissions
Improve handling of lock exceptions
Add note about developing on a production machine
Fix runtime memory increases
Fix validation of vgname in OpClusterSetParams
Fix removal of storage directory on shared file storage
Switch non-redundant check to disk template-based
Document the new --yes-do-it option for master-failover
Add option to force master-failover without voting
Update instance modify message
Force tap's MAC prefix to "fe"
Fix disk adoption interaction with ipolicy checks
Better NEWS file for 2.6.1
Conflicts:
lib/cmdlib.py: Trivial
tools/kvm-ifup.in: Space vs. tab
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 1 Nov 2012 15:59:01 +0000 (16:59 +0100)]
Merge branch 'stable-2.6' into devel-2.6
* stable-2.6:
Fix typo in gnt-instance man page
jqueue: Return jobs to queue when shutting down
gnt-debug delay: Add "--submit" option
Make hostname checks uniform between instance rename and add
Improve logging of new job submissions
Improve handling of lock exceptions
Add note about developing on a production machine
Fix runtime memory increases
Fix validation of vgname in OpClusterSetParams
Fix removal of storage directory on shared file storage
Switch non-redundant check to disk template-based
Document the new --yes-do-it option for master-failover
Add option to force master-failover without voting
Update instance modify message
Force tap's MAC prefix to "fe"
Fix disk adoption interaction with ipolicy checks
Better NEWS file for 2.6.1
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 1 Nov 2012 15:04:33 +0000 (16:04 +0100)]
Fix typo in gnt-instance man page
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 1 Nov 2012 13:54:03 +0000 (14:54 +0100)]
jqueue: Return jobs to queue when shutting down
When a job is still waiting for locks and the queue is shutting down,
they should be returned and not actually start processing. Until now
jobs which transitioned from “queued” to “waiting” were already
considered to be running as far as the shutdown code was concerned.
This fixes issue 296.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 1 Nov 2012 14:23:54 +0000 (15:23 +0100)]
gnt-debug delay: Add "--submit" option
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 31 Oct 2012 13:17:09 +0000 (14:17 +0100)]
Remove duplicate workerpool test
Commit
52c47e4e (July 2010) added the exact test twice, probably due to
a copy & paste error.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 31 Oct 2012 16:10:19 +0000 (17:10 +0100)]
Make hostname checks uniform between instance rename and add
Currently, we have instance rename doing extra checks on the host
name, to prevent accidental wrong renames; however, instance create
doesn't do these checks (issue 291), which (if DNS is misconfigured)
can lead to hard to diagnose errors.
This patch abstracts the name checking from LUInstanceRename into a
separate function, which is then reused in both instance rename and
instance create.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Iustin Pop [Wed, 31 Oct 2012 13:47:18 +0000 (14:47 +0100)]
Improve logging of new job submissions
This addresses issue 290: when receiving new jobs, logging is
incomplete, and we don't have the job ID(s) and/or summaries
logged. Only later, when the job is queried for or being processed, we
know more.
This is not good when troubleshooting, so let's improve the initial
logging.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Iustin Pop [Wed, 31 Oct 2012 13:31:48 +0000 (14:31 +0100)]
Improve handling of lock exceptions
There are two issues with lock exceptions right now:
- first, we don't log the original error; this is fine for now
(locking.py always returns the same error here), but in general is
brittle: if locking.py would start returning more information, we'd
completely miss that
- second, an actual honest lock conflict is not an internal error;
it's simply an optimistic lock failing, and as such we should not
return internal error, but rather resource_not_unique
This addresses issue 287.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Iustin Pop [Tue, 30 Oct 2012 15:49:33 +0000 (16:49 +0100)]
Add note about developing on a production machine
This is the bit of documentation missing for issue 170. Doing
development on a machine which already has Ganeti installed kind of
works, but only when the installed and the developed version are very
similar, and even then it can be problematic.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 30 Oct 2012 13:43:09 +0000 (14:43 +0100)]
Fix runtime memory increases
Commit
2c0af7da which added the runtime memory changes functionality
had a small typo (wrong name); I've rewritten this to only compute the
delta once, for simplicity.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 30 Oct 2012 11:53:26 +0000 (12:53 +0100)]
Fix validation of vgname in OpClusterSetParams
This variable can be empty, when we want to disable LVM, so we can't
use TMaybeString.
Fixes issue 285.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 30 Oct 2012 11:29:16 +0000 (12:29 +0100)]
Fix removal of storage directory on shared file storage
This patch makes _RemoveDisks symmetric to _CreateDisks with respect
to file-based storage: _CreateDisks uses "in constants.DTS_FILEBASED",
whereas _RemoveDisks was not update and only uses "==
constants.DT_FILE". This results in stale directories left on the
filesystem.
Fixes issue 262.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Tue, 30 Oct 2012 11:14:37 +0000 (12:14 +0100)]
Switch non-redundant check to disk template-based
Currently, the warning/notice about non-redundant instances in cluster
verify is based non empty secondaries list (how old is this?); the
proper way to check this nowadays is via DTS_MIRRORED.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Bernardo Dal Seno [Mon, 29 Oct 2012 17:44:35 +0000 (18:44 +0100)]
Fix permission for socket directory
The directory must we writable also by the confd daemon user.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Mon, 29 Oct 2012 12:01:58 +0000 (13:01 +0100)]
Document the new --yes-do-it option for master-failover
Sorry, this should have gone in the previous commit, I forgot about
it.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Iustin Pop [Mon, 29 Oct 2012 11:21:59 +0000 (12:21 +0100)]
Add option to force master-failover without voting
This fixes issue 282.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Iustin Pop [Fri, 26 Oct 2012 13:27:11 +0000 (15:27 +0200)]
Convert Luxi results to Ganeti errors
This a bit too complex patch converts the result of Luxi calls
(submitJob, query*, etc.) from Result to ErrorResult. It then
immediately revers this in the HTools/Backend/Luxi module, where we
don't need necessarily the full error type (just a nice error
message), and does the same in Hbal's job execution functions.
While at first sight this doesn't seem to do much, what we get is
actual error messages from Ganeti, plus improvements to the result
parsing: instead of "can't parse char", we now get properly (note,
wrapped manually):
Executing jobset for instances instance1, …
Job submission error: Failure: the job queue is marked for drain and
doesn't accept new requests
Or:
Job submission error: Unhandled exception: LuxiError "parsing job
id: cannot parse string 'a956101'"
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Michael Hanselmann [Fri, 5 Oct 2012 01:32:44 +0000 (03:32 +0200)]
cfgupgrade: Write file for file storage paths
When file storage is used this file is now mandatory.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 5 Oct 2012 00:59:12 +0000 (02:59 +0200)]
backend: Switch to new file storage directory verification
The configuration is no longer used for verifying file storage paths.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Fri, 5 Oct 2012 00:43:47 +0000 (02:43 +0200)]
Check allowed file storage paths during cluster-verify
Some paths, such as /bin or /usr/lib, should not be used for file
storage. This patch implements a check during cluster verification to
fail in case such a path has been used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 24 Oct 2012 00:01:42 +0000 (02:01 +0200)]
Make Paramiko an optional dependency for listrunner
With the move away from “setup-ssh”, Paramiko is no longer necessary to
configure SSH on nodes.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 23:58:37 +0000 (01:58 +0200)]
Remove setup-ssh
It has been superseeded by “prepare-node-join”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 18:16:25 +0000 (20:16 +0200)]
gnt-node add: Use prepare-node-join
This patch changes “gnt-node add” to use the newly added
“prepare-node-join” tool. Hereby Paramiko is no longer a hard dependency
for setting up SSH on nodes.
In “gnt_cluster.py”, a positional parameter is no longer passed as a
keyword parameter.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 23:16:50 +0000 (01:16 +0200)]
prepare-node-join: Use ssh.GetAllUserFiles
Instead of building the dictionary locally, the global version in
“ssh.py” can be used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 23:10:36 +0000 (01:10 +0200)]
ssh: Add function to get all of user's SSH files
This new function returns the file paths for all of a user's SSH-related
files (RSA, DSA and authorized_keys).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 19:25:30 +0000 (21:25 +0200)]
RunCmd: Support standard input file descriptor
This patch changes “utils.RunCmd” to accept a file-like object or a
numeric file descriptor which will be used as the command's standard
input. One use-case will be to pass all necessary data to
“prepare-node-join”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Thu, 25 Oct 2012 15:29:51 +0000 (17:29 +0200)]
Factorize job selection in “gnt-job cancel”
This will also be used for changing jobs' priorities. All parameters to
the common function are non-optional.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 23:55:53 +0000 (01:55 +0200)]
utils.x509: Factorize code to extract X509 certificate
This will be useful in “gnt-node add”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 23:11:45 +0000 (01:11 +0200)]
prepare_node_join: Move daemon SSH files to constants
This dictionary will also be useful in “gnt-node add”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 23:24:19 +0000 (01:24 +0200)]
prepare-node-join: Swap private and public keys
Other places, such as “ssh.GetUserFiles”, use a structure where the
private key comes before the private key. Until now prepare-node-join
did the opposite, that is the public key came first. To avoid confusion
and potential bugs, this is changed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 22:26:16 +0000 (00:26 +0200)]
prepare-node-join: Use public key directly for auth…_keys
A public key already includes the necessary prefix (“ssh-rsa” or
“ssh-dss”), so there is no need to add it again.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 22:55:39 +0000 (00:55 +0200)]
ssh.GetUserFiles: Parameter to disable directory check
Without this parameter, either an error would be raised or “.ssh” would
have to be created. Now it is possible to retrieve the paths without
requiring the “.ssh” directory to exist.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Fri, 26 Oct 2012 07:32:35 +0000 (09:32 +0200)]
Move htools backends to a separate directory
Five modules under the HTools/ directories are backend
implementations, so let's move them to a separate directory, to more
clearly show the hierarchy. I wanted to do this for a while, but
merging between branches is always an issue, so let's do it know since
we have an opportunity.
This patch contains the actual renames, the required changed module
names, imports, etc., but no other changes.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 26 Oct 2012 11:48:45 +0000 (13:48 +0200)]
Fix lint issue in Test/…/THH.hs
Sorry!
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 26 Oct 2012 08:22:21 +0000 (10:22 +0200)]
Fix X509CertError definition in Haskell codebase
Thanks Dato for catching this.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 26 Oct 2012 06:49:48 +0000 (08:49 +0200)]
Fix a few issues found by newer hlint
Testing with a newer hlint found a few minor issues; but all are real,
valid recommendations:
- don't use "if cond then f x else f y", but "f (if cond then x else y)"
- "if a then b else True" is equivalent to the simpler "not a || b"
- and as usual, one more ignore to our "testing basic properties"
module
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Thu, 25 Oct 2012 14:04:56 +0000 (16:04 +0200)]
Add a simple unittest for THH code
This is very THH specific, and applies to all serialisations generated
by THH, so I'm adding it in its own module.
Probably we should add some more generic tests, but in general THH
code is tested by the various definitions; this new field type however
is not (yet), so this is why I want this specific unittest.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Thu, 25 Oct 2012 13:48:05 +0000 (15:48 +0200)]
Add support for optional fields with null serialised
This follows a conversation we had for how to deal with
optional-but-required fields in JSON serialisations: fields which are
optional (can be either a given type or 'null'), but where the 'null'
value is required. There are just a few of these in the Python code,
but we should support them nevertheless.
The patch changes the 'isOptional' attribute from boolean to a custom
ADT, three-typed. This allows us to keep the same path on load (which
deals with both cases), but use a custom save path where we explicitly
save the 'null' value.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Fri, 26 Oct 2012 06:41:07 +0000 (08:41 +0200)]
Update instance modify message
Currently the message does not say explicitly that instance-initiated
reboots are useless to trigger the use of new parameters, per the
thread on the user mailing list. Let's improve it a bit.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Dato Simó [Thu, 25 Oct 2012 16:11:12 +0000 (17:11 +0100)]
Errors.hs: improve field names for ConfigVersionMismatch
Change {exp,act}Code to {exp,act}Ver, which gives a better idea that
the integer fields represent version numbers.
Also:
- errors.py: update OpPrereqError's docstring to note that an error
code is always expected as the second argument (it was previously
optional).
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Thu, 25 Oct 2012 14:21:25 +0000 (16:21 +0200)]
Remove unused cache implementation
Note that this commit has no Makefile.am changes, as the files were
not actually used. So it's better to actually remove them.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Dato Simó [Wed, 24 Oct 2012 14:28:57 +0000 (15:28 +0100)]
THH.hs: delete isOptional, no longer used
The isOptional function is no longer used after a150585 (“Convert
opcode TH code to the use of Field type”).
Signed-off-by: Dato Simó <dato@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michele Tartara [Tue, 23 Oct 2012 15:55:14 +0000 (17:55 +0200)]
Fix two typos in Ganeti administrator's guide
Fix a verb tense and add a missing verb.
Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Michael Hanselmann [Thu, 25 Oct 2012 12:54:53 +0000 (14:54 +0200)]
bdev: Remove unused import of itertools
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Michael Hanselmann [Wed, 17 Oct 2012 14:10:07 +0000 (16:10 +0200)]
bdev: Add verification for file storage paths
An earlier version of this patch series verified all paths in cmdlib in
the master daemon. With this change all that verification code is moved
to bdev to run inside the node daemon. The checks are much stricter
now--it is no longer possible to use forbidden paths (e.g. /bin) to
manipulate file storage devices (once these checks are being used).
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Michael Hanselmann [Wed, 24 Oct 2012 01:19:20 +0000 (03:19 +0200)]
jqueue: Factorize code to modify job
A new function will be added to change a job's priority.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Wed, 24 Oct 2012 02:08:42 +0000 (04:08 +0200)]
cli: Use callback for --priority
If the option is used elsewhere, the numeric value is directly
available.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Michael Hanselmann [Wed, 24 Oct 2012 00:37:34 +0000 (02:37 +0200)]
jqueue: Add docstring for _DetermineJobDirectories
Somehow this was missed in commit
0422250e.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Wed, 24 Oct 2012 00:38:27 +0000 (02:38 +0200)]
jqueue: Fix comments in _SubmitJobUnlocked
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Michael Hanselmann [Tue, 23 Oct 2012 21:52:44 +0000 (23:52 +0200)]
Drop SSHS_FORCE constant
It is not actually used.
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Thu, 25 Oct 2012 10:33:10 +0000 (12:33 +0200)]
Add a default `.ghci' file
This options file for ghci preseeds the correct include paths, so that
interactive sessions don't need to always pass these args.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Simon Deziel [Fri, 11 May 2012 16:00:24 +0000 (12:00 -0400)]
Force tap's MAC prefix to "fe"
Setting a high prefix discourages the bridge from adopting the tap's
MAC. Xen is not affected by this since the MAC is forced to
"fe:ff:ff:ff:ff:ff".
This addresses issue #217.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
(cherry picked from commit
be3f52f304767a719d4b0dc07c81a1aaea8432b0)
(this is critical enough that we want it on stable-2.6)
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Fri, 12 Oct 2012 12:37:20 +0000 (14:37 +0200)]
Convert query path from string errors to GanetiException
This patch converts all the call paths from 'Result' (which contains
just string errors) to 'ErrorResult', which holds
GanetiException-encoded errors. We can now return proper
OpPrereq/OpExec errors to the clients of the luxi/query socket.
The patch touches many files as we had to convert the entire call
chains in a single round. But it should be pretty straightforward
otherwise:
- change 'Result' into 'ErrorResult'
- add error annotations: change "Bad msg" into "Bad (XXXEror msg)"
- add a helper function for confd, where we don't send to client
formatted exceptions, to convert back from ErrorResult into Result
- change tests similarly, where needed
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Fri, 12 Oct 2012 10:37:27 +0000 (12:37 +0200)]
Add exception utility functions
In Python, formatError also returns the exit code, but I find that
splitting them leads to clearer code.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Sun, 7 Oct 2012 22:02:20 +0000 (00:02 +0200)]
Add an Errors module mirroring the Python one
As described in the module doc string, while writing this it dawned
upon me that we're mixing all errors together into a single hierarchy
(well, type on the Haskell side), which is not good. Some errors are
used purely within noded, some in the CLI frontends, etc. so these
should not be the same type; frontend functions should only be able to
raise frontend errors, not backend ones.
As to this patch itself, I've used again Template Haskell to generate
both the data type and the serialisation functions, as the initial
version, hand-written, seemed too prone to errors due to string
matching.
A small unittest for checking serialisation consistency is also added.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Iustin Pop [Fri, 12 Oct 2012 09:09:05 +0000 (11:09 +0200)]
Abstract Luxi template functionality
These are almost generic, so let's change the signatures a bit a make
them fully so.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Adeodato Simo <dato@google.com>
Iustin Pop [Fri, 12 Oct 2012 08:51:26 +0000 (10:51 +0200)]
Abstract a few types in THH.hs
This 'simple' way of defining objects will be used also for errors, so
let's make it less Luxi-specific.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Guido Trotter [Wed, 24 Oct 2012 12:11:51 +0000 (14:11 +0200)]
Add missing empty line
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Iustin Pop [Wed, 24 Oct 2012 10:25:30 +0000 (12:25 +0200)]
Remove dumb-allocator code from devel/upload
The 'dumb-allocator' has been removed almost two years ago (commit
6f547f96, “Remove dumb-allocator”), let's remove this special casing
for it from devel/upload.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Iustin Pop [Wed, 24 Oct 2012 10:23:25 +0000 (12:23 +0200)]
Fix devel/upload restart of daemons
While running with a wrong --prefix/--sysconfdir, I saw that
devel/upload actually uses a hardcoded path for the init script, even
though it installs it in the correct place.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
Guido Trotter [Wed, 24 Oct 2012 10:08:48 +0000 (12:08 +0200)]
Add small design for Linux HA integration
This documents the status (or wanted status) of some example ocf modules
I've written for Ganeti. They are far from perfect, but they should be
shipped as a starting point for other people who want to run this to
improve on.
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Guido Trotter [Fri, 4 Nov 2011 09:57:27 +0000 (09:57 +0000)]
Replace @LIBDIR@ in .in files
Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 23 Oct 2012 16:19:01 +0000 (18:19 +0200)]
Improve devel/upload command line handling
To people not used to it, it was completely non-obvious why
"./devel/upload" didn't do anything.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>
Iustin Pop [Tue, 23 Oct 2012 14:01:18 +0000 (16:01 +0200)]
Improve logging of AssertionErrors
Currently, when we have an assertion error raised from cmdlib, it looks like this:
[cluster] root@node4:~# gnt-instance grow-disk instance1 0 1G
Failure: command execution error:
This is very very confusing. This patch adds a bit of traceback
formatting to improve this as follows:
[cluster] root@node4:~# gnt-instance grow-disk instance1 0 1G
Failure: command execution error:
Internal assertion error: please report this as a bug.
Error message: ''; location:
File "/usr/lib/python2.6/dist-packages/ganeti/cmdlib.py", line 11954, in CheckPrereq
assert False
This is not perfect, but at least it shows better what the problem is.
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>